invalid byte sequence in US-ASCII

[twillouer]

Hi,

thanks for your work.

Affected Puppet, Ruby, OS and module versions/distributions

• Puppet: 3.7.5
• Distribution: debian 8
• Module version: v1.1.0

What are you seeing

invalid byte sequence in US-ASCII in mongo.rb

I jump from 0.17.0 to v1.1.0 and I got
'invalid byte sequence in US-ASCII,change from {md5}38efb877a97fd0e162252e4681d65c46 to {md5}b9939023cdbe3bd5d3352e1db68017be failed: invalid byte sequence in US-ASCII,invalid byte sequence in US-ASCII,change from {md5}85a74779d028eff9872831590a7693ab to {md5}8e29585c9aab0293f8ad5e52f15d9eb5 failed: invalid byte sequence in US-ASCII,removed,Finished catalog run in 7.81 seconds'

on all my machine.

in script 'mongodb_replset.rb' and/or 'mongo.rb'

any help ?

[alexjfisher]

Do you know in which file resource this is? My guess is that puppet is choking whilst trying to generate a diff to show you. I've come across this before in another module. See https://github.com/theforeman/puppet-tftp/pull/58/files#diff-0ac866172efd29d0ddc2aa75ae6f182dR34

[twillouer]

I got (in syslog) :

Dec  6 01:26:35 puppet-agent[22672]: invalid byte sequence in US-ASCII
Dec  6 01:26:35  puppet-agent[22672]: (/File[/var/lib/puppet/lib/puppet/provider/mongodb_replset/mongo.rb]/content) change from {md5}38efb877a97fd0e162252e4681d65c46 to {md5}b9939023cdbe3bd5d3352e1db68017be failed: invalid byte sequence in US-ASCII
Dec  6 01:26:36  puppet-agent[22672]: invalid byte sequence in US-ASCII
Dec  6 01:26:36  puppet-agent[22672]: (/File[/var/lib/puppet/lib/puppet/type/mongodb_replset.rb]/content) change from {md5}85a74779d028eff9872831590a7693ab to {md5}8e29585c9aab0293f8ad5e52f15d9eb5 failed: invalid byte sequence in US-ASCII

if i try to apply with my user

echo $LANG
en_US.UTF-8

sudo puppet agent -tv
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: /File[/var/lib/puppet/lib/puppet/provider/mongodb_replset/mongo.rb]/content: 
--- /var/lib/puppet/lib/puppet/provider/mongodb_replset/mongo.rb	2017-06-28 12:42:59.374744111 +0200
+++ /tmp/puppet-file20171206-24995-1wayhxf	2017-12-06 01:40:20.564745653 +0100
@@ -1,14 +1,12 @@
 #
-# Author: François Charlier <francois.charlier@enovance.com>
+# Author: Francois Charlier <francois.charlier@enovance.com>
 #
 
 require File.expand_path(File.join(File.dirname(__FILE__), '..', 'mongodb'))
-Puppet::Type.type(:mongodb_replset).provide(:mongo, :parent => Puppet::Provider::Mongodb) do
+Puppet::Type.type(:mongodb_replset).provide(:mongo, parent: Puppet::Provider::Mongodb) do
+  desc 'Manage hosts members for a replicaset.'
 
-  desc "Manage hosts members for a replicaset."
-
-  confine :true =>
-    begin
+  confine true:     begin
       require 'json'
       true
     rescue LoadError
@@ -17,7 +15,7 @@
 
   mk_resource_methods
 
-  def initialize(resource={})
+  def initialize(resource = {})
     super(resource)
     @property_flush = {}
   end
@@ -27,7 +25,7 @@
   end
 
   def self.instances
-    instance = get_replset_properties
+    instance = replset_properties
     if instance
       # There can only be one replset per node
       [new(instance)]
@@ -38,9 +36,8 @@
 
   def self.prefetch(resources)
     instances.each do |prov|
-      if resource = resources[prov.name]
-        resource.provider = prov
-      end
+      resource = resources[prov.name]
+      resource.provider = prov if resource
     end
   end
 
@@ -59,7 +56,7 @@
 
   def flush
     set_members
-    @property_hash = self.class.get_replset_properties
+    @property_hash = self.class.replset_properties
   end
 
   private
@@ -70,9 +67,9 @@
 
   def rs_initiate(conf, master)
     if auth_enabled && auth_enabled != 'disabled'
-      return mongo_command("rs.initiate(#{conf})", initialize_host)
+      mongo_command("rs.initiate(#{conf})", initialize_host)
     else
-      return mongo_command("rs.initiate(#{conf})", master)
+      mongo_command("rs.initiate(#{conf})", master)
     end
   end
 
@@ -107,29 +104,27 @@
   def master_host(hosts)
     hosts.each do |host|
       status = db_ismaster(host)
-      if status.has_key?('primary')
-        return status['primary']
-      end
+      return status['primary'] if status.key?('primary')
     end
     false
   end
 
-  def self.get_replset_properties
-    conn_string = get_conn_string
+  def self.replset_properties
+    conn_string = conn_string
     begin
       output = mongo_command('rs.conf()', conn_string)
-    rescue Puppet::ExecutionFailure => e
+    rescue Puppet::ExecutionFailure
       output = {}
     end
     if output['members']
-      members = output['members'].collect do |val|
+      members = output['members'].map do |val|
         val['host']
       end
       props = {
-        :name     => output['_id'],
-        :ensure   => :present,
-        :members  => members,
-        :provider => :mongo,
+        name: output['_id'],
+        ensure: :present,
+        members: members,
+        provider: :mongo
       }
     else
       props = nil
@@ -144,24 +139,24 @@
       begin
         Puppet.debug "Checking replicaset member #{host} ..."
         status = rs_status(host)
-        if status.has_key?('errmsg') and status['errmsg'] == 'not running with --replSet'
-          raise Puppet::Error, "Can't configure replicaset #{self.name}, host #{host} is not supposed to be part of a replicaset."
+        if status.key?('errmsg') && status['errmsg'] == 'not running with --replSet'
+          raise Puppet::Error, "Can't configure replicaset #{name}, host #{host} is not supposed to be part of a replicaset."
         end
 
-        if auth_enabled and status.has_key?('errmsg') and (status['errmsg'].include? "unauthorized" or status['errmsg'].include? "not authorized")
+        if auth_enabled && status.key?('errmsg') && (status['errmsg'].include?('unauthorized') || status['errmsg'].include?('not authorized'))
           Puppet.warning "Host #{host} is available, but you are unauthorized because of authentication is enabled: #{auth_enabled}"
           alive.push(host)
         end
 
-        if status.has_key?('set')
-          if status['set'] != self.name
-            raise Puppet::Error, "Can't configure replicaset #{self.name}, host #{host} is already part of another replicaset."
+        if status.key?('set')
+          if status['set'] != name
+            raise Puppet::Error, "Can't configure replicaset #{name}, host #{host} is already part of another replicaset."
           end
 
           # This node is alive and supposed to be a member of our set
           Puppet.debug "Host #{host} is available for replset #{status['set']}"
           alive.push(host)
-        elsif status.has_key?('info')
+        elsif status.key?('info')
           Puppet.debug "Host #{host} is alive but unconfigured: #{status['info']}"
           alive.push(host)
         end
@@ -169,47 +164,45 @@
         Puppet.warning "Can't connect to replicaset member #{host}."
       end
     end
-    return alive
+    alive
   end
 
   def set_members
     if @property_flush[:ensure] == :absent
       # TODO: I don't know how to remove a node from a replset; unimplemented
-      #Puppet.debug "Removing all members from replset #{self.name}"
-      #@property_hash[:members].collect do |member|
+      # Puppet.debug "Removing all members from replset #{self.name}"
+      # @property_hash[:members].collect do |member|
       #  rs_remove(member, master_host(@property_hash[:members]))
-      #end
+      # end
       return
     end
 
-    if ! @property_flush[:members].empty?
+    if !@property_flush[:members].empty?
       # Find the alive members so we don't try to add dead members to the replset
       alive_hosts = alive_members(@property_flush[:members])
       dead_hosts  = @property_flush[:members] - alive_hosts
       Puppet.debug "Alive members: #{alive_hosts.inspect}"
       Puppet.debug "Dead members: #{dead_hosts.inspect}" unless dead_hosts.empty?
-      raise Puppet::Error, "Can't connect to any member of replicaset #{self.name}." if alive_hosts.empty?
+      raise Puppet::Error, "Can't connect to any member of replicaset #{name}." if alive_hosts.empty?
     else
       alive_hosts = []
     end
 
-    if @property_flush[:ensure] == :present and @property_hash[:ensure] != :present and !master_host(alive_hosts)
-      Puppet.debug "Initializing the replset #{self.name}"
+    if @property_flush[:ensure] == :present && @property_hash[:ensure] != :present && !master_host(alive_hosts)
+      Puppet.debug "Initializing the replset #{name}"
 
       # Create a replset configuration
-      hostconf = alive_hosts.each_with_index.map do |host,id|
-        arbiter_conf = ""
-        if rs_arbiter == host
-          arbiter_conf = ", arbiterOnly: \"true\""
-        end
+      hostconf = alive_hosts.each_with_index.map do |host, id|
+        arbiter_conf = ''
+        arbiter_conf = ', arbiterOnly: "true"' if rs_arbiter == host
         "{ _id: #{id}, host: \"#{host}\"#{arbiter_conf} }"
       end.join(',')
-      conf = "{ _id: \"#{self.name}\", members: [ #{hostconf} ] }"
+      conf = "{ _id: \"#{name}\", members: [ #{hostconf} ] }"
 
       # Set replset members with the first host as the master
       output = rs_initiate(conf, alive_hosts[0])
-      if output['ok'] == 0
-        raise Puppet::Error, "rs.initiate() failed for replicaset #{self.name}: #{output['errmsg']}"
+      if output['ok'].zero?
+        raise Puppet::Error, "rs.initiate() failed for replicaset #{name}: #{output['errmsg']}"
       end
 
       # Check that the replicaset has finished initialization
@@ -220,7 +213,7 @@
         begin
           if db_ismaster(alive_hosts[0])['ismaster']
             Puppet.debug 'Replica set initialization has successfully ended'
-            return
+            return true
           else
             Puppet.debug "Wainting for replica initialization. Retry: #{n}"
             sleep retry_sleep
@@ -228,40 +221,35 @@
           end
         end
       end
-      raise Puppet::Error, "rs.initiate() failed for replicaset #{self.name}: host #{alive_hosts[0]} didn't become master"
+      raise Puppet::Error, "rs.initiate() failed for replicaset #{name}: host #{alive_hosts[0]} didn't become master"
 
     else
       # Add members to an existing replset
-      Puppet.debug "Adding member to existing replset #{self.name}"
-      if master = master_host(alive_hosts)
-        master_data = db_ismaster(master)
-        current_hosts = master_data['hosts']
-        current_hosts = current_hosts + master_data['arbiters'] if master_data.has_key?('arbiters')
-        Puppet.debug "Current Hosts are: #{current_hosts.inspect}"
-        newhosts = alive_hosts - current_hosts
-        Puppet.debug "New Hosts are: #{newhosts.inspect}"
-        newhosts.each do |host|
-          output = {}
-          if rs_arbiter == host
-            output = rs_add_arbiter(host, master)
-          else
-            output = rs_add(host, master)
-          end
-          if output['ok'] == 0
-            raise Puppet::Error, "rs.add() failed to add host to replicaset #{self.name}: #{output['errmsg']}"
-          end
-        end
-      else
-        raise Puppet::Error, "Can't find master host for replicaset #{self.name}."
+      Puppet.debug "Adding member to existing replset #{name}"
+
+      master = master_host(alive_hosts)
+      raise Puppet::Error, "Can't find master host for replicaset #{name}." unless master
+
+      master_data = db_ismaster(master)
+      current_hosts = master_data['hosts']
+      current_hosts += master_data['arbiters'] if master_data.key?('arbiters')
+      Puppet.debug "Current Hosts are: #{current_hosts.inspect}"
+      newhosts = alive_hosts - current_hosts
+      Puppet.debug "New Hosts are: #{newhosts.inspect}"
+
+      newhosts.each do |host|
+        output = {}
+        output = rs_arbiter == host ? rs_add_arbiter(host, master) : rs_add(host, master)
+        raise Puppet::Error, "rs.add() failed to add host to replicaset #{name}: #{output['errmsg']}" if output['ok'].zero?
       end
     end
   end
 
-  def mongo_command(command, host, retries=4)
+  def mongo_command(command, host, retries = 4)
     self.class.mongo_command(command, host, retries)
   end
 
-  def self.mongo_command(command, host=nil, retries=4)
+  def self.mongo_command(command, host = nil, retries = 4)
     begin
       output = mongo_eval("printjson(#{command})", 'admin', retries, host)
     rescue Puppet::ExecutionFailure => e
@@ -270,15 +258,14 @@
     end
 
     # Dirty hack to remove JavaScript objects
-    output.gsub!(/\w+\((?!")(\d+).+?(?<!")\)/, '\1')  # Remove extra parameters from 'Timestamp(1462971623, 1)' Objects
-    output.gsub!(/\w+\((.+?)\)/, '\1')
+    output.gsub!(%r{\w+\((\d+).+?\)}, '\1') # Remove extra parameters from 'Timestamp(1462971623, 1)' Objects
+    output.gsub!(%r{\w+\((.+?)\)}, '\1')
 
-    #Hack to avoid non-json empty sets
-    output = "{}" if output == "null\n"
+    # Hack to avoid non-json empty sets
+    output = '{}' if output == "null\n"
+    output = '{}' if output == "\nnull\n"
 
     # Parse the JSON output and return
     JSON.parse(output)
-
   end
-
 end

Notice: /File[/var/lib/puppet/lib/puppet/provider/mongodb_replset/mongo.rb]/content: content changed '{md5}38efb877a97fd0e162252e4681d65c46' to '{md5}b9939023cdbe3bd5d3352e1db68017be'
Notice: /File[/var/lib/puppet/lib/puppet/type/mongodb_replset.rb]/content: 
--- /var/lib/puppet/lib/puppet/type/mongodb_replset.rb	2016-06-16 11:40:35.982669901 +0200
+++ /tmp/puppet-file20171206-24995-5xwt2m	2017-12-06 01:40:21.664859048 +0100
@@ -1,9 +1,9 @@
 #
-# Author: François Charlier <francois.charlier@enovance.com>
+# Author: Francois Charlier <francois.charlier@enovance.com>
 #
 
 Puppet::Type.newtype(:mongodb_replset) do
-  @doc = "Manage a MongoDB replicaSet"
+  @doc = 'Manage a MongoDB replicaSet'
 
   ensurable do
     defaultto :present
@@ -14,20 +14,20 @@
   end
 
   newparam(:name) do
-    desc "The name of the replicaSet"
+    desc 'The name of the replicaSet'
   end
 
   newparam(:arbiter) do
-    desc "The replicaSet arbiter"
+    desc 'The replicaSet arbiter'
   end
 
   newparam(:initialize_host) do
-    desc "Host to use for Replicaset initialization"
+    desc 'Host to use for Replicaset initialization'
     defaultto '127.0.0.1'
   end
 
-  newproperty(:members, :array_matching => :all) do
-    desc "The replicaSet members"
+  newproperty(:members, array_matching: :all) do
+    desc 'The replicaSet members'
 
     def insync?(is)
       is.sort == should.sort

Notice: /File[/var/lib/puppet/lib/puppet/type/mongodb_replset.rb]/content: content changed '{md5}85a74779d028eff9872831590a7693ab' to '{md5}8e29585c9aab0293f8ad5e52f15d9eb5'

[ekohl]

Looks like the file was already changed not to include UTF-8 characters so I'm going to close this. Please open a new PR if the issue is still present in the latest version.