New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

file.recurse fails on file contents encoding not utf-8 #48777

Closed
jonasgit opened this Issue Jul 26, 2018 · 6 comments

Comments

Projects
None yet
3 participants
@jonasgit

jonasgit commented Jul 26, 2018

Description of Issue/Question

Trying to distribute files with contents encoded in ISO-8859-1. However salt assumes systemdefault
utf-8 and thus fails on illegal byte sequences when trying to merge updates.

Setup

One saltmaster on Cent OS 7. One minion on Cent OS 7.

state file foo.sls:

webfoo:
  file.recurse:
    - name: /tmp/foo/
    - source: salt://foo/
    - user: root
    - group: root
    - file_mode: 444
    - dir_mode: 555
    - makedirs: True

fileone.tar.gz
filetwo.tar.gz

Steps to Reproduce Issue

Populate directory foo using fileone.tar (plain ascii I believe).
Distribute file using salt '*' state.apply foo
Update file on saltmaster using filetwo.tar (iso-8859-1 contents added).
Try distribute file using salt '*' state.apply foo
Get error:
ID: webfoo
Function: file.recurse
Name: /tmp/foo/
Result: False
Comment: #### /tmp/foo/index.html ####
Unable to manage file: 'utf8' codec can't decode byte 0xe4 in position 1: invalid continuation byte

Versions Report

salt --versions-report

Salt Version:
Salt: 2018.3.2

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Jul 13 2018, 13:06:57)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: centos 7.5.1804 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-862.9.1.el7.x86_64
system: Linux
version: CentOS Linux 7.5.1804 Core

@Ch3LL

This comment has been minimized.

Contributor

Ch3LL commented Jul 26, 2018

i'm having a hard time replicating this:

local:
----------
          ID: webfoo
    Function: file.recurse
        Name: /tmp/foo/
      Result: True
     Comment: Recursively updated /tmp/foo/
     Started: 15:05:07.260157
    Duration: 53.367 ms
     Changes:   
              ----------
              /tmp/foo/fileone.tar.gz:
                  ----------
                  diff:
                      New file
                  mode:
                      0444
              /tmp/foo/filetwo.tar.gz:
                  ----------
                  diff:
                      New file
                  mode:
                      0444

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  53.367 ms
[root@36f0b56b4b2b foo]# ls /tmp/foo/
fileone.tar.gz  filetwo.tar.gz

is 2018.3.2 the version of your minion and master?

@Ch3LL Ch3LL added this to the Blocked milestone Jul 26, 2018

@jonasgit

This comment has been minimized.

jonasgit commented Jul 26, 2018

Sorry, I missed a crucial step: The foo directory should be populated with the content of the tar file, not the tar file itself. The tar files contains a file index.html and the objective is to replace it on the minion.

And yes the version should be identical on both server and minion in my set up. Version report above is from server and from minion is:
Salt Version:
Salt: 2018.3.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Jul 13 2018, 13:06:57)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.5.1804 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-862.9.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7.5.1804 Core
@Ch3LL

This comment has been minimized.

Contributor

Ch3LL commented Jul 27, 2018

k looks like i'm able to replicate this now :)

Here is a docker container to easily replicate the issue:

  1. docker run -it -v ~/git/salt/:/testing/ ch3ll/issues:48777 (where ~/git/salt/ is a local cloned git repo of salt)
  2. salt-call --local state.sls test -ldebug
@Ch3LL

This comment has been minimized.

Contributor

Ch3LL commented Jul 27, 2018

ping @terminalmage ^

@terminalmage

This comment has been minimized.

Member

terminalmage commented Aug 3, 2018

@Ch3LL this should be fixed in #48934. Can you test on your end?

@Ch3LL

This comment has been minimized.

Contributor

Ch3LL commented Aug 6, 2018

works like a charm :)

@Ch3LL Ch3LL closed this Aug 6, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment