file.append fails when encoding doesn't match system encoding

[squidpickles]

Description of Issue/Question

Using file.append state on a file with an encoding that's incompatible with the system encoding fails.
For example, I have a file containing ISO-8859-1 encoded data. The system encoding is ASCII. If I try to use file.append, it fails with the following error:

An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 1944, in call
    **cdata['kwargs'])
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1959, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/states/file.py", line 5669, in append
     slines = slines.decode(__salt_system_encoding__)
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xed in position 4: ordinal not in range(128)

If the system encoding were UTF-8, it would similarly fail because 0xed is not a valid UTF-8 byte.

In the simplest case, it would be helpful if the state.file functions that work with existing files could take an encoding, much as managed files do.

Setup

See attached zip file with Docker config that reproduces the bug

Steps to Reproduce Issue

1. Build attached config: docker build salt-bug -t salt-bug:latest
2. Run configuration: docker run -it --rm salt-bug:latest
   Output:

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 1944, in call
    **cdata['kwargs'])
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1959, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/states/file.py", line 5669, in append
    slines = slines.decode(__salt_system_encoding__)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xed in position 4: ordinal not in range(128)

local:
----------
          ID: /bugfile
    Function: file.append
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3/dist-packages/salt/state.py", line 1944, in call
                  **cdata['kwargs'])
                File "/usr/lib/python3/dist-packages/salt/loader.py", line 1959, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python3/dist-packages/salt/states/file.py", line 5669, in append
                  slines = slines.decode(__salt_system_encoding__)
              UnicodeDecodeError: 'ascii' codec can't decode byte 0xed in position 4: ordinal not in range(128)
     Started: 21:35:25.452802
    Duration: 39.99999999996362 ms
     Changes:

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:  40.000 ms

Versions Report

Salt Version:
           Salt: 2018.3.0-n/a-501caa9

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1

System Versions:
           dist: debian 9.6
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 4.9.125-linuxkit
         system: Linux
        version: debian 9.6

salt-bug.zip

[dwoz]

I've been wondering if this issue would come up. I think we need to add an encoding parameter to most of the file states.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[calmzhu]

The same problem occured in me when run state module file.replace.

Currently fix this with a walkaround, diy salt builtin utils.files.py like this

diff --git a/salt/utils/files.py b/salt/utils/files.py
index ebb9dcac..b76b6e42 100644
--- a/salt/utils/files.py
+++ b/salt/utils/files.py
@@ -391,7 +391,7 @@ def fopen(*args, **kwargs):
             if 'b' in kwargs['mode']:
                 binary = True
         if not binary:
-            kwargs['encoding'] = __salt_system_encoding__
+            kwargs['encoding'] = get_encoding(args[0])
 
     if six.PY3 and not binary and not kwargs.get('newline', None):
         kwargs['newline'] = ''

Is there any other way to do this without need to change salt builtin modules.

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[LHozzan]

Problem is still persistant for me in our development environment:

Error:

          ID: mysqlchk service definition
    Function: file.append
        Name: /etc/services
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3.6/site-packages/salt/state.py", line 1981, in call
                  **cdata['kwargs'])
                File "/usr/lib/python3.6/site-packages/salt/loader.py", line 1977, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/states/file.py", line 5553, in append
                  slines = slines.decode(__salt_system_encoding__)
              UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 627047: ordinal not in range(128)
     Started: 15:08:37.162850
    Duration: 5.621 ms
     Changes:   

State:

mysqlchk service definition:
  file.append:
    - name: /etc/services
    - text: mysqlchk        {{ salt['pillar.get']('sqldb:checkport') }}/tcp                #/opt/mysqlcheck/check.sh

Rendered data:

mysqlchk service definition:
  file.append:
    - name: /etc/services
    - text: mysqlchk        3000/tcp                #/opt/mysqlcheck/check.sh

Version info:

Salt Version:
           Salt: 3000.3
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: 4.0.5
      gitpython: Not Installed
         Jinja2: 2.11.1
        libgit2: Not Installed
       M2Crypto: 0.35.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: 1.3.12
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Apr  2 2020, 13:34:55)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 15.3.0
          smmap: 3.0.4
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.8.2003 Core
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 3.10.0-1127.19.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7.8.2003 Core

File info:

[root@1 ~]# file -i /etc/services
/etc/services: text/x-c; charset=us-ascii

Aditional info: (maybe hotfix)

• this problem appeared, when I run state via salt-call command directly on machine
• this problem disappeared, when I run state from SaltMaster server to affected machine via salt '...' command

If you need aditional information, please ask.