Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

salt file.contains module will throw when attempting to parse utf-8 encoded files with BOM #54357

Open
untra opened this issue Aug 30, 2019 · 1 comment

Comments

@untra
Copy link

commented Aug 30, 2019

Description of Issue

  • tldr 2018.3 windows minion + adversarial file + sudo salt 'win' file.contains c:\bom.txt 'BOM' = UnicodeDecodeError

Original description from here

馃憢 Heya folks encountering the UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)

We have encountered this issue when using the salt file.contains module to validate settings in configuration files we thought were appropriately ascii / utf-8 encoded.

The problem turned out to be a utf-8 byte-order-marker at the start of the file, which is a three byte EF BB BF sequence which is clearly outside the ascii range. This invisible byte triplet is hidden in files viewed in windows notepad, and can cause headaches with salt blowing up unable to handle a seemingly reasonable ascii / utf-8 encoded files.

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

  • 2018.3 master on linux
  • 2018.3 windows minion ('win')
  • The adversarial file, placed in c:\bom.txt when opened and viewed in windows notepad will look like this:
THIS FILE IS DA BOM

when copied to a linux machine and opened with vim we saw (actual contents):

茂禄驴THIS FILE IS DA BOM

and then viewed with a hex editor the file would reveal

EF BB BF 54 48 49 53 20 46 49 4c 45 20 49 53 20 44 41 20 42 4f 4d

Steps to Reproduce Issue

  • command to run on the master: sudo salt 'win' file.contains c:\bom.txt 'BOM'
  • Expected result: True
  • Actual outcome: UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)

Versions Report

sorry will get a proper readout to you later here ya go

Salt Version:
           Salt: 2018.3.4

Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Nov  6 2016, 00:28:07)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.3.1611 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-514.26.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.3.1611 Core
@Ch3LL

This comment has been minimized.

Copy link
Contributor

commented Aug 30, 2019

ping @saltstack/team-windows any ideas here?

@Ch3LL Ch3LL added this to the Blocked milestone Aug 30, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can鈥檛 perform that action at this time.