Using six.u() function doesn't work if package name is str and not 'u'

[rallytime]

Description of Issue/Question

When investigating the fix for #33605, it because apparent that if a package name with a non-ascii character in it is passed via a state, the state compiler will stack trace and complain about unicode.

Normally the fix for such a situation is to use the six.u() function to make sure the unicode string is compatible across Python versions. However, in this case, the package name, even though it contains unicode characters, is being passed as a normal string rather than a unicode string, so using six.u() has no effect and the stack trace is still present.

The immediate issue with the Windows package was resolved with the fix in #33670, but there is a greater underlying issue and this fix was just a band-aide. This issue is to track the greater issue revolving around the state name and the old or new package names that are passed to the state compiler from the pkg modules.

Setup

The following state, while it will return False as the package isn't a valid name, should get passed the state compiler. Instead it stacktraces inside the state compiler:

# /srv/salt/non-ascii.sls
Français:
  pkg.installed

Here is a simple state that doesn't contain non-ascii chars for testing purposes:

# /srv/salt/ascii.sls
not-a-package:
  pkg.installed

Steps to Reproduce Issue

Run the state above with salt-call --local state.sls non-ascii:

root@rallytime:~# salt-call --local state.sls non-ascii
[INFO    ] Loading fresh modules for state activity
[INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://non-ascii.sls'
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe7' in position 4: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/root/SaltStack/salt/salt/scripts.py", line 349, in salt_call
    client.run()
  File "/root/SaltStack/salt/salt/cli/call.py", line 58, in run
    caller.run()
  File "/root/SaltStack/salt/salt/cli/caller.py", line 134, in run
    ret = self.call()
  File "/root/SaltStack/salt/salt/cli/caller.py", line 197, in call
    ret['return'] = func(*args, **kwargs)
  File "/root/SaltStack/salt/salt/modules/state.py", line 950, in sls
    ret = st_.state.call_high(high_)
  File "/root/SaltStack/salt/salt/state.py", line 2214, in call_high
    chunks = self.compile_high_data(high)
  File "/root/SaltStack/salt/salt/state.py", line 1290, in compile_high_data
    chunks = self.order_chunks(chunks)
  File "/root/SaltStack/salt/salt/state.py", line 1229, in order_chunks
    chunks.sort(key=lambda chunk: (chunk['order'], '{0[state]}{0[name]}{0[fun]}'.format(chunk)))
  File "/root/SaltStack/salt/salt/state.py", line 1229, in <lambda>
    chunks.sort(key=lambda chunk: (chunk['order'], '{0[state]}{0[name]}{0[fun]}'.format(chunk)))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe7' in position 4: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/root/SaltStack/salt/salt/scripts.py", line 349, in salt_call
    client.run()
  File "/root/SaltStack/salt/salt/cli/call.py", line 58, in run
    caller.run()
  File "/root/SaltStack/salt/salt/cli/caller.py", line 134, in run
    ret = self.call()
  File "/root/SaltStack/salt/salt/cli/caller.py", line 197, in call
    ret['return'] = func(*args, **kwargs)
  File "/root/SaltStack/salt/salt/modules/state.py", line 950, in sls
    ret = st_.state.call_high(high_)
  File "/root/SaltStack/salt/salt/state.py", line 2214, in call_high
    chunks = self.compile_high_data(high)
  File "/root/SaltStack/salt/salt/state.py", line 1290, in compile_high_data
    chunks = self.order_chunks(chunks)
  File "/root/SaltStack/salt/salt/state.py", line 1229, in order_chunks
    chunks.sort(key=lambda chunk: (chunk['order'], '{0[state]}{0[name]}{0[fun]}'.format(chunk)))
  File "/root/SaltStack/salt/salt/state.py", line 1229, in <lambda>
    chunks.sort(key=lambda chunk: (chunk['order'], '{0[state]}{0[name]}{0[fun]}'.format(chunk)))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe7' in position 4: ordinal not in range(128)

Here's what the second state looks like when a state doesn't exist, but there are no non-ascii chars (on Ubuntu):

root@rallytime:~# salt-call --local state.sls ascii
<snipped>
[INFO    ] Completed state [not-a-package] at time 20:02:04.480015 duration_in_ms=1726.965
local:
----------
          ID: not-a-package
    Function: pkg.installed
      Result: False
     Comment: Problem encountered installing package(s). Additional info follows:

              errors:
                  - E: Unable to locate package not-a-package
     Started: 20:02:02.753050
    Duration: 1726.965 ms
     Changes:

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

root@rallytime:~# salt-call --versions
Salt Version:
           Salt: 2016.3.0

Dependency Versions:
           cffi: 0.8.6
       cherrypy: 5.0.1
       dateutil: 2.4.2
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 4.4.0-x86_64-linode63
         system: Linux
        version: Ubuntu 14.04 trusty

ping @s0undt3ch

[s0undt3ch]

The way to address this and future Unicode issues is to turn into Unicode any data salt receives externally and only use Unicode internally.

[damon-atkins]

This why I added py3 and unicode handling into module.reg so that win pkg can work correctly. But it sounds like their is more to it. It outputs only unicode but excepts both str and unicode as input. Until every thing which calls it is converted to unicode.

[cachedout]

I agree with @damon-atkins and @s0undt3ch here. Salt should be Unicode-everywhere in the core, especially in the state compiler.

[damon-atkins]

This is my post from the PY3 thread

"from __future__ import unicode_literals" or "from __future__ import absolute_import, unicode_literals"
I have been slowing adding this to windows, since much of Windows is Unicode
It needs to be added to all *.py files to help with porting. Can it be added post Carbon Release RC1 to develop?
FYI https://docs.djangoproject.com/en/1.10/topics/python3/

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[rallytime]

Oh, I missed this the other day with the stale notification, but this should be completely resolved with all of our unicode work for 2018.3.0.