Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File Tree external pillar fails to render templates in python 3 #49927

Open
thetaurean opened this issue Oct 8, 2018 · 3 comments

Comments

Projects
None yet
2 participants
@thetaurean
Copy link
Contributor

commented Oct 8, 2018

Description of Issue/Question

The File Tree external pillar fails to render templates in python 3. The root cause is the file_tree module passes bytes to the template renderer instead of a string.

passes bytes instead of string to SLS_ENCODER.

Setup

In your salt master config

  - file_tree:
      root_dir: /srv/pillar
      render_default: yaml
      renderer_whitelist:
        - yaml
      template: True

Steps to Reproduce Issue

Simple add a properly formatted yaml hosts file to /srv/pillar/hosts/minion_id and try to refresh pillars.
You will see that salt fails with the following error.

2018-10-08 03:51:39,344 [salt.pillar      :972 ][ERROR   ][11949] Execption caught loading ext_pillar 'file_tree':
  File "/usr/lib/python3/dist-packages/salt/pillar/__init__.py", line 962, in ext_pillar
    key)
  File "/usr/lib/python3/dist-packages/salt/pillar/__init__.py", line 881, in _external_pillar_data
    ext = self.ext_pillars[key](self.minion_id, pillar, **val)
  File "/usr/lib/python3/dist-packages/salt/pillar/file_tree.py", line 443, in ext_pillar
    template)
  File "/usr/lib/python3/dist-packages/salt/pillar/file_tree.py", line 536, in _ext_pillar
    template)
  File "/usr/lib/python3/dist-packages/salt/pillar/file_tree.py", line 258, in _construct_pillar
    whitelist=renderer_whitelist)
  File "/usr/lib/python3/dist-packages/salt/template.py", line 142, in compile_template_str
    ofile.write(SLS_ENCODER(template)[0])
2018-10-08 03:51:39,345 [salt.pillar      :1019][CRITICAL][11949] Pillar render error: Failed to load ext_pillar file_tree: Can't convert 'bytes' object to str implicitly
2018-10-08 03:51:43,537 [salt.utils.parsers:1045][WARNING ][11928] Master received a SIGTERM. Exiting.

Changing lines 141-142 in template.py as follows is a temporary fix

with salt.utils.files.fopen(fn_, 'wb') as ofile:
        ofile.write(SLS_ENCODER(template.decode('utf-8'))[0])

The more appropriate fix probably belongs in file_tree.py:

else:
                data = contents # decode here
                if template is True:
                    data = salt.template.compile_template_str(template=contents,
                                                              renderers=renderers,
                                                              default=render_default,
                                                              blacklist=renderer_blacklist,
                                                              whitelist=renderer_whitelist)

Suggest using salt.utils.stringutils.to_str

Versions Report

Salt Version:
Salt: 2018.3.2

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.4.2
docker-py: Not Installed
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.5.2 (default, Nov 23 2017, 16:37:01)
python-gnupg: 0.3.8
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: Ubuntu 16.04 xenial
locale: UTF-8
machine: x86_64
release: 4.4.0-87-generic
system: Linux
version: Ubuntu 16.04 xenial

@garethgreenaway

This comment has been minimized.

Copy link
Member

commented Oct 8, 2018

@thetaurean Thanks for the report. Looks like this should be a fairly easy fix, would you be able to make the change and submit a PR?
@terminalmage FYI.

@thetaurean

This comment has been minimized.

Copy link
Contributor Author

commented Oct 8, 2018

I'd be honored 🥇 fork, branch, and PR from my fork yes?

@garethgreenaway

This comment has been minimized.

Copy link
Member

commented Oct 8, 2018

@thetaurean Yes please. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.