Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2019.2.1: Jinja from import broken #54765

Closed
awerner opened this issue Sep 26, 2019 · 8 comments

Comments

@awerner
Copy link
Contributor

commented Sep 26, 2019

Description of Issue

Starting with salt version 2019.2.1, the following statement is broken:

{% from "formula/map.jinja" import defaults with context %}

This is the error that occurs:

           Rendering SLS 'base:formula' failed: Jinja error: formula/map.jinja
       Traceback (most recent call last):
         File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 394, in render_jinja_tmpl
           output = template.render(**decoded_context)
         File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
           return self.environment.handle_exception(exc_info, True)
         File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
           reraise(exc_type, exc_value, tb)
         File "<template>", line 1, in top-level template code
         File "/usr/lib/python2.7/site-packages/salt/utils/jinja.py", line 189, in get_source
           raise TemplateNotFound(template)
       TemplateNotFound: formula/map.jinja
       
       ; line 1
       
       ---
       {% from "formula/map.jinja" import defaults with context %}    <======================

The same setup is working with 2019.2.0.

Setup

formula/map.jinja:

{% set defaults = {'foo': 'bar'} %}

formula/init.sls:

{% from "formula/map.jinja" import defaults with context %}
salt-call --local state.apply formula

Steps to Reproduce Issue

This happens for us using our kitchen-salt based setup. So only salt-call and salt-minion are installed in the vms.

Versions Report

Salt Version:
           Salt: 2019.2.1
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.6.1810 Core
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 3.10.0-957.1.3.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

Also happens with Debian 8, Debian 9, Ubuntu 16.04 LTS, Ubuntu 18.04 LTS.

@lomeroe

This comment has been minimized.

Copy link
Contributor

commented Sep 26, 2019

seems to be from the changes in #53563

If I revert the "file_client" function in salt/utils/jinja.py back to the state it was in in 2019.2.0, things start working again for me.

It seems that in my case, when file caching fails, the wrong fileclient is being returned.

Changing the function to the following to add more logging:

    def file_client_new(self):
        '''
        Return a file client. Instantiates on first call.
        '''
        # If there was no file_client passed to the class, create a cache_client
        # and use that. This avoids opening a new file_client every time this
        # class is instantiated
        log.debug('ELM - self._file_client == {}'.format(self._file_client))
        if self._file_client is None:
            log.debug('ELM - SaltCacheLoader._cached_client == {}'.format(SaltCacheLoader._cached_client))
            if not SaltCacheLoader._cached_client:
                SaltCacheLoader._cached_client = salt.fileclient.get_file_client(
                    self.opts, self.pillar_rend)
            self._file_client = SaltCacheLoader._cached_client
        log.debug('ELM - returning self._file_client == {}'.format(self._file_client))
        return self._file_client

Then returns

[DEBUG   ] ELM - self._file_client == None
[DEBUG   ] ELM - SaltCacheLoader._cached_client == <salt.fileclient.PillarClient object at 0x7f80f8a01d10>
[DEBUG   ] ELM - returning self._file_client == <salt.fileclient.PillarClient object at 0x7f80f8a01d10>
[DEBUG   ] ELM self.cached == []
[DEBUG   ] ELM - caching template path/to/defaults.yml
[DEBUG   ] ELM - saltpath == salt://path/to/defaults.yml
[DEBUG   ] ELM - self._file_client == <salt.fileclient.PillarClient object at 0x7f80f8a01d10>
[DEBUG   ] ELM - returning self._file_client == <salt.fileclient.PillarClient object at 0x7f80f8a01d10>
[DEBUG   ] ELM - file should be cached
[DEBUG   ] ELM - filepath == /var/cache/salt/minion/files/base/path/to/defaults.yml
[DEBUG   ] Jinja Error

Adding similar debug logging to the "old" function retuns a file client of <salt.fileclient.FSClient object at 0x7f02ee4a5c50>

Seems the PillarClient object is being loaded when the pillar data is rendered (my example is a masterless minion) and then is not being overwritten with a FSClient object to get the necessary files from the fileserver...that's just a guess though

ping @twangboy

@kiemlicz

This comment has been minimized.

Copy link
Contributor

commented Sep 27, 2019

Did someone found some workaround?
It disables the whole release for me... and it's the only release for Debian 10

@awerner

This comment has been minimized.

Copy link
Contributor Author

commented Sep 27, 2019

@kiemlicz For other distri we pin to 2019.2.0 right now..
Have you tried to use the Debian 9 repo for 2019.2.0 with Debian 10?

@kiemlicz

This comment has been minimized.

Copy link
Contributor

commented Sep 27, 2019

Based on #52590
It is impossible to do so.

I remember trying before 2019.2.1 release without any success. Why should it change now?

myii added a commit to myii/ssf-formula that referenced this issue Oct 1, 2019
* The current pre-salted `opensuse` images are using `mirror.23media.com`
  - https://github.com/netmanagers/salt-image-builder/blob/d6e57f6b22570530a627c89a94fed02754a3197d/scripts/prepare.sh#L99
  - This has been proving to be unreliable in recent days, causing failed builds
* Preparing new pre-salted images would pull in `2019.2.1`, which is a no-go
  - netmanagers/salt-image-builder#7
  - saltstack/salt#54765
* This workaround is to use bootstrapped images in the interim
@3issamben

This comment has been minimized.

Copy link

commented Oct 4, 2019

I can also confirm that I'm seeing the exact same issue (Salt 2019.2.1, CentOS 7.5 with Python 2.7)

syphernl added a commit to syphernl/mariadb-formula that referenced this issue Oct 7, 2019
See saltstack/salt#54765 for context
syphernl added a commit to syphernl/nginx-formula that referenced this issue Oct 7, 2019
See saltstack/salt#54765 for context
@OrangeDog

This comment has been minimized.

Copy link
Contributor

commented Oct 16, 2019

In contrast, I'm not seeing this issue.

Salt Version:
           Salt: 2019.2.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.26.0
        libnacl: Not Installed
       M2Crypto: 0.32.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.2
         Python: 3.6.8 (default, Oct  7 2019, 12:59:55)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: ISO-8859-1
        machine: x86_64
        release: 4.15.0-65-generic
         system: Linux
        version: Ubuntu 18.04 bionic
@cachedout

This comment has been minimized.

Copy link
Collaborator

commented Oct 21, 2019

Should this be closed now that #54780 is closed?

@KChandrashekhar

This comment has been minimized.

Copy link

commented Oct 21, 2019

Closing this as #54780 fixes the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.