LocalClient file cache confuse pillar and state files

[bersace]

Hi,

I hit a bug when using extends in top.sls.

Here is the settings :

# salt-call --versions
           Salt: 2014.7.4-199-g3c91459
         Python: 2.7.3 (default, Mar 13 2014, 11:03:55)
         Jinja2: 2.7.4-dev
       M2Crypto: Not Installed
 msgpack-python: 0.4.6
   msgpack-pure: Not Installed
       pycrypto: Not Installed
        libnacl: Not Installed
         PyYAML: 3.10
          ioflo: Not Installed
          PyZMQ: 14.5.0
           RAET: Not Installed
            ZMQ: 4.0.5
           Mako: 0.7.0
# cat /etc/salt/minion.d/00-masterless.conf
environment: base
file_client: local
fileserver_backend:
  - roots

file_roots:
  base:
    - /srv/salt/

log_level: debug

pillar_roots:
  base:
    - /srv/pillar/
$ tree /srv
.
├── pillar
│   ├── lib.sls
│   └── top.sls
└── salt
    ├── lib.sls
    └── top.sls
$ cat pillar/lib.sls 
# PILLAR
{{ env }}:

{%- block all %}
# pillar/lib.sls
  '*':
    - pillar-lib
{%- endblock %}
$ cat pillar/top.sls 
{% extends 'lib.sls' %}

{% block all -%}
{{ super() }}
    - pillar-top
{% endblock -%}
$ cat salt/lib.sls 
# SALT
{{ env }}:

{%- block all %}
# salt/lib.sls
  '*':
    - salt-lib
{%- endblock %}
$ cat salt/top.sls 
{% extends 'lib.sls' %}

{% block all -%}
{{ super() }}
    - salt-top
{% endblock -%}

Here is the result of show_top:

# salt-call state.show_top
local:
    ----------
    base:
        - salt-lib
        - pillar-top

I expect :

local:
    ----------
    base:
        - salt-lib
        - salt-top

It looks like jinja block all of pillar/top.sls is used instead of salt/top.sls :(

I investigate a bit and found that local client seems guilty. get_file('salt://top.sls) returns /srv/pillar/top.sls when searching instead of /srv/salt/top.sls :(

I keep debugging this and try do propose a fix. Do you have a clue ?

This is also broken in develop.

Regards,
Étienne

[bersace]

Here is a custom trace :

...
> /usr/local/src/salt/salt/state.py(2234)get_tops()
   2233             tops[self.opts['environment']] = [
-> 2234                     compile_template(
   2235                         self.client.cache_file(

ipdb> c
ROOTS {'base': ['/srv/salt/']} /var/cache/salt/minion/files/states/base/top.sls
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://top.sls'
CUSTOM find_file {'base': ['/srv/pillar/']} top.sls
ROOTS {'base': ['/srv/salt/']} /var/cache/salt/minion/files/states/base/top.sls
CUSTOM find_file {'base': ['/srv/pillar/']} top.sls
[INFO    ] Fetching file from saltenv 'base', ** done ** 'top.sls'
[DEBUG   ] Jinja search path: ['/var/cache/salt/minion/files/states/base']
CUSTOM load fileserver {'base': ['/srv/salt/']}
[DEBUG   ] Updating roots fileserver cache
CUSTOM find_file {'base': ['/srv/salt/']} lib.sls
CUSTOM find_file {'base': ['/srv/salt/']} top.sls
CUSTOM get_file_client {'base': ['/srv/salt/']} False <salt.fileclient.FSClient object at 0x499e090>
ROOTS {'base': ['/srv/salt/']} /var/cache/salt/minion/files/states/base/lib.sls
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://lib.sls'
CUSTOM find_file {'base': ['/srv/salt/']} lib.sls
ROOTS {'base': ['/srv/salt/']} /var/cache/salt/minion/files/states/base/lib.sls
CUSTOM find_file {'base': ['/srv/salt/']} lib.sls
[INFO    ] Fetching file from saltenv 'base', ** done ** 'lib.sls'
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/states/base/top.sls:
# SALT
base:
# salt/lib.sls
  '*':
    - salt-lib
    - pillar-top

[bersace]

There is a side effect in Highstate class. self.client is modified to get pillars, but not restored before fetching top.sls.

[bersace]

The problem is here :

ipdb> self.client.channel.fs.servers['roots.serve_file'].func_globals['__opts__']['file_roots']         
{'base': ['/srv/pillar/']}     
ipdb> self.client.channel.fs.opts['file_roots']
{'base': ['/srv/salt/']}

The bug is that fileservers are loaded with pillar settings.
:-)

[bersace]

Jinja reinstanciate it's client. This is why lib.sls is correct.

[bersace]

The line https://github.com/saltstack/salt/blob/2014.7/salt/state.py#L2982 load fileserver with pillar file_roots. This break next call to fileserver.

[bersace]

This line : https://github.com/saltstack/salt/blob/2014.7/salt/pillar/__init__.py#L116

[jfindlay]

@bersace, thanks for working on this.

[bersace]

The bug is also in 2015.2. We should port the fix. I'm not planning on it right now.

[bersace]

Looks like the bug is still present with file_client: remote. I'm trying to fix this right now.

[bersace]

That's really strange. When minion queries state lib.sls, i see on minion :

[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://lib.sls'
*** CUSTOM *** {'path': 'lib.sls', 'saltenv': 'base', 'cmd': '_serve_file', 'loc': 0}
*** CUSTOM *** {'path': 'lib.sls', 'saltenv': 'base', 'cmd': '_serve_file', 'loc': 69}
[INFO    ] Fetching file from saltenv 'base', ** done ** 'lib.sls'
{{ env }}:

{%- block all %}
  '*':
    - pillar-lib
{%- endblock %}

And on master

[INFO    ] AES payload received with command _serve_file
*** CUSTOM *** serve_file /usr/local/src/salt-local-bug/pillar/lib.sls 0
{{ env }}:

{%- block all %}
  '*':
    - pillar-lib
{%- endblock %}

EOF
[INFO    ] AES payload received with command _serve_file
*** CUSTOM *** serve_file /usr/local/src/salt-local-bug/salt/lib.sls 69

EOF

Why the hell does it read two different files ? :/ Looks like a side effect. Pillar caching is really broken :(

[bersace]

That global __opts__ is sometimes messed :(

[bersace]

Ok, the fileserver is different from time to time :( I don't know where is the fileserver targetting.

[bersace]

It looks like some workers does not have the same fileserver settings.

[eliasp]

@bersace is this possibly fixed by #27933?

[bersace]

I switched to Ansible.