Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pillar include stopped working #51832

Closed
nocturo opened this issue Feb 26, 2019 · 6 comments

Comments

@nocturo
Copy link
Contributor

commented Feb 26, 2019

Description of Issue/Question

I have used git for ext_pillar for a while now and when upgrading to 2019.2.0 no pillars get rendered for me. As a quick test I downgraded to 2018.3.4 and everything works as expected.

ext_pillar config:

ext_pillar:
  - git:
    - master git@github.com:user/pillar.git:
       - root: pillar
       - privkey: /root/.ssh/id_rsa
       - pubkey: /root/.ssh/id_rsa.pub

This straight up doesn't load up the pillar top.sls if I have root: specified. If I remove the option and put top.sls in git "root" it then loads it and I get error that now it can't find the SLS (as expected since the path is not relative anymore to the specified root in the config which I've been using).

Steps to Reproduce Issue

Configure exit_pillar git pillar with root: and top.sls in the root.

Versions Report

           Salt: 2019.2.0

Dependency Versions:
           cffi: 1.9.1
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.6
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.2
         Python: 3.4.9 (default, Feb  5 2019, 14:36:09)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.1.3.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core
@Ch3LL

This comment has been minimized.

Copy link
Contributor

commented Feb 26, 2019

im not able to replicate this with:

ext_pillar:                                                                                                                                                                                          
  - git:                                                                                                                                                                                             
    - master https://github.com/Ch3LL/Ch3LLScripts.git:                                                                                                                                              
      - root: salt/pillars

although i did have to patch mine with https://github.com/saltstack/salt/pull/51304/files but thats because i have a newer pygit version then you do.

Do you see any error's or further information in the debug logs when this occurs?

@Ch3LL Ch3LL added this to the Blocked milestone Feb 26, 2019
@nocturo

This comment has been minimized.

Copy link
Contributor Author

commented Feb 26, 2019

OK I was vague as it seems more involved than I first thought. I have some include:s happening in the pillar and it looks like this: https://github.com/nocturo/pillar

If you put that as your ext_pillar, with root:

ext_pillar:
  - git:
    - master https://github.com/nocturo/pillar.git
      - root: pillar

and that's the root cause of this. I've just initially assumed since it errored out due to SLS path when trying top.sls directly at top level that it was working, but it doesn't on that repo with or without root when include: is called to call a pillar.

--- edit
Actually this is not related to git at all, I've just tried cloning that repo into /srv and using pillar_roots:

pillar_roots:
  base:
    - /srv/pillar

and it still doesn't render, so the include with init.sls seems broken?

2019-02-26 23:00:41,597 [salt.template    :120 ][DEBUG   ][25064] Rendered data from file: /srv/pillar/top.sls:
base:
  '*':
    - pillar/test/roottestdir

2019-02-26 23:00:41,598 [salt.loaded.int.render.yaml:68  ][DEBUG   ][25064] Results of YAML rendering:
OrderedDict([('base', OrderedDict([('*', ['pillar/test/roottestdir'])]))])
2019-02-26 23:00:41,608 [salt.template    :59  ][DEBUG   ][25064] compile template: /srv/pillar/pillar/test/roottestdir/init.sls
2019-02-26 23:00:41,608 [salt.utils.jinja :70  ][DEBUG   ][25064] Jinja search path: ['/srv/pillar']
2019-02-26 23:00:41,624 [salt.template    :26  ][PROFILE ][25064] Time (in seconds) to render '/srv/pillar/pillar/test/roottestdir/init.sls' using 'jinja' renderer: 0.015855073928833008
2019-02-26 23:00:41,625 [salt.template    :120 ][DEBUG   ][25064] Rendered data from file: /srv/pillar/pillar/test/roottestdir/init.sls:
include:
  - pillar/test/test

2019-02-26 23:00:41,625 [salt.loaded.int.render.yaml:68  ][DEBUG   ][25064] Results of YAML rendering:
OrderedDict([('include', ['pillar/test/test'])])
2019-02-26 23:00:41,626 [salt.template    :26  ][PROFILE ][25064] Time (in seconds) to render '/srv/pillar/pillar/test/roottestdir/init.sls' using 'yaml' renderer: 0.0004544258117675781
@nocturo nocturo changed the title top.sls not loading with ext_pillar gitfs specified root path pillar include stopped working Feb 26, 2019
@Ch3LL

This comment has been minimized.

Copy link
Contributor

commented Feb 28, 2019

thanks for the additional information. looks like i was able to replicate this and bisect it to fbab73a

which is this PR: #45269

also here is a docker container to quickly replicate the issue:

  1. docker run -it -v ~/git/salt/:/testing/ ch3ll/issues:51832 (where ~/git/salt is a local cloned git repo of salt)
  2. salt-master -d; salt-minion -d
  3. salt '*' pillar.items (and it will be empty on 2019.2.0 and return pillar on 2018.3.3)
@Ch3LL

This comment has been minimized.

Copy link
Contributor

commented Feb 28, 2019

I realized that the reason its failing is because of this:

include:
  - pillar/test/test

if you change that to:

include:
  - pillar.test.test

it will work. Since we were previously supporting the / notation we still need to get this fixed up still though

@nocturo

This comment has been minimized.

Copy link
Contributor Author

commented Mar 1, 2019

Ok, thanks for the insight, not that of a big deal to change to keep using fluorine. Thanks!

@dmurphy18

This comment has been minimized.

Copy link
Contributor

commented Mar 7, 2019

Actually this is easy to reproduce and it is breaking the nightly builds when upgraded to 2019.2.0 salt-master.
root@c7master:/usr/lib/python2.7/site-packages/salt# cat /srv/pillar/pkgbuild.sls

set version to build

{% set build_version = '2019_2_0' %}

{% if build_version != '' %}
include:
- .versions.{{build_version}}.pkgbuild
{% endif %}

Problem is that it is not recursively processing the jinja2 derived include file .versions.2019_2_0.pkgbuild

area is in render_pstate:
'''
matched_pstates = []
for sub_sls in state.pop('include'):
if isinstance(sub_sls, dict):
sub_sls, v = next(six.iteritems(sub_sls))
defaults = v.get('defaults', {})
key = v.get('key', None)
else:
key = None

                        try:
                            matched_pstates += fnmatch.filter(self.avail[saltenv], sub_sls)
                        except KeyError:
                            errors.extend(
                                ['No matching pillar environment for environment '
                                 '\'{0}\' found'.format(saltenv)]
                            )

                    for sub_sls in set(matched_pstates):

'''
The matched_pstates += fnmatch.filter(self.avail[saltenv], sub_sls) will come back empty

salt-call --local pillar.items

when minion installed on same machine as master

@KChandrashekhar KChandrashekhar added this to In Progress in Salt Core Workspace Mar 18, 2019
@KChandrashekhar KChandrashekhar moved this from In Progress to Pending Verification in Salt Core Workspace Mar 18, 2019
@KChandrashekhar KChandrashekhar added this to Pending Verification in Salt Open Workspace Mar 18, 2019
@KChandrashekhar KChandrashekhar modified the milestones: March 29th, Priority Apr 5, 2019
@waynew waynew closed this Jul 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Salt Core Workspace
Pending Verification
Salt Open Workspace
  
Pending Verification
5 participants
You can’t perform that action at this time.