external git pillar terrible performance.

[hunkeelin]

Description of Issue/Question

Simple changes in pillar structure takes up to 2 minutes to update using pillar.ext_pillar. Where does, simply specifying file_roots with a desinate pillar directory is instant.

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

Steps to Reproduce Issue

external pillar setup in /etc/salt/master

fingerprint_hash_type: sha512
worker_threads: 15
auto_accept: False
autosign_timeout: 120
autosign_file: /etc/salt/autosign.conf
file_roots:
    base:
        - /srv/salt/
fileserver_backend:
    - roots
    - git

gitfs_provider: gitpython
gitfs_remotes:
    - root@git:/data/salt-formula.git
gitfs_env_whitelist:
    - base
ext_pillar:
    - git:
        - master root@git:/data/salt-pillar.git
pillar_merge_lists: True
reactor:
    - 'salt/fileserver/gitfs/update':
        - /srv/reactor/update_fileserver.sls
state_verbose: False

Inside git:/data/salt-pillar.git/hooks/post-receive

#!/bin/bash
# hook trigger
sudo salt-call event.fire_master update salt/fileserver/gitfs/update

cat /srv/reactor/update_fileserver.sls

update_fileserver:
    runner.fileserver.update

Simple pillar changes like changing a name or correcting a typo then git push will take up to 2 minutes to generate. Where does if I replace:

ext_pillar:
    - git:
        - master root@git:/data/salt-pillar.git

with

pillar_roots:
    base:
        - /data/pillar

and simply do pillar in the traditional way, changes are instant.

Versions Report

salt 2017.7.2 (Nitrogen)

[gtmanfred]

I do not believe that the fileserver update triggers the git_pillar to update. I believe it only updates the fileserver.

@terminalmage is this correct? Is there anything out there specifically that updates the pillar git bare clone immediately?

Thanks,
Daniel

[hunkeelin]

@gtmanfred It doesn't matter what's in my githook because when I run pillar I made sure I ran. salt '*' saltutil.refresh_pillar. Even after that command it still takes 2 minutes.

[gtmanfred]

Refresh pillar just refreshes it for the minion which would require that the master updates the bare git checkout of the git pillar repository.

If that hasn't happened yet (and I am not sure it is triggered by updating the fileserver, because that just updates the gitfs_remotes afaik) then the pillar_refresh won't do anything.

[hunkeelin]

Then what is the correct command to refresh pillar from gitfs_remotes?

…

On Mon, Dec 4, 2017 at 8:57 AM, Daniel Wallace ***@***.***> wrote: Refresh pillar just refreshes it for the minion which would require that the master updates the bare git checkout of the git pillar repository. If that hasn't happened yet (and I am not sure it is triggered by updating the fileserver, because that just updates the gitfs_remotes afaik) then the pillar_refresh won't do anything. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#44791 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGPPe_Yno5F1mv5OdNFQN2EmrdhQanPDks5s9CSMgaJpZM4Qy6Us> .

[gtmanfred]

Looks like you can use this

https://docs.saltstack.com/en/latest/ref/runners/all/salt.runners.git_pillar.html#salt.runners.git_pillar.update

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.