custom modules won't sync on gitfs only configuration

[MarshM4ll0w]

I see that this has been discussed a lot over the years, but I have the same issue and I can't figure out how to overcome it. In #15210 @basepi said:

I think this is already supported via gitfs. Just put _modules/ and _states/ (etc) directories inside of your git repository, and they will get synced.

I cannot get that to work. I feel like I have tried everything. My salt setup is a bit...unorthodox, but I feel like I am following everything the documentation says is necessary. I am using gitfs exclusively (except for pillars) and I have several gitfs_remotes. I have repos set up for each node group so the admins with access to their repo can't step on the toes or see the code of another group. I also have a repo used exclusively for the top file and another for common states that will be used by multiple, if not all, node groups. I don't know if that is relevant information or not.

In my common repo I have several states and they work perfectly, but I also have a folder named _modules with my custom modules in it. When I run salt '*' cp. list_master I can see the modules directory and the module files.

[me@mymachine roots]# salt minion1 cp.list_master
local:
    - common/_modules/hello.py
    - common/_modules/passmod.py
    - common/init.sls
    - common/roar.sls
    - common/lemons.sls
    - a/init.sls 
    - b/init.sls
    - c/files/configile.conf
    - c/init.sls
    - top.sls

I renamed some of those to mask their use, but regardless all of the states work and I can see proof that the _modules folder is being pulled. However sync_all produces evidence that salt has no idea that these modules exist.

[me@mymachine roots]# salt minion1 saltutil.sync_all
minion1:
    ----------
    beacons:
    grains:
    log_handlers:
    modules:
    output:
    renderers:
    returners:
    states:
    utils:

salt minion1 sys.list_modules and salt minion1 sys.list_functions don't show their existence either. I've gone through the process of deleting /var/cache/salt/master/gitfs many times to no avail.

My master config is as follows:

[me@mymachine roots]# grep "^[^#]" /etc/salt/master
id: mymaster
worker_threads: 10
file_roots:
  base:
    - /srv/salt/
fileserver_backend:
  - git
gitfs_provider: pygit2
gitfs_env_whitelist:
  - base
gitfs_remotes:
  - https://user:pass@mygitserver.com/Salt/top.git
  - https://user:pass@mygitserver.com/Salt/common.git:
    - mountpoint: salt://common
  - https://user:pass@mygitserver.com/Salt/a.git:
    - mountpoint: salt://a
  - https://user:pass@mygitserver.com/Salt/b.git:
    - mountpoint: salt://b
  - https://user:pass@mygitserver.com/Salt/c.git:
    - mountpoint: salt://c
pillar_roots:
  base:
    - /srv/pillars
    - /srv/secret
log_level_logfile: debug

I have run out of ideas so if anyone has a suggestion I would love to hear it. It is important that I keep this repo setup and I need to be able to create custom modules in each repo. Common is the only one with modules in it now, but in the future they probably all will.

[basepi]

I think the mountpoints are throwing it off. the _modules and _states directories must be in the root of your fileserver, if I remember correctly. We don't recurse through looking for those directories.

[MarshM4ll0w]

Thanks for your response!

That makes perfect sense, but will make my setup impossible. I will put the _modules directory into the top repo as a test. That will place it at the root. Long term though I'm not sure how to proceed. Perhaps I should set up each repo as such:

mygitserver.com/Salt/a.git
  _modules
    mymodule.py
  _states
    mystate.py
  a
    init.sls

Then I could remove the mountpoints and the states would still be accessible via common or common.roar. This introduces the issue of duplicate module names though. If there was a mytest.py in 2 different repos I assume it would cause a problem.

[basepi]

Technically I don't think it would. Pretty sure the minion would take whichever one it found first.

I'll have to think more on this problem. We would potentially add some logic for checking mountpoints, but that's pretty hacky, as the fileserver backend is supposed to be invisible as far as the fileclient is concerned....

[denis-kalinin]

@MarshM4ll0w is right - for some reason declaring root in gitfs_remotes is not working - you have to avoid using root attribute for remote git with custom salt's modules and states.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[tacerus]

Isn't this a bug? Is there a different way to use root together with a _modules directory?