Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

salt-call --local with gitfs #6660

Closed
aboe76 opened this issue Aug 13, 2013 · 65 comments
Closed

salt-call --local with gitfs #6660

aboe76 opened this issue Aug 13, 2013 · 65 comments
Assignees
Labels
Feature new functionality including changes to functionality and code refactors, etc.
Milestone

Comments

@aboe76
Copy link
Contributor

aboe76 commented Aug 13, 2013

I would like to propose the gitfs support for salt-call.

This would enable a lot of cool stuff: like easier deployment from saltstack-formulas
using docker.io or packer.io without the overhead of a complete salt-master installation.

Requirements as I think of is that salt-call needs to parse /etc/salt/master.d/ or
/etc/salt/minion.d/ files for correct information about what gitfs remotes there are..

@basepi
Copy link
Contributor

basepi commented Aug 13, 2013

This would be cool, thanks for the suggestion!

@mafrosis
Copy link
Contributor

+1 for this. I started looking at it today - but have the feeling I'll do more harm than good hacking it in! Pointers on where to start are very welcome.

@ruimarinho
Copy link

I was about to write about this, but I'm glad someone found a need for it too! I genuinely thought salt would simply read a gitfs remote under /etc/salt/minion ;)

@F30
Copy link
Contributor

F30 commented Mar 5, 2014

+1

2 similar comments
@pidah
Copy link

pidah commented Mar 6, 2014

+1

@aromanovich
Copy link

+1

@ryan-lane
Copy link
Contributor

+1 I could definitely use this for vagrant + masterless salt + formulas.

@ruimarinho
Copy link

I've found that salting a salt master is actually a very good alternative to salt-call --local if you accept the overhead of running another daemon locally.

@diegows
Copy link
Contributor

diegows commented Mar 12, 2014

+1 and I agree with @ryan-lane

@JackLeo
Copy link

JackLeo commented Mar 18, 2014

+1

9 similar comments
@gitfrage
Copy link

+1

@etuttle
Copy link
Contributor

etuttle commented Mar 25, 2014

+1

@ahdinosaur
Copy link

+1

@jsuchal
Copy link

jsuchal commented Apr 2, 2014

+1

@amahon
Copy link

amahon commented Apr 24, 2014

+1

@DeTeam
Copy link

DeTeam commented May 2, 2014

+1

@matthias-ollig
Copy link

+1

@mwildehahn
Copy link

+1

@johnnadratowski
Copy link

+1

@gepoch
Copy link

gepoch commented May 20, 2014

This would also bring support for using state.orch on purely gitfs backed systems. Orchestration files in a git repo just make sense.

@basepi
Copy link
Contributor

basepi commented May 20, 2014

@gepoch What's stopping you from using gitfs with state.orch now?

@gepoch
Copy link

gepoch commented May 20, 2014

At the moment (in 2014.1.4) salt-run state.orch some.state is giving me this:

salt:
    Data failed to compile:
----------
    No matching sls found for 'some.state' in env 'base'

I can access all of the formulae in the same git repo with no problem. Looking at the source (although this may have been a few patch versions ago) seemed to suggest that it wasn't capable of using the gitfs backend, but I could definitely be wrong about that.

It should just be a matter of configuring the gits_remotes to point to the right git repo, right?

@basepi
Copy link
Contributor

basepi commented May 20, 2014

Theoretically, yes. And the base environment is automatically mapped to the master branch of your git repo.

I should note that state.orchestrate has some pretty serious deficiencies in the 2014.1 branch -- there have been a lot of fixes and additions in the develop branch, so you may find that your issue has been fixed when we put out the release candidate for the next feature release. (In a week or two)

@gepoch
Copy link

gepoch commented May 20, 2014

Yeah, I'm wise to the master-base mapping and that's already okay.

I'm guessing it's in the develop code. Anyways, thanks very much! I'm looking forward to it :)

@joehoyle
Copy link
Contributor

joehoyle commented Jun 4, 2014

+1!

@davidkarlsen
Copy link

+1! This would be great for vagrant/docker/masterless salt boxes.

@cooncesean
Copy link

@davidkarlsen @ryan-lane What's the work around you two are using for a masterless Vagrant setup? Copy/paste the needed formulae into your local environment?

@ryan-lane
Copy link
Contributor

For vagrant I used master/minion, which kind of sucks since I'm aiming for masterless in production. Note that I'm mostly using gitfs to make it easier for devs to quickly test things from formulas.

@davidkarlsen
Copy link

Both options would be possible. Either do checkouts to a local dir and include this in a masterless, or set it up as master/slave on and use gitfs. I´d probably go for the latter.

@mafrosis
Copy link
Contributor

mafrosis commented Jun 5, 2014

FWIW a work around for this in masterless setup is to use git submodules to add to the necessary states.

@plainlystated
Copy link

+1 agree with @cackovic ... I just wasted an hour trying to get this to work, because the docs say it does.

@basepi
Copy link
Contributor

basepi commented Sep 2, 2014

@plainlystated Sorry! I created #15444 to solve this doc issue for the time being.

@plainlystated
Copy link

@basepi sweet thanks!

@terminalmage
Copy link
Contributor

This has to do with the fact that the LocalClient does not use the salt.fileserver.Fileserver class to allow for the use of the modular fileserver. It hard-codes the use of file_roots.

This means that none of the modular backends work, not just gitfs.

I will look into this.

@terminalmage terminalmage self-assigned this Sep 4, 2014
This was referenced Sep 19, 2014
@howardroark
Copy link
Contributor

+1 This is really the only thing that keeps me from using "formulas". I don't work with large numbers of servers that have a need to stay up for long so I never really end up needing a "master" server. I just use vagrant-aws with Salt in "local" mode and bootstrap all my states with a shell script and git. It would be sooooo much nicer to just configure that bit in the minion as a gitfs.

IMHO this is the key to getting more people using Salt with Vagrant. As of right now you can't really use formulas properly.

@terminalmage
Copy link
Contributor

This should be fixed in 2014.7.0.

@howardroark
Copy link
Contributor

I am super excited if that is true! Looks like that release is about to wrap up as well :)

@howardroark
Copy link
Contributor

@terminalmage
Copy link
Contributor

@howardroark Regarding the initial content of your last post, the log message was there because Salt has switched to pygit2 as the preferred interface to git in 2014.7.0. GitPython is still supported, and as you can see from the log there, it did detect GitPython.

You can squelch that warning by setting gitfs_provider: gitpython in your minion config.

The git external pillar is not the same thing as gitfs and has not yet been modified work with pygit2. We are targeting a rewrite for one of the next 2 feature releases.

@howardroark
Copy link
Contributor

@terminalmage Thanks for detailed response :) I deleted my last post because I soon realized that my core issue was not related to that message. Great tip though! Seems pygit2 and libgit2 are having install issues on Ubunut 14.04, either way gitpython was a much simpler install.

@terminalmage
Copy link
Contributor

Yeah, this is covered in the GitFS Walkthrough. A relatively new version of pygit2 is required, in order to support repo authentication, and it's not yet packaged on many Ubuntu releases. We've considered creating a PPA for this, but it's lower on the priority list with a new feature release imminent.

@howardroark
Copy link
Contributor

@terminalmage I am not sure if I am setting things up wrong on the minion, but Jinja seems to have trouble finding files.

state_verbose: True

file_client: local

gitfs_provider: gitpython

fileserver_backend:
  - roots
  - git

gitfs_remotes:
  - https://github.com/saltstack-formulas/nginx-formula.git

file_roots:
  base:
    - /home/user/salt/root

pillar_roots:
  base:
    - /home/user/salt/pillar
salt-call state.highstate -l debug
sh: 0: getcwd() failed: No such file or directory
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: ubuntu
[DEBUG   ] Configuration file path: /etc/salt/minion
[DEBUG   ] Reading configuration from /etc/salt/minion
sh: 0: getcwd() failed: No such file or directory
[DEBUG   ] Jinja search path: ['/home/user/salt/pillar']
[DEBUG   ] Rendered data from file: /home/user/salt/pillar/top.sls:
base:
  '*':
    - server

# vim: set ft=yaml et sw=2 ts=2 sts=2 :

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('base', OrderedDict([('*', ['server'])]))])
[INFO    ] gitpython gitfs_provider enabled
[DEBUG   ] Fetching from https://github.com/saltstack-formulas/nginx-formula.git
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Jinja search path: ['/home/user/salt/pillar']
[DEBUG   ] Rendered data from file: /home/user/salt/pillar/top.sls:
base:
  '*':
    - server

# vim: set ft=yaml et sw=2 ts=2 sts=2 :

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('base', OrderedDict([('*', ['server'])]))])
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://top.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'top.sls'
[DEBUG   ] Jinja search path: ['/home/user/salt/root']
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/top.sls:
base:
  '*':
    - server

# vim: set ft=yaml et sw=2 ts=2 sts=2 :

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('base', OrderedDict([('*', ['server'])]))])
[DEBUG   ] Syncing all
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/modules'
[INFO    ] Syncing modules for environment 'base'
[INFO    ] Loading cache from salt://_modules, for base)
[DEBUG   ] Fetching from https://github.com/saltstack-formulas/nginx-formula.git
[INFO    ] Caching directory '_modules/' for environment 'base'
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_modules'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/states'
[INFO    ] Syncing states for environment 'base'
[INFO    ] Loading cache from salt://_states, for base)
[INFO    ] Caching directory '_states/' for environment 'base'
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_states'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[INFO    ] Syncing grains for environment 'base'
[INFO    ] Loading cache from salt://_grains, for base)
[INFO    ] Caching directory '_grains/' for environment 'base'
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_grains'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/renderers'
[INFO    ] Syncing renderers for environment 'base'
[INFO    ] Loading cache from salt://_renderers, for base)
[INFO    ] Caching directory '_renderers/' for environment 'base'
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_renderers'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/returners'
[INFO    ] Syncing returners for environment 'base'
[INFO    ] Loading cache from salt://_returners, for base)
[INFO    ] Caching directory '_returners/' for environment 'base'
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_returners'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/outputters'
[INFO    ] Syncing outputters for environment 'base'
[INFO    ] Loading cache from salt://_outputters, for base)
[INFO    ] Caching directory '_outputters/' for environment 'base'
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_outputters'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/utils'
[INFO    ] Syncing utils for environment 'base'
[INFO    ] Loading cache from salt://_utils, for base)
[INFO    ] Caching directory '_utils/' for environment 'base'
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_utils'
[DEBUG   ] Refreshing modules...
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] SaltEvent PUB socket URI: ipc:///var/run/salt/minion/minion_event_1d41c853af_pub.ipc
[DEBUG   ] SaltEvent PULL socket URI: ipc:///var/run/salt/minion/minion_event_1d41c853af_pull.ipc
[DEBUG   ] Sending event - data = {'_stamp': '2014-11-21T13:27:47.907346'}
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://server.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'server.sls'
[DEBUG   ] Jinja search path: ['/home/user/salt/root']
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/server.sls:
include:
  - nginx

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('include', ['nginx'])])
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://nginx.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://nginx/init.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'nginx/init.sls'
[DEBUG   ] Jinja search path: ['/home/user/salt/root']
[DEBUG   ] Fetching from https://github.com/saltstack-formulas/nginx-formula.git
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://nginx/map.jinja'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'nginx/map.jinja'
[CRITICAL] Rendering SLS 'base:nginx' failed: Jinja error: nginx/map.jinja
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 286, in render_jinja_tmpl
    output = template.render(**unicode_context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python2.7/dist-packages/salt/utils/jinja.py", line 132, in get_source
    raise TemplateNotFound(template)
TemplateNotFound: nginx/map.jinja

; line 1

---
{% from "nginx/map.jinja" import nginx as nginx_map with context %}    <======================

include:
  - nginx.common
{% if salt['pillar.get']('nginx:use_upstart', nginx_map['use_upstart']) %}
  - nginx.upstart
[...]
---
[DEBUG   ] LazyLoaded .returner
local:
    Data failed to compile:
----------
    Rendering SLS 'base:nginx' failed: Jinja error: nginx/map.jinja
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 286, in render_jinja_tmpl
    output = template.render(**unicode_context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python2.7/dist-packages/salt/utils/jinja.py", line 132, in get_source
    raise TemplateNotFound(template)
TemplateNotFound: nginx/map.jinja

; line 1

---
{% from "nginx/map.jinja" import nginx as nginx_map with context %}    <======================

include:
  - nginx.common
{% if salt['pillar.get']('nginx:use_upstart', nginx_map['use_upstart']) %}
  - nginx.upstart
[...]
---

@khrisrichardson
Copy link

@howardroark that's a duplicate of #17963.

@howardroark
Copy link
Contributor

Thanks @khrisrichardson ! Cool, well at least I know I am on the right track. This is soo close! It's gotta work with formulas to be official though.

@howardroark
Copy link
Contributor

#18860

@terminalmage I guess this will be officially working as soon as a new 2014.7.x release is cut. Any idea if there is a plan to do that in the near future?

Thanks for getting all this working btw!

@terminalmage
Copy link
Contributor

Yes, the release should be cut soon and handed off to QA for a round of testing, then handed off to packagers before it is officially released, hopefully next week.

@aboe76
Copy link
Contributor Author

aboe76 commented Feb 27, 2015

@terminalmage I think this issue can be closed with the release of 2014.7.x

@terminalmage
Copy link
Contributor

Good call, thanks for the reminder.

@bashu
Copy link

bashu commented Oct 20, 2015

still not fixed?

@mafrosis
Copy link
Contributor

This is fixed @bashu

@adityajain21
Copy link

I want to use gitfs in masterless mode.
Any links or threads on that?

@aboe76
Copy link
Contributor Author

aboe76 commented Jun 9, 2016

@adityajain21 no there isn't any documentation specific for the masterless setup,
but the master config can be used in the minion

something like:

file_client: local

gitfs_provider: gitpython

fileserver_backend:
  - roots
  - git

gitfs_remotes:
  - https://github.com/saltstack-formulas/salt-formula.git

file_roots:
  base:
    - /srv/salt

pillar_roots:
  base:
    - /srv/pillar

@adityajain21
Copy link

adityajain21 commented Jun 9, 2016

I inserted these lines in the /etc/salt/minion file.
But when running, I cannot see the git repo's there.

@aboe76
Copy link
Contributor Author

aboe76 commented Jun 9, 2016

@adityajain21, this is just an example, please adjust, if you use pygitfs or dulwhich or gitpython,
and the gitfs repo will somewhere in in /var/cache/salt/minion/

@terminalmage
Copy link
Contributor

@adityajain21 There is nothing special that you need to do to use gitfs in masterless mode, aside from adding the gitfs configuration to /etc/salt/minion instead of /etc/salt/master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature new functionality including changes to functionality and code refactors, etc.
Projects
None yet
Development

No branches or pull requests