Auto install ansible roles

[willthames]

Proposal: Auto Install Ansible Roles

Author: Will Thames @willthames

Date: 2016/04/14

• Status: New
• Proposal type: core behaviour
• Targeted Release: 2.2
• Associated PR: Auto install roles ansible#15444
• Estimated time to implement: 2 weeks

Motivation

To use the latest (or even a specific) version of a playbook with the
appropriate roles, the following steps are typically required:

git pull upstream branch
ansible-galaxy install -r path/to/rolesfile.yml -p path/to/rolesdir -f
ansible-playbook run-the-playbook.yml

The most likely step in this process to be forgotten is the middle step. While
we can improve processes and documentation to try and ensure that this step is
not skipped, we can improve ansible-playbook so that the step is not required.

Problems

What problems exist that this proposal will solve?

• Running ansible-playbook with an outdated set of roles
• Running ansible-playbook without roles installed at all

Solution proposal

Approach 1: Specify rolesfile and rolesdir in playbook

Provide new rolesdir and rolesfile keywords:

- hosts: application-env
  become: True
  rolesfile: path/to/rolesfile.yml
  rolesdir: path/to/rolesdir
  roles:
  - roleA
  - { role: roleB, tags: role_roleB }

Running ansible-playbook against such a playbook would cause the roles listed in
rolesfile to be installed in rolesdir.

Add new configuration to allow default rolesfile, default rolesdir and
whether or not to auto update roles (defaulting to False)

Advantages

• Existing mechanism for roles management is maintained
• Playbooks are not polluted with roles 'meta' information (version, source)

Disadvantage

• Adds two new keywords
• Adds three new configuration variables for defaults

Approach 2: Allow rolesfile inclusion

Allow the roles section to include a roles file:

- hosts: application-env
  become: True
  roles:
  - include: path/to/rolesfile.yml

Running this playbook would cause the roles to be updated from the included
roles file.

This would also be functionally equivalent to specifying the roles file
content within the playbook:

- hosts: application-env
  become: True
  roles:
  - src: https://git.example.com/roleA.git
    scm: git
    version: 0.1
  - src: https://git.example.com/roleB.git
    scm: git
    version: 0.3
    tags: role_roleB

Advantages

• The existing rolesfile mechanism is maintained
• Uses familiar inclusion mechanism

Disadvantage

• Separate playbooks would need separate rolesfiles. For example, a provision
  playbook and upgrade playbook would likely have some overlap - currently
  you can use the same rolesfile with ansible-galaxy so that the same
  roles are available but only a subset of roles is used by the smaller
  playbook.
• The roles file would need to be able to include playbook features such
  as role tagging.
• New configuration defaults would likely still be required (and possibly
  an override keyword for rolesdir and role auto update)

Approach 3:

Author: chouseknecht@chouseknecht

Date: 24/02/2016

This is a combination of ideas taken from IRC, the ansible development group, and conversations at the recent contributor's summit. It also incorporates most of the ideas from Approach 1 (above) with two notables exceptions: 1) it eliminates maintaining a roles file (or what we think of today as requirements.yml); and 2) it does not include the definition of rolesdir in the playbook.

Here's the approach:

• Share the role install logic between ansible-playbook and ansible-galaxy so that ansible-playbook can resolve and install missing roles at playbook run time simply by evaluating the playbook.

• Ansible-galaxy installs or preloads roles also by examining a playbook.

• Deprecate support for requirements.yaml (the two points above make it unnecessary).

• Make ansible-playbook auto-downloading of roles configurable in ansible.cfg. In certain circumstance it may be desirable to disable auto-download.

• Provide one format for specifying a role whether in a playbook or in meta/main.yml. Suggested format:

  {
      'scm': 'git',
      'src': 'http://git.example.com/repos/repo.git',
      'version': 'v1.0',
      'name': 'repo’
  }
  

• For roles installed from Galaxy, Galaxy should provide some measure of security against version change. Galaxy should track the commit related to a version. If the role owner changes historical versions (today tags) and thus changes the commit hash, the affected version would become un-installable.

• Refactor the install process to encompass the following :

  • Idempotency - If a role version is already installed, don’t attempt to install it again. If symlinks are present (see below), don’t break or remove them. (Note by @willthames: Improve ansible-galaxy handling of role versions ansible#12904)

  • Provide a --force option that overrides idempotency.

  • Install roles via tree-ish references, not just tags or commits (This PR is merged).

  • Support a whitelist of role sources. Galaxy should not be automatically assumed to be part of the whitelist.

  • Continue to be recursive, allowing roles to have dependencies specified in meta/main.yml.

  • Continue to install roles in the roles_path.

  • Use a symlink approach to managing role versions in the roles_path. Example:

    roles/
       briancoca.oracle_java7.v1.0
       briancoca.oracle_java7.v2.2
       briancoca.oracle_java7.qs3ih6x
       briancoca.oracle_java7 =>  briancoca.oracle_java7.qs3ih6x
    

Dependencies (optional)

• Improve ansible-galaxy handling of role versions ansible#12904

Testing (optional)

Plan to improve the existing galaxy tests

Documentation (optional)

Update to various roles docs

Anything else?

The author's preferred solution remains number 1.

The problem with approach 3 is that version pinning becomes a playbook concern.
Having rolesfile specify roles versions, and playbooks make use of those roles, reduces the overall complexity of both. Using a shared roles_path directory for all roles will work if the playbook does specify versions, but no other tool pins versions in the source code itself (i.e. moving to this approach would be different to how pip, maven etc. specify the versions to use separately to the source code)

[willthames]

I'm not really convinced that Approach 3 is actually a different approach to 2, just better documented. The advantages and disadvantages to approach 2 apply to approach 3.

[tima]

I agree that the role management has room for improvement. I appreciate the work and thought that has been put in to this. I think option 3 is the strongest, but not all the way there. There are spots that need to be refined and clarified before I could feel comfortable with what it being proposed.

• Under “Problems” it reads "There is not a consistent format for specifying a role in a playbook or a dependent role in meta/main.yml.” This needs to be clarified in that there is a consistent format to specify a role in these places. What’s absent is the source and versioning information.
• Embedding versions in a playbook seems problematic in terms of maintence and workflow. Without a requirements file, teams will forced to keep separate versions of playbooks just to use different versions of roles.
• How would a developer building out or enhancing a role on their local machine use this mechanism? Would they? If not, what should developers do and how?
• How is the symlink in option 3 determined? Is that assumed to be the latest/last head from master when installed? What happens when “upgraded?"
• "For roles installed from Galaxy, Galaxy should provide some measure of security against version change.” This should not be limited to just galaxy. This could happen with internal roles in large organizations with independent and often distributed teams all contributing Ansible content.
• How would this mechanism assure a role is complete and unaltered for security and reliability purposes? Some sort of MD5 is probably needed especially for content coming in from an external source like Galaxy.
• The whitelisting mechanism needs to be expanded upon and then vetted.
• "Galaxy should track the commit related to a version.” Where is this stored? How does it avoid it getting decoupled from the playbook as it moves around? Such as what QA tested with v1.1 was the v1.1 that went in to prod. This is an issue now where no version or specific commit is defined at all, but something this proposal may want to consider.
• Assuming multiple simultaneous versions are cached/cloned to the controller machine, how would this mechanism handle the cleanup of old and unused versions of roles?
• Does the proposed format for specifying a role have any defaults? Could use more examples for differing scenarios be provides, the latest from galaxy, latest from galaxy with a version, with a tag, local file system, internal git repo, zip file etc. be provided?
• It’s not clear where this from option 3 would be implemented: "Provide a --force option that overrides idempotency.”
• "Make ansible-playbook auto-downloading of roles configurable in ansible.cfg. In certain circumstance it may be desirable to disable auto-download.” Perhaps this should be off by default so a user acknowledges they want to pull in external content.
• How would this mechanism handle roles that have packaged plugins such as callback? Roles have been floated as a means of packaging and sharing plugins. This use has been under documented and few use it currently, but it should be addressed now to avoid later issues.
• These is a bit outside of this proposal, but worth noting. Roles and galaxy gets conflated often and this proposal further contributes to that. Also, the lines between Galaxy the website/repo/API and galaxy the installer/cli is also blurred and can be confusing.

[willthames]

@tima I'll try and address each in turn. For full disclosure, I am an advocate for Option 1, Option 2 came from @bcoca and Option 3 came from @chouseknecht .

ansible/ansible#15444 is a draft implementation of Option 1.

This proposal seems to get bogged down in lots of things that aren't just 'auto install roles' - we don't have to solve every single galaxy/roles problem with this proposal - some concerns might be better as parallel proposals or PRs

Under “Problems” it reads "There is not a consistent format for specifying a role in a playbook or a dependent role in meta/main.yml.” This needs to be clarified in that there is a consistent format to specify a role in these places. What’s absent is the source and versioning information.

I can't see where it says that at all. There is the possibility of versioning info in meta/main.yml - the only place that doesn't have versions is playbooks but as you say, those should be separate concerns anyway.

Embedding versions in a playbook seems problematic in terms of maintence and workflow. Without a requirements file, teams will forced to keep separate versions of playbooks just to use different versions of roles.

I agree. I say that as an advantage of Option 1

How would a developer building out or enhancing a role on their local machine use this mechanism? Would they? If not, what should developers do and how?

This is an interesting problem that is not addressed by any of these solutions, but it is not a change from current situation. One approach might be to allow a src: /path/to/roles/dir to reference a local role (at least then it's obvious at review time that it's a local, unversioned role). However, this could be addressed entirely independently of this proposal.

How is the symlink in option 3 determined? Is that assumed to be the latest/last head from master when installed? What happens when “upgraded?"

The implementation details for Option 3 will have to come from @bcoca or @chouseknecht

"For roles installed from Galaxy, Galaxy should provide some measure of security against version change.” This should not be limited to just galaxy. This could happen with internal roles in large organizations with independent and often distributed teams all contributing Ansible content.

Again, this is the current situation and is actually orthogonal to this proposal.Agree, this metadata could be contained in .galaxy_install_info. However, this would not solve the problem you mention below of having a different v1.1 in QA to prod. The PR that introduced tree-ish versions would allow you to point at commit IDs rather than tags, but that's not really very user friendly (due to ordering concerns - it's difficult to see which commit ID is newer). Making role installation absolutely rock solid seems like a hard problem.

How would this mechanism assure a role is complete and unaltered for security and reliability purposes? Some sort of MD5 is probably needed especially for content coming in from an external source like Galaxy.

SHA commit id should be enough for git/hg based roles. The security of Galaxy service itself is not something I wish to address in this proposal

The whitelisting mechanism needs to be expanded upon and then vetted.

We'd also need to decide on what to do in the default case, which would presumably be an empty whitelist and auto install switched off. Should ansible-galaxy then refuse to install roles? Or does the whitelist only apply to auto role installation, which might be unexpected by those people actually relying on the whitelist.

"Galaxy should track the commit related to a version.” Where is this stored? How does it avoid it getting decoupled from the playbook as it moves around? Such as what QA tested with v1.1 was the v1.1 that went in to prod. This is an issue now where no version or specific commit is defined at all, but something this proposal may want to consider.

I don't think I want to address Galaxy service concerns in this proposal at all - but if Galaxy provides the SHA id we can store it in .galaxy_install_info. I can see that this is definitely a potential problem, but I'm not sure how this can be solved here

Assuming multiple simultaneous versions are cached/cloned to the controller machine, how would this mechanism handle the cleanup of old and unused versions of roles?

I think the multiple version stuff should be out of scope for auto installation

Does the proposed format for specifying a role have any defaults? Could use more examples for differing scenarios be provides, the latest from galaxy, latest from galaxy with a version, with a tag, local file system, internal git repo, zip file etc. be provided?

I don't propose changing the current role specification.

It’s not clear where this from option 3 would be implemented: "Provide a --force option that overrides idempotency.”

"Make ansible-playbook auto-downloading of roles configurable in ansible.cfg. In certain circumstance it may be desirable to disable auto-download.” Perhaps this should be off by default so a user acknowledges they want to pull in external content.

I would definitely make this off by default and configurable in ansible.cfg

How would this mechanism handle roles that have packaged plugins such as callback? Roles have been floated as a means of packaging and sharing plugins. This use has been under documented and few use it currently, but it should be addressed now to avoid later issues.

I don't see why this would need to change. This is just a different mechanism for installing roles. The use of those roles would happen in the same way.

These is a bit outside of this proposal, but worth noting. Roles and galaxy gets conflated often and this proposal further contributes to that. Also, the lines between Galaxy the website/repo/API and galaxy the installer/cli is also blurred and can be confusing.

I agree - I found that I was unclear as to where the code separation would be.

At least this would mean that you wouldn't need to use ansible-galaxy to install roles!

[bcoca]

Some of the issues i've put into the implementation already or have been covered by @tima above, but I still think its worth noting a few things:

• Option 1, the last thing we want is more ways to declare/reference a role, I want to push to reduce the number, not increase it. I can also see how the new directives will complicate role loading and make a push to make it even more dynamic which has many repercussions (we are not even done with dynamic include issues ...).
• Option 2 looks like nothing I've mentioned, you must be confusing it with my proposal to be able to execute role task files using the include directive, that proposal did not involve changing how you imported or referenced roles, it ONLY dealt with task file execution. In any case I think it overloads include to do what roles already should do and having many of the same problems as Option 1.
• Elements of option 3 are things I've suggested, but also as part of a deeper rethinking on how to handle roles. I don't think we can implement this option easily unless we first standardize on 1 role declaration format and lay out clearly how installation and versioning are going to work (aka designing a package management system). That said I think it is closest to a good solution, but it needs a lot of details and work, as pointed out above.
• about the symlink, this is something I think we can handle in the way 'slots' are handled in some distributions or 'alternatives' (again part of package management).
• I've proposed the whitelisting ONLY for autodownload, galaxy can function 'as is'. The difference is implicit install vs explicit one. Extending this to galaxy might have its uses but i would reverse the default to keep current behavior.
• I don't see how autoinstall can ignore the multiple versions unless we dismiss having that option.
• Using the chekcout SHA will only tell us the origin of the current version, it will not detect customization. So having a checksum of what the role should be is needed to detect this situation, we should then ask for an override or specific flag to 'blow away with new' or 'restore'.
• The objective of this should not be to not remove ansible-galaxy, just to complement it. I don't think we should overload playbook by merging every galaxy option into it and making it a dual purpose command line (setting/cleaning up requirements and actual execution). This feature should be narrow in scope and only add a portion of 'galaxy' to playbook, the 'autodownload', AS part of execution, not migrating functionality from one CLI to another.
• One added benefit to implementing this, is that ansible-pull will get it for 'free' as it calls ansible-playbook internally.

[willthames]

Option 1, the last thing we want is more ways to declare/reference a role, I want to push to reduce the number, not increase it. I can also see how the new directives will complicate role loading and make a push to make it even more dynamic which has many repercussions (we are not even done with dynamic include issues ...).

There are no new way with Option 1. The new directives (roles_file and (optionally) roles_path) just cause the roles to be installed. I don't understand what I've said to suggest that there is any change here.

Option 2 looks like nothing I've mentioned, you must be confusing it with my proposal to be able to execute role task files using the include directive, that proposal did not involve changing how you imported or referenced roles, it ONLY dealt with task file execution. In any case I think it overloads include to do what roles already should do and having many of the same problems as Option 1.

I can't find where I got that impression from, apologies for misrepresenting your position.

Elements of option 3 are things I've suggested, but also as part of a deeper rethinking on how to handle roles. I don't think we can implement this option easily unless we first standardize on 1 role declaration format and lay out clearly how installation and versioning are going to work (aka designing a package management system). That said I think it is closest to a good solution, but it needs a lot of details and work, as pointed out above.

As far as I'm concerned, you already have pretty much standardized on one format (the yaml roles file). The old style roles format was deprecated with zero consultation, but I can live with that.

about the symlink, this is something I think we can handle in the way 'slots' are handled in some distributions or 'alternatives' (again part of package management).

I don't have any thoughts on this.

I've proposed the whitelisting ONLY for autodownload, galaxy can function 'as is'. The difference is implicit install vs explicit one. Extending this to galaxy might have its uses but i would reverse the default to keep current behavior.

Ok. Thanks for clarifying that - I can live with that, I think we just call the variable role_auto_install_whitelist or something to make it obvious it only applies to auto installs.

I don't see how autoinstall can ignore the multiple versions unless we dismiss having that option.

Right - I think I see now. This is less manageable if people use one central roles location, although those people are currently asking for problems now. Put it this way, I don't see this as an issue of auto install really, which is just a shortcut - one thing that could help is that if you specify roles_file in the playbook, you must include roles_path too, which would lead to more playbook specific roles than relying on ansible.cfg's roles_path or C.DEFAULT_ROLES_PATH

Using the chekcout SHA will only tell us the origin of the current version, it will not detect customization. So having a checksum of what the role should be is needed to detect this situation, we should then ask for an override or specific flag to 'blow away with new' or 'restore'.

This does not need to be part of this proposal, it is a separate concern.

The objective of this should not be to not remove ansible-galaxy, just to complement it. I don't think we should overload playbook by merging every galaxy option into it and making it a dual purpose command line (setting/cleaning up requirements and actual execution). This feature should be narrow in scope and only add a portion of 'galaxy' to playbook, the 'autodownload', AS part of execution, not migrating functionality from one CLI to another.

Yep, understood from your feedback on the implementation PR. Ripping that additional functionality will greatly simplify the change, which is a good thing.

One added benefit to implementing this, is that ansible-pull will get it for 'free' as it calls ansible-playbook internally.

This is indeed a happy benefit.

[bcoca]

There are still several formats to specify roles:

• 2 in play roles:
• role dependency format
• 2 requirements formats
• tarball format

Most of them are YAML, I would like to simplify to 1 or 2 (tarball being the 2nd). I'm not sure then if your rolesfile.yml is different or overlaps with one of the existing formats.

[willthames]

I think there's a difference between role usage and role specification (and role source within the roles specification)

• the two play role formats are role usage (I assume you mean the role name string vs the role dict) - I certainly think expanding it's usage further to role specification would be a mistake.
• the role dependency format in meta/main.yml could be changed to use the yaml format - again, this is outside of the scope of this proposal (as it can be done completely independently)
• we seem to be consolidating on one requirements format, even though I've fixed all the issues with the second requirements format so that it still worked with 2.0 (I would have liked to have been consulted on deprecating a format that I implemented - but I can understand the desire for simplifying the codebase, and the YAML format is more flexible).
• the tarball is a source within the role specification, rather than a specification itself, unless there's a way to use tarballs that I'm not aware of

The roles_file field that I propose to use in playbooks just uses the exact same roles file as ansible-galaxy's existing -r option.

[avantassel]

We are using Ansible tower with github enterprise and It would be nice to not have to download the role and save it in order to use it. This way you could always get the latest master if you want and it. Also running locally you would not need to .gitignore these roles.

It would be great to use the existing galaxy base and use the existing username.rolename syntax without having to run ansible-galaxy install username.rolename.

ansible.cfg + roles could look like

[galaxy]
server=https://github-enterprise.example.com

- roles:
   - { role: username.rolename, remote_src: yes }

And/Or requirements.yml + roles could look like

---
- src: https://github-enterprise.example.com/username/rolename
  scm: git
  name: rolename

- roles:
   - { role: rolename, remote_src: yes }

[icopp]

It would be great to use the existing galaxy base and use the existing username.rolename syntax without having to run ansible-galaxy install username.rolename.

This, plus some basic version pinning, is the only thing keeping me from breaking up some existing Ansible projects into lots of small pieces to put on Galaxy. It's not a huge amount of friction, but it's constantly there.

It's also all the more striking that ansible-galaxy init produces boilerplate that assumes integration with galaxy.ansible.com, but doesn't actually take into account Galaxy dependencies. Putting a role: username.rolename dependency works for the galaxy.ansible.com UI, but doesn't actually work for the generated .travis.yml unless you go in and add ansible-galaxy install to the Travis code yourself.

[cjw296]

To be clear, I'm a massive 👎 on auto-install. Explicit is better than implicit and all that.

From my comment on ansible/galaxy-issues#49:

We're basically talking about installing dependent library software. In the Python world, we already have pip/virtualenv and conda as comparable examples, both of which support "make my current environment like this specification which comes from a file" in an idempotent way. requirements.yml gives us the specification, now we just need the idempotent install, rather than requiring the --force hack.

Autoinstall, for me, would be akin to python magically downloading and installing a package as part of an import statement. Sure, you can implement one of the import hooks to do it, but most people would make decidedly unhappy faces in your general direction if you did.

[tima]

This ansible-galaxy autoinstall function is, at a minimum, a core level implementation of what already been in Ansible for Tower for sometime.

[cjw296]

I neither use, not want to use, Ansible Tower.

[willthames]

@cjw296 this proposal is completely unrelated to Tower, other than that this would provide a benefit of Tower in core (Tower ensures that the correct role versions are installed).

And saying that it's like python installing a package as part of an import statement, given the amount of ludicrous code that protects people who currently have old libraries (do git grep LooseVersion in ansible-modules-core, for example), the possibility of mismatched versions has to be solved somewhere. I just want a way to solve the problem that obtaining the latest version of playbooks and roles does not protect you from running the playbooks against an old set of roles. If there an alternative way that means that optional auto updates are run, I'm open to suggestions.

[sivel]

I am also be a big -1 on auto installation. This has been solved in setuptools/pkg_resources by specifying a requirement and raising a VersionConflict when the version doesn't meet the requirement.

I would be much more in favor of aborting and instructing the user to upgrade.

[willthames]

@sivel that would be acceptable too. We'd still need to specify where those version specifications came from for a particular playbook though.

[geerlingguy]

See related / similar effort on the Python level: https://github.com/pypa/pipfile (and semi-related issue: ansible/galaxy-issues#165).

[cjw296]

@willthames - right, but requirements.yml seems like the perfect place for those version specifications to come from. It's frustrating that this proposal is being used as a way of blocking implementation of a more sane implementation of the code that uses requirements.yml other than making users use --force all the time.

[willthames]

@cjw296 that seems orthogonal to this proposal - aren't you just asking for ansible/ansible#12904 but without the need to force (which I'd be happy with tbh, but I was asked to put the force back in ansible/ansible#12904 (comment))

[willthames]

This proposal is completely unrelated - here I want ansible-playbook to update the roles, without needing ansible-galaxy to be involved at all. See ansible/ansible#15444 for a proposed implementation - the code is now out of date but the principle still seems reasonable.

[cjw296]

@willthames - I'm afraid @chouseknecht decided they were related. Perhaps you could ask them to re-open ansible/galaxy-issues#49 instead?

[willthames]

@cjw296 I'd say it's not a ansible/galaxy-issue though, but an ansible/ansible issue, which is why ansible/ansible#12904 exists as a solution for some of the idempotency requirements, although #23 takes the solution for conflicting role requirements further.

[cjw296]

@willthames - have to be honest, I'm not that fussed about details, I just want ansible-galaxy install -r requirements.yml to work when I've already run it once, without requiring --force, since that re-downloads all the roles.

[antonbormotov]

Is there any plan to get it done by nearest future?

[omi22]

Is there any update on this proposal ? I would love to download roles are as pre_task to my playbook

[xcambar]

You know the drill: no update means no update.

[samyscoub]

In the past ( 1.9.x releases ), I used a callback plugin for that job, but not working with the ansible 2 releases.
Why callback_plugins are not executed before playbook's read in the ansible version >=2 ?

[antonbormotov]

@samyscoub What plugin have you used to achieve it? I can't see any appropriate plugin in this list.

[samyscoub]

@antonbormotov I've made my own custom callback with this function ( working only on ansible 1.x ), and with 2 extra options from ansible-playbook galaxy_noexec when you don't want to auto refresh galaxy retrieving, and galaxy_force to force roles download :

  def playbook_on_start(self):
        galaxy_noexec = self.playbook.extra_vars.get('galaxy_noexec')
        requirements_file = "external_roles/requirements.yml"
        galaxy_force = ""
        if self.playbook.extra_vars.get('galaxy_force'):
            galaxy_force = " --force"
        if galaxy_noexec:
            print('ANSIBLE-GALAXY | Skipping ansible-galaxy as requested by galaxy_noexec variable.')
            return
        if not os.path.isfile(os.getcwd() + '/' + requirements_file):
            print('ANSIBLE-GALAXY | No <' + requirements_file + '> file found. Skipping ansible_galaxy.')
            pass
        ansible_galaxy_exe = "ansible-galaxy"
        try:
            call([ansible_galaxy_exe, 'version'], stdout=PIPE, stderr=PIPE)
        except OSError:
            print("ANSIBLE-GALAXY | %s not found in path. Please Re Install ansible." % ansible_galaxy_exe)
            print('ANSIBLE-GALAXY | Current $PATH variable: %s' % os.environ['PATH'])
            sys.exit(1)
        print("ANSIBLE-GALAXY | Downloading remote roles dependencies into :\n<" + os.getcwd() + "roles> directory")
        print('ANSIBLE-GALAXY | If you want to skip that, set \'-e galaxy_noexec=True\' into the ansible-playbook command line.')
        print('ANSIBLE-GALAXY | If you want to force ansible-galxy to retrieve already downloaded roles, set \'-e galaxy_force=True\' into the ansible-playbook command line.')
        os.system("cd " + os.getcwd() + " && " + ansible_galaxy_exe + " install -r " + requirements_file + "" + galaxy_force)
        pass

I've writtent the same code ( adapted to ansible V2 callback requirements ),

def v2_playbook_on_start(self, playbook):
        self.playbook = playbook
        self.playbook_name = os.path.basename(self.playbook._file_name)
        
        ansible_galaxy_exe = "ansible-galaxy"
        galaxy_options = {
                           'galaxy_noexec': 'no',
                           'galaxy_force': 'no',
                           'galaxy_file': 'roles/requirements.yml',
                         }

        if self._display.verbosity > 1:
            from os.path import basename
            self._display.banner("PLAYBOOK: %s" % basename(playbook._file_name))

        if self._display.verbosity > 3:
            if self._options is not None:
                for option in dir(self._options):
                    if option.startswith('_') or option in ['read_file', 'ensure_value', 'read_module']:
                        continue
                    val =  getattr(self._options,option)
                    if val:
                        self._display.vvvv('%s: %s' % (option,val))

        if self._display.verbosity > 1:
            self._display.display('%s | v2_playbook_on_start function from callback' % (self.callback_prefix))
        
        galaxy_noexec =""
        if self._options and self._options.extra_vars:
            result = [m.group(1) for m in (re.search('^galaxy_noexec=(.*)$', l) for l in self._options.extra_vars) if m]
            if len(result) > 0:
                galaxy_options['galaxy_noexec'] = result.pop()
           
            result = [m.group(1) for m in (re.search('^galaxy_force=(.*)$', l) for l in self._options.extra_vars) if m]
            if len(result) > 0:
                galaxy_options['galaxy_force'] = result.pop()

            result = [m.group(1) for m in (re.search('^galaxy_file=(.*)$', l) for l in self._options.extra_vars) if m]
            if len(result) > 0:
                galaxy_options['galaxy_file'] = result.pop()

        if self._display.verbosity > 1:
            self._display.display('%s | galaxy_noexec: %s' % (self.callback_prefix,galaxy_options['galaxy_noexec']))
            self._display.display('%s | galaxy_force: %s' % (self.callback_prefix,galaxy_options['galaxy_force']))
            self._display.display('%s | galaxy_file: %s' % (self.callback_prefix,galaxy_options['galaxy_file']))
        
        galaxy_force = ""
        if galaxy_options['galaxy_force'] == 'yes':
            galaxy_force = " --force"
        if galaxy_options['galaxy_noexec'] == 'yes':
            self._display.notice("%s | Skipping ansible-galaxy as requested by galaxy_noexec variable." % (self.callback_prefix))
            return
        try:
            call([ansible_galaxy_exe, 'version'], stdout=PIPE, stderr=PIPE)
        except OSError:
            self._display.display("%s | `%s` executable file not found in path. Please re-install ansible." % (self.callback_prefix,ansible_galaxy_exe),color=C.COLOR_ERROR)
            if self._display.verbosity > 1:
                self._display.display("%s | Current $PATH variable: %s' % os.environ['PATH']" % (self.callback_prefix),color=C.COLOR_ERROR)
            sys.exit(1)

        self._display.display("%s | ANSIBLE will try to download roles automatically" % (self.callback_prefix),color=C.COLOR_WARN)
        self._display.display("%s | If you want to skip that behaviour, set \'-e galaxy_noexec=yes\' into the ansible-playbook command line." % (self.callback_prefix),color=C.COLOR_WARN)
        if galaxy_options['galaxy_force'] == 'no':
            self._display.display("%s | If you want to force ansible-galaxy to retrieve already downloaded roles, set \'-e galaxy_force=yes\' into the ansible-playbook command line." % (self.callback_prefix),color=C.COLOR_WARN)
        if os.path.isfile(galaxy_options['galaxy_file']):
            self._display.display("%s | Downloading remote roles dependencies into roles directory set into galaxy conf file <%s>" % (self.callback_prefix,galaxy_options['galaxy_file']),color=C.COLOR_VERBOSE)
            self._display.display("%s | If you want to overwrite this default file, set \'-e galaxy_file=/SOME_FILE\' into the ansible-playbook command line." % (self.callback_prefix),color=C.COLOR_WARN)
            os.system("cd " + os.getcwd() + " && " + ansible_galaxy_exe + " install -r " + galaxy_options['galaxy_file'] + galaxy_force)
        else:
            self._display.display("%s | the default galaxy conf file <%s> does not exists, aborting" % (self.callback_prefix),color=C.COLOR_ERROR)
            self._display.display("%s | please set a different galaxy conf file with \'-e galaxy_file=/SOME_FILE\' extra_var option into the ansible-playbook command line." % (self.callback_prefix,galaxy_options['galaxy_file']),color=C.COLOR_ERROR)
            return
        pass

but the behaviour is not the same with ansible V2 => roles retrieving check from playbook is made before callback launching, and my own callback can't work when role has not been downloaded once 😕

[schweikert]

An alternative proposal that I think addresses the motivation behind this issue:

The main problem that I have with the current idiom of running ansible-galaxy install before ansible-playbook, is that when I change something in requirements.yml, I can't easily guarantee that other persons also working with the same playbook will run ansible-galaxy. There is the dangerous possibility that somebody will rollout the new version of the playbook with old copies of external roles.

Instead of implementing auto-downloading of requirements in ansible-playbook, what would perfect for me, would be to instruct ansible-playbook on how to only verify that the requirements from requirements.yml are fulfilled. I.e. it could quickly (and locally) compare if the versions specified in there, match with what is downloaded, and only abort with an error message in case there is a version mismatch. I can then execute ansible-galaxy, that doesn't bother me.

This would have the advantage that it would be a lot less intrusive and "magical", while at the same time solving the problem of making sure that everybody uses the same version of the external roles.

[bcoca]

@schweikert, none of that is possible unless we nail down 'role versioning', which is why I consider it a requirement before we even think of auto downloads.

[jchristi]

@bcoca Has that not been done yet? The devel documentation claims it is a feature of 2.5: http://docs.ansible.com/ansible/devel/roadmap/ROADMAP_2_5.html#role-versioning

[geerlingguy]

It seems, though, that for the purposes of auto-install, or version compare, we could do a pretty blind/dumb "does the version in rolename/meta/.galaxy_install_info exactly match the version in the requirements file?". If it matches, don't install/reinstall. If it doesn't match, install/reinstall the role.

Or am I missing something? I know for more advanced features like "give me 3.0 or later" (or "v3 or later") will require a more hard-lined approach to versioning, but this seems like a lot more limited/achievable scope.

[mpedrolli]

+1 for this proposal

[jleaders]

Ansible could achieve the dream of being a metaphorical "one button provisioner", but because of the lack of this feature it is truly a provisioner that can't even provision itself. Ironic.

[samyscoub]

@jleaders 👍

[vladimirtiukhtin]

Any news? Waiting this in ansible-pull...

[agaffney]

With the inclusion of include_role in 2.4-ish, it's now possible to run ansible-galaxy inside your playbook and use those downloaded roles in the same playbook.

[willthames]

@agaffney with respect, that's a massive hack. Playbook authors shouldn't have to do that just to have roles auto-installed. It could just be seamless.

[agaffney]

It's definitely a bit of a hack, but it does address some of the previous concerns about using wrapper scripts and somebody running the playbook without the wrapper. However, it's not really that bad to have command: ansible-galaxy install -f requirements.yml in pre_tasks and then use include_role instead of roles:.

[agaffney]

And as a side note, the Ansible devs have generally rejected specific feature requests for things that it's already possible to do in your playbook with only a little extra effort.

[colans]

I suppose this is a hack too, but making this the first task in a role ensures all dependencies requirements are downloaded before doing anything else.

- name: Download required roles
  local_action: command ansible-galaxy install -r {{ role_path }}/roles/requirements.yml
  become: false

See ansible/ansible#56046 (comment) for why we can't call these dependencies, which I believe can get downloaded automatically, even though we should be able to.

[vladimirtiukhtin]

@colans I believe your paticular issue is already solved using "meta" and "dependencies: []". Doing ansible-galaxy against such role automatically downloads requirements dependencies

[colans]

@vladimirtiukhtin My point was that meta/dependencies doesn't work properly. If I add a role there and then call it with import_role or include_role, it actually runs twice. At the beginning, with defaults (incorrect), and then again when needed with proper passed-in variable settings. To fix, you need to not list the dependency as a dependency, which is counter-intuitive. The "dependency" really has to become a "requirement". So even though a role is a dependency, you can't explicitly state it as such.

See the issue I referenced for more information.

[jhg03a]

Relying on an ansible-pull playbook to fetch dependencies forces you to use include_* instead of import_*. This means you're giving up the syntax checking prior to execution. This might create secondary issues should there be a bug in a role that isn't caught until half-way through the playbook execution. You might still be able to do convoluted things wherein the playbook being executed is just a wrapper for two tasks: install dependencies & import_playbook. That still feels worse though.