Add support to install collections from git repositories

changelogs/fragments/69154-install-collection-from-git-repo.yml

@@ -0,0 +1,4 @@
+minor_changes:
+  - ansible-galaxy - Allow installing collections from git repositories.
+  - ansible-galaxy - Requirement entries for collections now support a 'type' key to indicate whether the collection is a galaxy artifact, file, url, or git repo.
+  - ansible-galaxy - Support both 'galaxy.yml' and 'galaxy.yaml' files for collections.

docs/docsite/rst/dev_guide/developing_collections.rst

@@ -45,7 +45,7 @@ Collections follow a simple data structure. None of the directories are required
 
 
 .. note::
-    * Ansible only accepts ``.yml`` extensions for :file:`galaxy.yml`, and ``.md`` for the :file:`README` file and any files in the :file:`/docs` folder.
+    * Ansible only accepts ``.md`` extensions for the :file:`README` file and any files in the :file:`/docs` folder.
     * See the `ansible-collections <https://github.com/ansible-collections/>`_ GitHub Org for examples of collection structure.
     * Not all directories are currently in use. Those are placeholders for future features.
 
@@ -339,6 +339,19 @@ installs the collection in the first path defined in :ref:`COLLECTIONS_PATHS`, w
 
 Next, try using the local collection inside a playbook. For examples and more details see :ref:`Using collections <using_collections>`
 
+.. _collections_scm_install:
+
+Installing collections from a git repository
+--------------------------------------------
+
+You can also test a version of your collection in development by installing it from a git repository.
+
+.. code-block:: bash
+
+   ansible-galaxy collection install git+https://github.com/org/repo.git,devel
+
+.. include:: ../shared_snippets/installing_collections_git_repo.txt
+
 .. _publishing_collections:
 
 Publishing collections

docs/docsite/rst/galaxy/user_guide.rst

@@ -83,6 +83,10 @@ Downloading a collection for offline use
 
 .. include:: ../shared_snippets/download_tarball_collections.txt
 
+Installing a collection from a git repository
+---------------------------------------------
+
+.. include:: ../shared_snippets/installing_collections_git_repo.txt
 
 Listing installed collections
 -----------------------------
@@ -302,6 +306,10 @@ Use the following example as a guide for specifying roles in *requirements.yml*:
       scm: git
       version: "0.1"  # quoted, so YAML doesn't parse this as a floating-point value
 
+.. warning::
+
+   Use SSH keys instead of embedding credentials into a SCM URL.
+
 Installing roles and collections from the same requirements.yml file
 ---------------------------------------------------------------------
 

docs/docsite/rst/shared_snippets/installing_collections_git_repo.txt

@@ -0,0 +1,84 @@
+You can install a collection in a git repository by providing the URI to the repository instead of a collection name or path to a ``tar.gz`` file. The collection must contain a ``galaxy.yml`` file, which will be used to generate the would-be collection artifact data from the directory. The URI should be prefixed with ``git+`` (or with ``git@`` to use a private repository with ssh authentication) and optionally supports a comma separated version (for example, a commit or tag).
+
+.. warning::
+
+   Use SSH keys instead of embedding credentials into a git URL.
+
+.. code-block:: bash
+
+   # Install a collection in a repository using the latest commit on the branch 'devel'
+   ansible-galaxy collection install git+https://github.com/organization/repo_name.git,devel
+
+   # Install a collection from a private github repository
+   ansible-galaxy collection install git@github.com:organization/repo_name.git
+
+   # Install a collection from a local git repository
+   ansible-galaxy collection install git+file:///home/user/path/to/repo/.git
+
+In a ``requirements.yml`` file, you can also use the ``type`` and ``version`` keys in addition to using the ``git+repo,version`` syntax for the collection name.
+
+.. code-block:: yaml
+
+   collections:
+     - name: https://github.com/organization/repo_name.git
+       type: git
+       version: devel
+
+Git repositories can be used for collection dependencies as well. This can be helpful for local development and testing but built/published artifacts should only have dependencies on other artifacts.
+
+.. code-block:: yaml
+
+   dependencies: {'git@github.com:organization/repo_name.git': 'devel'}
+
+Default repository search locations
+-----------------------------------
+
+There are two paths searched in a repository for collections by default.
+
+The first is the ``galaxy.yml`` file in the top level of the repository path. If the ``galaxy.yml`` file exists it's used as the collection metadata and the individual collection will be installed.
+
+.. code-block:: text
+
+   ├── galaxy.yml
+   ├── plugins
+   │   ├── lookup
+   │   ├── modules
+   │   └── module_utils
+   └─── README.md
+
+The second is a ``galaxy.yml`` file in each directory in the repository path (one level deep). In this scenario, each directory with a ``galaxy.yml`` is installed as a collection.
+
+.. code-block:: text
+
+   directory
+   ├── docs
+   ├── galaxy.yml
+   ├── plugins
+   │   ├── inventory
+   │   └── modules
+   └── roles
+
+Specifying the location to search for collections
+-------------------------------------------------
+
+If you have a different repository structure or only want to install a subset of collections, you can add a fragment to the end of your URI (before the optional comma-separated version) to indicate which path ansible-galaxy should inspect for ``galaxy.yml`` file(s). The path should be a directory to a collection or multiple collections (rather than the path to a ``galaxy.yml`` file).
+
+.. code-block:: text
+
+   namespace
+   └── name
+       ├── docs
+       ├── galaxy.yml
+       ├── plugins
+       │   ├── README.md
+       │   └── modules
+       ├── README.md
+       └── roles
+
+.. code-block:: bash
+
+   # Install all collections in a particular namespace
+   ansible-galaxy collection install git+https://github.com/organization/repo_name.git#/namespace/
+
+   # Install an individual collection using a specific commit
+   ansible-galaxy collection install git+https://github.com/organization/repo_name.git#/namespace/name/,7b60ddc245bc416b72d8ea6ed7b799885110f5e5

docs/docsite/rst/shared_snippets/installing_multiple_collections.txt

@@ -13,7 +13,11 @@ You can also setup a ``requirements.yml`` file to install multiple collections i
      version: 'version range identifiers (default: ``*``)'
      source: 'The Galaxy URL to pull the collection from (default: ``--api-server`` from cmdline)'
 
-The ``version`` key can take in the same range identifier format documented above.
+The supported keys for collection requirement entries are ``name``, ``version``, ``source``, and ``type``.
+
+The ``version`` key can take in the same range identifier format documented above. If you're installing a collection from a git repository instead of a built collection artifact, the ``version`` key refers to a git tree-ish.
+
+The ``type`` key can be set to ``galaxy``, ``url``, ``file``, and ``git``. If ``type`` is omitted, the ``name`` key is used to implicitly determine the source of the collection.
 
 Roles can also be specified and placed under the ``roles`` key. The values follow the same format as a requirements
 file used in older Ansible releases.

lib/ansible/cli/galaxy.py

@@ -518,6 +518,7 @@ def _parse_requirements_file(self, requirements_file, allow_old_format=True):
             - name: namespace.collection
               version: version identifier, multiple identifiers are separated by ','
               source: the URL or a predefined source name that relates to C.GALAXY_SERVER_LIST
+              type: git|file|url|galaxy
 
         :param requirements_file: The path to the requirements file.
         :param allow_old_format: Will fail if a v1 requirements file is found and this is set to False.
@@ -590,6 +591,10 @@ def parse_role_req(requirement):
                     if req_name is None:
                         raise AnsibleError("Collections requirement entry should contain the key name.")
 
+                    req_type = collection_req.get('type')
+                    if req_type not in ('file', 'galaxy', 'git', 'url', None):
+                        raise AnsibleError("The collection requirement entry key 'type' must be one of file, galaxy, git, or url.")
+
                     req_version = collection_req.get('version', '*')
                     req_source = collection_req.get('source', None)
                     if req_source:
@@ -601,9 +606,9 @@ def parse_role_req(requirement):
                                                     req_source,
                                                     validate_certs=not context.CLIARGS['ignore_certs']))
 
-                    requirements['collections'].append((req_name, req_version, req_source))
+                    requirements['collections'].append((req_name, req_version, req_source, req_type))
                 else:
-                    requirements['collections'].append((collection_req, '*', None))
+                    requirements['collections'].append((collection_req, '*', None, None))
 
         return requirements
 
@@ -705,12 +710,13 @@ def _require_one_of_collections_requirements(self, collections, requirements_fil
             for collection_input in collections:
                 requirement = None
                 if os.path.isfile(to_bytes(collection_input, errors='surrogate_or_strict')) or \
-                        urlparse(collection_input).scheme.lower() in ['http', 'https']:
+                        urlparse(collection_input).scheme.lower() in ['http', 'https'] or \
+                        collection_input.startswith(('git+', 'git@')):
                     # Arg is a file path or URL to a collection
                     name = collection_input
                 else:
                     name, dummy, requirement = collection_input.partition(':')
-                requirements['collections'].append((name, requirement or '*', None))
+                requirements['collections'].append((name, requirement or '*', None, None))
         return requirements
 
     ############################