Add support for inventories stored in a Git repository

[Nektototam]

Title: Add support for inventories stored in a Git repository

Description:
Currently, Ansible Semaphore only supports inventories stored locally within a project. However, for many users, especially those who use static inventories, it would be extremely useful to have the ability to store inventory files in a Git repository.

I propose adding the following functionality:

1. Ability to specify the URL of a Git repository in the project settings, where inventory files are stored.
2. When running tasks and templates, Semaphore will clone the specified repository and use the inventories from there.
3. If a Git repository for inventories is specified in the project, they will take precedence over the project's local inventories.
4. Add support for specifying a particular branch/tag/commit of the repository to allow fixing the version of the inventory used.

Justification:
Storing inventories in Git will provide the following benefits:

• Ability to version control and track the history of inventory changes
• Centralized storage, eliminating the need to copy inventories into each project
• Ability to use a single inventory for multiple projects/environments
• Simplification of collaborative work on inventories within a team

This functionality will be especially useful for projects with complex multi-level static inventories, as well as for those who practice the GitOps approach.

Possible implementation:

1. Add fields to the Project model: inventory_repo_url, inventory_repo_branch, inventory_repo_key
2. Implement cloning of the repository and reading inventories from it on the backend
3. Add fields to the project creation/editing interface for specifying the parameters of the Git repository for inventories
4. When running tasks, use the inventories from the repository if it is specified. If not, use the project's local inventories as it is now.

I would appreciate consideration of this proposal. I'm ready to provide additional details and participate in the discussion of the implementation. Thank you!

[lostb1t]

+1 for this. I version control everything so im blocked by this.

[fatihusta]

We already make this using ansible for git and http server

mkdir /tmp/project
mkdir /tmp/inventory
mkdir /tmp/http_server

create dummy inventory git project

cd /tmp/inventory

# create inventory file
cat > inventory <<EOF
# Host Group Definition
[group1]
host1 ansible_host=192.0.2.1
host2 ansible_host=192.0.2.2

# All Host Group variables
[all:vars]
ansible_ssh_private_key_file=keys/ssh_key
ansible_connection=ssh
ansible_user=ubuntu
EOF

commit changes

git init
git add .
git commit -m "inventory created"

copy this content into http_server folder /tmp/http_server/example.json

[
  {
    "name": "host3",
    "groups": [
      "group1",
      "group2"
    ],
    "ansible_connection": "ssh",
    "ansible_user": "ubuntu",
    "ansible_host": "192.0.2.3",
    "ansible_port": "22"
  },
  {
    "name": "host4",
    "groups": [
      "group1",
      "group2"
    ],
    "ansible_connection": "ssh",
    "ansible_user": "ubuntu",
    "ansible_host": "192.0.2.4",
    "ansible_port": "22"
  }
]

Run http server

cd /tmp/http_server
python3 -m http.server

Open new terminal

copy this playbook content into project folder. /tmp/project/playbook.yml

---
- hosts: localhost
  connection: local
  gather_facts: false
  vars:
    # FROM Git
    inventory_git_branch: main # you can take this value via extra-vars
    inventory_git_token: !vault |
      $ANSIBLE_VAULT;1.1;AES256
      66343838333462386131376436303434323333616630306536643432326537636135666138663361
      3438366236323138303531356333393032656639623737330a333539636636343138613037656131
      65303433323764633266363739393563333662346532343330356236366232663736316364346636
      6530393935356334650a323135313536376566633862643432373236653038633932386137336664
      6365
    #inventory_git_repo: https://oauth2:{{ inventory_git_token }}@gitlab.com/yourorganization/inventory.git
    inventory_git_repo: file:///tmp/inventory
    # FROM HTTP URL
    inventory: "{{ project_name | default('example') or 'example' }}" # you can take this value via extra-vars
    inventory_url: http://localhost:8000/{{ inventory }}.json
    password: password
    #password: !vault |
    #  $ANSIBLE_VAULT;1.1;AES256
    #  66343838333462386131376436303434323333616630306536643432326537636135666138663361
    #  3438366236323138303531356333393032656639623737330a333539636636343138613037656131
    #  65303433323764633266363739393563333662346532343330356236366232663736316364346636
    #  6530393935356334650a323135313536376566633862643432373236653038633932386137336664
    #  6365
  tasks:
    - name: Clone latest inventory from git to local
      delegate_to: localhost
      run_once: true
      vars:
        ansible_become: false
      environment:
        GIT_TERMINAL_PROMPT: "0"
      ansible.builtin.git:
        repo: "{{ inventory_git_repo }}"
        dest: "./inventory"
        version: "{{ inventory_git_branch }}"
        clone: true
        update: false
        force: true
        recursive: false
      changed_when: false
      ignore_errors: true

    - name: Refresh inventory for git
      meta: refresh_inventory

    - name: Get inventories from url
      ansible.builtin.uri:
        url: "{{ inventory_url }}" 
        method: GET
        headers:
          X-Token: "{{ password }}"
        return_content: true
        status_code: 200
        body_format: json
      register: inventory
      changed_when: false
      failed_when: false
      retries: 5
      delay: 15

#    - name: print inventory file
#      ansible.builtin.debug:
#        var: inventory.json
#      when: inventory.status is defined
#            and inventory.status == 200

    - name: Generate dynamic inventory(in-memory)
      ansible.builtin.add_host:
        name: "{{ item.name }}"
        groups: "{{ item.groups }}"
        ansible_ssh_private_key_file: keys/ssh_key
        ansible_connection: "{{ item.ansible_connection }}" # optional
        ansible_user: "{{ item.ansible_user }}"
        ansible_host: "{{ item.ansible_host }}"
        ansible_port: "{{ item.ansible_port }}" # optional
        # ... possible to add extra variables for hosts
      changed_when: false
      loop: "{{ inventory.json }}"
      loop_control:
        label: "inventory_hostname: {{ item.name }}, ansible_host: {{ item.ansible_host }}"
      when: inventory.status is defined
            and inventory.status == 200

- name: Validate that inventory was refreshed
  hosts: all # or you can add group1 or gorup2 or both like group1:group2
  gather_facts: false
  tasks:
    - name: Validate deployable groups
      ansible.builtin.debug:
        var: groups
      run_once: true
      delegate_to: localhost

    - name: Validate deployable hosts
      ansible.builtin.debug:
        var: inventory_hostname

run playbook

cd /tmp/project

ansible-playbook playbook.yaml -i inventory/inventory

Example output

PLAY [localhost] ************************************************************************************************************************************************************

TASK [Clone latest inventory from git to local] *****************************************************************************************************************************
ok: [localhost]

TASK [Refresh inventory for git] ********************************************************************************************************************************************

TASK [Get inventories from url] *********************************************************************************************************************************************
ok: [localhost]

TASK [Generate dynamic inventory(in-memory)] ********************************************************************************************************************************
ok: [localhost] => (item=inventory_hostname: host3, ansible_host: 192.0.2.3)
ok: [localhost] => (item=inventory_hostname: host4, ansible_host: 192.0.2.4)

PLAY [Validate that inventory was refreshed] ********************************************************************************************************************************

TASK [Validate deployable groups] *******************************************************************************************************************************************
ok: [host3 -> localhost] => {
    "groups": {
        "all": [
            "host3",
            "host4",
            "host1",
            "host2"
        ],
        "group1": [
            "host1",
            "host2",
            "host3",
            "host4"
        ],
        "group2": [
            "host3",
            "host4"
        ],
        "ungrouped": []
    }
}

TASK [Validate deployable hosts] ********************************************************************************************************************************************
ok: [host3] => {
    "inventory_hostname": "host3"
}
ok: [host4] => {
    "inventory_hostname": "host4"
}
ok: [host1] => {
    "inventory_hostname": "host1"
}
ok: [host2] => {
    "inventory_hostname": "host2"
}

PLAY RECAP ******************************************************************************************************************************************************************
host1                      : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
host2                      : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
host3                      : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
host4                      : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

[pwannenmacher]

Will there be an option to select a Git repo as an investoy in the UI in the near future?

[fiftin]

Duplicate #902