Inventory update based on project and not job

[Giusepe92]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• API
• UI

SUMMARY

We are using AWX 9.1.1
We defined a project named : DEPLOY_DEV
for this project, we allowed branch override, and we have not specified any branch.

We defined a job template : DEPLOY_DEV_AUTO
this job template ask for branch on launch, which will point to the branch SCM where we have our playbook and our inventory.

We defined an invenotry named: INVENTORY_DEV
this inventory is supposed to point to the same branch SCM as job template on run.

Expected Behaviour :
When we launch a job based on job template DEPLOY_DEV_AUTO and specify branch "V1" , DEPLOY_DEV and INVENTORY_DEV should be synchronized with branch "V1", then the job is launched

Actual Behaviour :
When we launch a job based on job template DEPLOY_DEV_AUTO and specify branch "V1" , DEPLOY_DEV is synchronized with branch "V1",INVENTORY_DEV is synchronized with "master" (since the field branch in project is empty) . it fails, which is normal since it's looking for inventory in master branch.
We don't want to put anything in branch field for project since we have a new branch for every job, so a new inventory and a new playbook.

ENVIRONMENT

• AWX version: 9.1.1

[AlanCoding]

Is this the same as #5698? If it is, I just want to consolidate, and set priority.

[Giusepe92]

Yes

[linkeal]

We are hitting the exact same problem, also AWX 9.1.1. I somehow have the feeling that it is related to the fact that there is a "prompt on launch" checkbox missing in the inventory. At least I've also tried to create a workflow with an scm branch definition on launch which updates first the project and then the inventory and then executed the template and the same problem persists, that the project is correctly checked out from a branch but the inventory is checked out from master.

Would be really helpful to get this fixed because we would also like to use AWX for our CI/CD and not create a separate project/inventory/template for every branch.
Thanks!

[hardeaux]

Same issue here - looking for best of both worlds where inventory/playbooks tracked in gitlab + ability for users to easily test their inventory branch changes with AWX before requesting merge.

glad there is already an issue opened - please fix.

-thanks

[idelsink]

We have the same issue, We have an inventory with multiple sources both from AWS and sourced from a project.

When launching the project with a branch, for example devel the inventory sync is triggered with the branch configured in the project (for example master).

For now we can work around this by configuring the project's branch with the branch we want to target with the inventory, but this is not feasible in the long run of course.

(We're using AWX 10.0.0 with ansible v2.9.5)

[Throckmortra]

same issue here

[eharley94]

We are also hitting the same issue. It mean we have to update the project to the specific branch, which can cause issues for all of the scheduled jobs that are running if that branch is a development branch we are testing for something else.

[stewardb]

I am also experiencing this issue. Looking to have the project shared with developers using their own branch and we need to be able to have it schedule jobs and use the inventory files from that specific branch. They only way I can think of now is to create a new project for each scheduled run?

[nelsonyaccuzzi]

Same issue, I would like to have a test branch with its own inventory so I can test the changes on the playbooks and also in the inventory.

[delamart]

Same issue here, we would like to trigger a job on a specific commit/tag/branch with a built-in inventory for CI/CD and the inventory doesn't follow the override.

Any news on this issue ?

[tsimson]

Is there any update here? I'm encountering the same issue.

[EzraBrooks]

The lack of ability to version an inventory identically to a project is causing us very frustrating Change Control issues. This is a critical feature. Is anyone on the project looking at fixing this?

[tsimson]

I'm getting around this issue by re-reading group variables from within the playbook which is on the feature branch. This is very clunky so I'm hoping this gets some attention.

[colixxx]

@AlanCoding

Hey! Are there any changes in this direction?
For server changes, we want to build on inventory, not playbooks / roles. Therefore, we want to run the playbooks project by modifying the inventory project branch.
This would add a lot of flexibility to the processes and save everyone from a lot of playbooks.

P.S My AWX version is 14.1.0

[luhahn]

It would be really great to have this feature, this makes AWX pretty useless when running with pull requests in CI/CD...
(AWX 19.2.2)

[sobedrinker42]

It seems like this thread has gone stale. Can anyone give a status if this is being looked at or not?

also - just curious how others have gone around this issue

[MatthieuLeMee]

I need this feature too since when have linked AWX on a full CICD git chain.

[ljanatka]

I would also appreciate this to be fixed, because my awx automation now requires manual steps to set the tag/branch of inventory. Any update on this issue?

[bladedoyle]

Please increase priority, this is a critical missing function.

[gurleenkaurmago]

I faced similar issue when trying to run playbooks in Redhat Automation platform 2.2.1-1. the branch override option doesnt seem to work correctly... branch override option lets me choose a different branch in template from the one mentioned in project but it fails with the error .
I dont know if this problem relates to the one being discussed here because of the version but if it helps someone... After allowing concurrent executions the jobs can run successfully with an alternative branch. Potentially a file or something was being locked causing the job to error.

[anttiah]

Same issue here. I'm stuck with the branch specified in the project from which my inventory is sourced. I've tried all kinds of workarounds, like daisy-chaining two job templates in a workflow template, and changing scm_branch between those two jobs using set_stats module, but to no avail. History of the latter job shows that scm_branch was indeed set to correct value, but it had no effect.

Ansible playbook should be inventory-driven but not being able to dynamically choose different versions/tags/branches of the inventory fights against this principle.

[fosterseth]

Users can now specify an scm_branch override on the inventory source ("allow project branch override" must be enabled). Hopefully this allows users to get around the problem that @Giusepe92 pointed out in this issue.

[anttiah]

Users can now specify an scm_branch override on the inventory source ("allow project branch override" must be enabled). Hopefully this allows users to get around the problem that @Giusepe92 pointed out in this issue.

Does it work if inventory source is updated as part of a workflow, by using Inventory Sync node, and scm_branch is specified as an external variable for the node? Like, for example, by set_stats module in a preceding workflow node.

[fosterseth]

@anttiah isn't not an extra variable that you can pass into the inventory sync. You need to set scm_branch by making api calls to the inventory source endpoint

in general, scm_branch is not meant to be changed frequently or on-the-fly. Changing scm_branch for this inventory source changes it for other jobs that may be running in the system that use this inventory source.

Given that, you could write a simple playbook or task using awx collection to set the inventory source scm_branch as part of your workflow.

[anttiah]

Thats a bummer because one of my planned use cases was validating the inventory before it gets merged in GitLab. It would nice to be able to change branch on the fly based on the branch of the merge request web hook.

Doing validation this way needs a web hook enabled workflow with a couple of steps but it's still doable. First playbook takes branch from GitLab web hook parameters and stores it into scm_branch (maybe?), after that scm-backed inventory source pulls the branch validating its syntax, and finally validation playbook runs against the inventory checking our business requirements. AWX then sends workflow job status back to GitLab and it will define if merge is allowed.

I've already done this with hard-coded branch and it works, but not being able to change branches dynamically makes it less useful.

[Tobias-Pe]

@fosterseth @AlanCoding

The original description of the Bug still applies in AWX 22.1.0; 3 years later...

Reproduction steps i took:

1. Template gets passed the scm_branch variable
2. Project fetches Repository content on the scm_branch
3. 🐛Inventory fetches Repository content on the main branch

Therefore Pullrequest #13644 with the Intent "Aims to solve #5692", didn't solve this issue here and should be picked up again for development.

The workaround that @fosterseth proposed is unclear to me.
Also saying scm_branch should not be changed on the fly is contradictory to the fact that you can change it on the fly perfectly for the project-sync. Then enabling to non-dynamicaly change the scm_branch for inventories is just causing confusion for developers and possibly much time investment.

[anttiah]

in general, scm_branch is not meant to be changed frequently or on-the-fly. Changing scm_branch for this inventory source changes it for other jobs that may be running in the system that use this inventory source.

After thinking this a bit I realized that we might not be talking about same use case. I agree that changing inventory's scm_branch on-the-fly might not be good idea for ordinary playbook runs. However, when running built-in SCM Update and Inventory Sync jobs as part of a workflow, being able to dynamically set scm_branch so that desired branch gets pulled and synced is paramount.

[Maisteru]

Still facing same issue as other colleagues. This fix #13644 doesn't seem to help at all. You added option to change scm_branch, but statically. What we expect is to change it per job run. I understand it can cause conflicts mentioned here: #13587 but maybe you can offer some workaround with caching inventory during job execution?

[Maisteru]

Short note from my side. I know it is not solving the issue, but it might be workaround in some cases.
vars_files:
- inventory/group_vars/all.yml

importing variable yaml file like this to each playbook will update all variables based on branch defined in job template (it means you can set it during call). They are updated by inventory on AWX side as well, but when you define it here it will overwrite variables from inventory. It helped in my case, so maybe it will save some time for you as well :)

[willthames]

I've just updated my PR that I created a few months back to fix this issue - I've added a test suite to validate the behaviour so if I'm missing any test cases, let me know

#14101

[person50002]

what about this hack for the meantime?

---
- name: load inventory
  hosts: localhost
  gather_facts: no

  tasks:

    - name: overwrite inventory
      ansible.builtin.copy:
        src: hosts.yaml
        dest: "{{ ansible_inventory_sources[0] }}"

    - name: load inventory
      ansible.builtin.meta: refresh_inventory

- name: your play