Feature: Allow selecting project branch / tag on JT and WFJT

[wenottingham]

ISSUE TYPE

• Feature Idea

COMPONENT NAME

• API
• UI

SUMMARY

Assorted requests we've received include items such as:

• "allow the user to upload a playbook directly for a job template"
• "allow prompting for project to use"
• "allow prompting for the SCM branch for the project"

All of these run afoul of the current underlying model in AWX:

• jobs are run from a single instance of a checked-out project
• this is shared by all JTs that use that project
• hence, each JT is explicitly "a project + a playbook path in that project", with a single SCM branch

This should be investigated to see if/how this linkage could be broken to enable use cases like above, and what that would entail.

ADDITIONAL INFORMATION

[matburt]

"allow prompting for project to use"
"allow prompting for the SCM branch for the project"

This is effectively an ad-hoc job-as-a-job-template (where everything is asked for at launch time)

[matburt]

There's a good bit of work to do on projects for this:

• Grab the branches
• Index files from the different branches
• This is pretty complex on SVN where we don't have all of the refs on initial checkout.

As @wenottingham said... a lot of our very basic fallback RBAC relies on the certainty of having a project defined... otherwise we have to fall back to org-admins and those delegated the specific job template.

[ffung]

+1 for this feature
we have a ci/cd setup for ansible projects.
The Ansible code in a project is checked (basically linting) in CI and released via a tag. We want to trigger a specific tag and/or commit-id after the build has run successfully and promote the code across environments.

[oroomans]

Hi,

we are in the same situation here.

Is this feature request in the roadmap now for short or middle term ?

Thanks.

[dhajoshi]

+1 , looking for similar feature to ask for tag/branch while running job and sync project from SCM using that branch/tag

[mabashian]

UI Implications

allow the user to upload a playbook directly for a job template

There are several places in the UI where the user can "upload" files but most notably is probably the logo file picker. I don't see too much issue with using that as a starting point for this feature. Assumption: If a file is uploaded then the user shouldn't be able to select/prompt for a project. We may need to build in a mechanism for removing that uploaded playbook if the user decides to migrate that project to source control. Error handling may be important here - what happens when the user uploads something nonsensical.

allow prompting for project to use

1. Add prompt on launch checkbox above project field
2. Validation on project field mirrors that of inventory. Either a default is provided or it's prompted or both. Form should stay disabled until one of those conditions is satisfied.
3. Project prompting will need to be added to JT launch modal. Will need to talk through UX. Adding another tab to that modal might be too much. We currently have tabs for Credential, Inventory, Other, Survey and Preview. Might it be time to do some consolidation?

allow prompting for the SCM branch for the project

1. We'll need to add the branch field to something. I assume that it would be the JT(?). This field would be a dropdown and would need the prompt on launch checkbox.
2. Add branch to the JT launch modal (wherever we decide to put project prompting)
3. We'll want to add the branch to the left-hand panel of the Job Results view

[wenottingham]

allow the user to upload a playbook directly for a job template

Whoops, we're not going to do that. We can strike that part of the feature.

[matburt]

Yeah lets not do that one ;)

The rest of these on the UI side look good. There's some complexity on the backend side of making sure we have the tree indexed for presenting a playbook list. Right now that's based entirely on what's on the filesystem but the files could be different between branches. This might not be fully possible depending on the scm type.

For Git it's important to point out that it's not just branches we're talking about... but anything that could be referenced as a point in the tree: branch, tag, individual commits, etc. So the wording needs to reflect that.

[matburt]

Some notes from design meeting:

Model semantics:

• Add field on the Project that informs allowing overriding the branch/version on the Job Template
• Keep existing branch/version field on the Project to act as a default
• adding branch specifier on job template
• Add Organization to Job Template (data migration of project org to job template). Of concern here is if a Job Template's project is null when a project is removed but the job template is retained.

Playbook indexing and selecting

• UI to allow free-form playbook selection on the job template list. Possibly change this field to make it more obvious that you can type free-form text into it.
• Playbooks only indexed at HEAD of default or given project branch
• branch-prompt-launch (but not playbook [yet?])

Workflows:

• branch prompting for workflows

Job Execution:

• Directory execution isolation via Runner or AWX itself under project lock.

• Explore branch and playbook listing/population or removing the capability

[jlmitch5]

UI notes

• Add allow override to project edit form for allowing branch to be changed at jt/wf level.
• Use inventory file in project select2 box for playbook on jt form (select + type in)
  • UX of current may not be great for users, figure out ideal case w Taufique and then see what’s possible w select2
  • We might need to refretch possible playbook list when branch changes just like when project changes.
• For branch we might also move to a select + type (like inventory in project)? TBD
• For SVN, there may be no options in either of these drop downs
• Prompting is just for branch (not project and not playbook)
  • We are assuming playbook exists on both branches
  • Schedules/WFJT need to be able to have the prompt (but @mabashian updating in one place will get you the ability in all 3)
• WF stuff
  • Selection of branch in a workflow node
  • Add branch to prompt for wf job

---

• We might add org lookup to JT with a different card

[AlanCoding]

Add Organization to Job Template (data migration of project org to job template). Of concern here is if a Job Template's project is null when a project is removed but the job template is retained.

I'm planning to file this as a separate issue after I've gathered more details about the expectation. For now, this is a related card covered by this point: #3432

[AlanCoding]

Created new issue to handle that point - #3903

[AlanCoding]

Link another related issue: #2357

Projects have conventionally had an scm_revision field associated with them. Project updates are the only thing that can modify that field on a "check" type update. Since the JT can pull any number of branches, a single value for this will no longer be sufficient.

My current thinking is that a project with allow_override=True will fetch all branches from the source, and we will have to maintain a dictionary with the revisions for all those branches. Somehow, we need to surface to the user what these are, and what branches were updated with the update. I am considering that this might be something which can be contributed back to Ansible core for the git module, for the case that the bare parameter is true.

[AlanCoding]

Keep existing branch/version field on the Project to act as a default

I have to strike "version" from this requirement, because otherwise it becomes self-contradictory.

Very important to the completion of this feature: We have 2 options. Either allow_override (option 1) makes the project always update before running the job template, or (option 2) we cache >1 project revision.

Option 1: job syncs that update

This sounds like "update on launch" behavior, but it would really be more like "update on sync". There's a difference in mechanics and user experience between the two. If a project has "update on launch", the project will reflect the updated version, but "update on sync" just pulls the remote to the execution instance, and there's no way to ever guarantee that code-state can ever be used again in a subsequent job. With that all said, I have accepted that some version of this is unavoidable (more on that later), and it is also the fastest route to having something that UI and QE can start working off of.

I hope it's easy to get agreement that if some non_default_branch updates every run, then it would be crazy for master to be pinned to an old version, and have project updates advance the SHA-1 that is used in the master branch, but no other branches.

It would be easy enough to have the project's "Update on launch" button force-checked if the user clicked the "allow override" button. In this case, we update any branch on every run, and that's that. This goes further - updates would need to be disabled completely for the project, because they would be meaningless.

However, I have found this can't be cleanly implemented because pull requests are part of the requirements, and those don't work in a one-off way of cloning ansible/ansible#57497

Option 2: recording branch revisions

This is where I wanted to do bare clones, and peel off branches from those into the running directory. Downside is that it's much more complex. Upside is that offline runs might still be workable.

Most obvious problem: this isn't known to be possible at all with hg and subversion.

Even if I ignore those problems, the Ansible git module still can't remove stale branches and will not accurately reflect when something changed. We can still either ignore those issues (and note it in docs), or work around to some limited extent.

Even if I ignore those omissions, there are still shortcomings for git itself. By default, github pull request type references will not be fetched. I can work around that for fetching the code, but not for recording the revision (cloning all references is not prohibitive for large repos, but inspecting head for them all is). That means that an implementation for this would be incomplete even for git.

The only alternative is that we allow users to enter their own refspec for the project. This is almost certainly too advanced for most users.

Current plan

For expediency, I hope to get as much done of option 1 as I can. It won't be simple. Parts of its implementation will look like the messy option 2.

[kdelee]

@dsesami I am ready to close, are you?

[dsesami]

@kdelee there was one issue we found (seemingly only UI-side) that @jlmitch5 discussed with me (a field wasn't getting cleared out on the form backend call when swapping between two projects). I don't think that has been merged yet, so I'd like to keep it open until john has confirmed w/ me that he's pushed that + I have ran it once. I will close it afterwards.