Embedded Ansible dependent on SCM for every playbook being run

[sigbjornaib]

Please see this conversation on gitter:

sigbjornaib @sigbjornaib 21:13
For the ansible-runner based Embedded Ansible in Ivanchuck and onwards, it seem like there is a "MIQ(GitRepository#update_repo)" and "MIQ(GitRepository#checkout)" happening every time a playbook is run. Is there a way to cache the git repository, equal to what was available in the AWX based Embedded Ansible used earlier?

Nick LaMuro @NickLaMuro 21:15
@sigbjornaib we do that and keep the repos in /var/www/miq/vmdb/data/git_repos:
https://github.com/ManageIQ/manageiq/blob/a5671aba/app/models/git_repository.rb#L7
the checkout that you see is us taking the "bare repo" that you see in that directory, and actually checkout a specific sha for the ansible run
this means that each run has it's own working directory to work with
we do (effectively) a git pull prior to that checkout on each run in case anything has been updated on the remote, and create the working directory sandbox for each run
hopefully that makes sense

sigbjornaib @sigbjornaib 21:21
Ok, thank you for the quick response. What happens with the job in MIQ if our Bitbucket server is offline when the git pull is running? Will it revert to using only the cached data in /var/www/miq/vmdb/data/git_repos ? And how long is the timeout?

Nick LaMuro @NickLaMuro 21:24
hmmm, good question
well, I found this code:
https://github.com/ManageIQ/manageiq/blob/a5671aba/lib/git_worktree.rb#L455-L463
Which is around our pull code for the "cached" bare git repo:
https://github.com/ManageIQ/manageiq/blob/a5671aba/lib/git_worktree.rb#L326-L328
I would guess is that it most likely would error out if the remote is offline, which probably isn't the answer you were hoping for

sigbjornaib @sigbjornaib 21:32
Ok. I am testing on an appliance just now by adding a firewalld direct rule rejecting all outgoing traffic to the bitbucket server and running a ansible playbook from Service Catalog Item. It seem to be waiting for the bitbucket server to become available. "Provision Step [check_completed] Status [Processed check_completed] Current Retry Number [5]"
Which is not what I was hoping for. I would say it's a rather large setback that after Ivanchuck you are dependent on your Bitbucket/SCM to be available to run Ansible playbooks in MIQ/CFME.

Nick LaMuro @NickLaMuro 21:34
Can you open an issue on GitHub so we can get that issue tracked?

https://gitter.im/ManageIQ/manageiq-providers-embedded_ansible?at=5e9f45d5f8b37f6dac6446d2

[sigbjornaib]

Even after I removed the firewalld rule, restoring the connection to the Bitbucket server, the job does not continue. The current state is: "Provision Step [check_completed] Status [Processed check_completed] Current Retry Number [22]".

[NickLaMuro]

Even after I removed the firewalld rule...

@sigbjornaib Can you please provide reproduction steps in the description instead of just a transcript of the conversation linked? This statement doesn't provide much context if you don't.

[sigbjornaib]

You may test what happens when there is no access to Bitbucket/SCM server by adding the following rule on a MIQ appliance, where 192.168.1.2 is the address of Bitbucket/SCM:

firewall-cmd --direct --add-rule ipv4 filter OUTPUT 1 -o eth0 -d 192.168.1.2 -j REJECT

Then submit an Ansible type Service Catalog Item request and watch the output of the evm.log on the Appliance. A request to update the repository is made, but it never completes and the status of the request under Services -> Requests in the webgui shows the "Current Retry Number" increasing without the job completing.

Remove the rule again:
firewall-cmd --direct --remove-rule ipv4 filter OUTPUT 1 -o eth0 -d 192.168.1.2 -j REJECT

Even after the connection to the Bitbucket/SCM server has been restored, the request never completes and after approx 90 minutes it reaches the State "Finished", however with a Status of "Error".

[gtanzillo]

/cc @NickLaMuro