ansible: environment is not restored after module execution in v0.2.1

[alexrussu]

Trying to use Mitogen with my old playbooks and I am getting 100% time this error, same place, when getting to a task that pushes to git :

fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["git", "push", "-u", "origin", "master"], "delta": "0:00:00.010616", "end": "2018-07-18 19:17:07.716223", "msg": "non-zero return code", "rc": 128, "start": "2018-07-18 19:17:07.705607", "stderr": "error: cannot run /tmp/tmpY2pVyC: No such file or directory\nfatal: unable to fork", "stderr_lines": ["error: cannot run /tmp/tmpY2pVyC: No such file or directory", "fatal: unable to fork"], "stdout": "", "stdout_lines": []}

task is:

• name: push to git
  command: "git push -u origin master"
  args:
  chdir: /tmp/git-repo/

ansible-playbook 2.6.1
config file = /opt/svn/OpenEPC/ansible_playbooks/ansible.cfg
configured module search path = [u'/home/aru/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/dist-packages/ansible
executable location = /usr/bin/ansible-playbook
python version = 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609]

[dw]

Hi Alex, thanks for the report. Does the error reproduce if you start the run at that task using --start-at="..."?

I tried to reproduce here but can't so far - on OS X. What is the host and target OS?

[dw]

The only place I can find that 'unable to fork' string within Ansible is in the "service" module, but I see plenty more projects on GitHub, so it's likely not being produced by Ansible.

Aaand looks like Git produces that message all by itself: https://github.com/git/git/blob/b7bd9486b055c3f967a870311e704e3bb0654e4f/connect.c#L1303

I just flew transatlantic so pretty tired, but will have another look tomorrow.

The only question I can think of just now is whether the target has any kind of restrictive environment (Docker container, low memory VM/host machine, weird ulimits set, etc)

[alexrussu]

Hi , thanks for the feedback. I tried moving around the task and running it from another role; only if I run it as the first thing in the play then it works ( or manually after). I am running on Ubunbtu 16.04 bare metal.

The "unable to fork" is Git indeed but I think the cause is "error: cannot run /tmp/tmpY2pVyC: No such file or directory".

The speed improvement is impressive but I have to stick to "low speed" at least for a while.

[dw]

The problem turned out to be really simple -- a regression introduced in d8e0c9e in v0.2.1 due to a missing docstring, led to not resetting the process environment after a module runs, if no custom environment variables were set in the playbook.

1 line fix, +50 lines of tests :)

The change will be on master in about 30 minutes once the tests finish. Sorry for the inconvenience, and thanks a ton for reporting it!

[dw]

Hi Alex,

Can you please ensure you're definitely running from the latest copy of master branch. Are you using a fresh Git checkout?

[alexrussu]

Hi David,

Yeah, here is my git log with #312 already merged:

commit 422c6e0b994f7f02b054b966360ff86ca3035c6e
Merge: def848f 15d68b3
Author: dw <dw@botanicus.net>
Date:   Thu Jul 19 11:23:20 2018 -0400

    Merge pull request #312 from dw/issue309
    
    issue #309: fix environment cleanup regression.

commit 15d68b3c326d0150fbdcc5b811f6fe64e4ad4a52
Author: David Wilson <dw@botanicus.net>
Date:   Thu Jul 19 10:58:38 2018 -0400

    issue #309: fix environment cleanup regression.
    
    Closes #309.

commit def848f118d0f1466bfc0da3f78e66ec6542bb6e
Merge: fd72720 09d077e
Author: dw <dw@botanicus.net>
Date:   Tue Jul 17 21:34:43 2018 +0100

    Merge pull request #308 from dw/dmw
    
    Dmw

I also checked the code to make sure I got the changes, here is the ansible_mitogen/runner.py file which has the changes from the merge:

290     def revert(self):
291         """
292         Revert changes made by the module to the process environment. This must
293         always run, as some modules (e.g. git.py) set variables like GIT_SSH                                                                   
294         that must be cleared out between runs.
295         """
296         os.environ.clear()
297         os.environ.update(self.original)
298 

Besides using master and including the config with the strategy plugins in Ansible should I do anything else maybe ? Thanks !

[dw]

Nope, that should be all that's required! Let me play with creating another reproduction for your issue -- the old one doesn't work any more :)

Would it be possible to share a description of any Git-related tasks in your playbook.

My reproduction used a 'git pull' step (which internally sets GIT_SSH) followed by the command task mentioned above, but perhaps there is another pattern that also triggers the behaviour.

[alexrussu]

Sure ! Thanks for all the support in debugging this !

I have a sequence of 6 tasks which involve git in one yaml file . The intention is to add some content to an empty git repo:

• name: Git init
  command: git init
  args:
  chdir: "{{ content_location }}"

• name: add Git remote
  command: git remote add origin ssh://git@...
  args:
  chdir: "{{ content_location }}"

• name: add content to git
  command: git add .
  args:
  chdir: "{{ content_location }}"

• name: commit
  command: git commit -m "foo"
  args:
  chdir: "{{ content_location }}"

• name: add Git branch
  command: git checkout -b {{ branch }}
  args:
  chdir: "{{ content_location }}"

• name: push Git branch
  command: git push -u origin "{{ branch }}"
  args:
  chdir: "{{ content_location }}"

[alexrussu]

Hi. Did you have time so far to check this ? Thanks.

[alexrussu]

Just checked again with master@898c06f1b9f1417b9f7c18465bee78eda7df2ec0 and it works fine ! I guess this can be closed . thanks ! note: tag v0.2.2 does not work.

[dw]

Hi Alex!

I'm going to reopen this just so I know to recheck prior to next release -- since we didn't find a root cause for your issue, I'm not sure what fixed it yet :)

Really busy with other work just now, but will be back to the stable branch in a couple more days. Thanks again!

[alexrussu]

No problem. Good luck !

[dw]

This is now on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://networkgenomics.com/mail/mitogen-announce/

Thanks for reporting this!