New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ansible: environment is not restored after module execution in v0.2.1 #309

Closed
alexrussu opened this Issue Jul 18, 2018 · 13 comments

Comments

Projects
None yet
2 participants
@alexrussu

alexrussu commented Jul 18, 2018

Trying to use Mitogen with my old playbooks and I am getting 100% time this error, same place, when getting to a task that pushes to git :

fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["git", "push", "-u", "origin", "master"], "delta": "0:00:00.010616", "end": "2018-07-18 19:17:07.716223", "msg": "non-zero return code", "rc": 128, "start": "2018-07-18 19:17:07.705607", "stderr": "error: cannot run /tmp/tmpY2pVyC: No such file or directory\nfatal: unable to fork", "stderr_lines": ["error: cannot run /tmp/tmpY2pVyC: No such file or directory", "fatal: unable to fork"], "stdout": "", "stdout_lines": []}

task is:

  • name: push to git
    command: "git push -u origin master"
    args:
    chdir: /tmp/git-repo/

ansible-playbook 2.6.1
config file = /opt/svn/OpenEPC/ansible_playbooks/ansible.cfg
configured module search path = [u'/home/aru/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/dist-packages/ansible
executable location = /usr/bin/ansible-playbook
python version = 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609]

@alexrussu alexrussu changed the title from "error: cannot run /tmp/tmpY2pVyC: No such file or directory\nfatal: unable to fork" to ansible: "error: cannot run /tmp/tmpY2pVyC: No such file or directory\nfatal: unable to fork" Jul 18, 2018

@dw

This comment has been minimized.

Owner

dw commented Jul 19, 2018

Hi Alex, thanks for the report. Does the error reproduce if you start the run at that task using --start-at="..."?

I tried to reproduce here but can't so far - on OS X. What is the host and target OS?

@dw

This comment has been minimized.

Owner

dw commented Jul 19, 2018

The only place I can find that 'unable to fork' string within Ansible is in the "service" module, but I see plenty more projects on GitHub, so it's likely not being produced by Ansible.

Aaand looks like Git produces that message all by itself: https://github.com/git/git/blob/b7bd9486b055c3f967a870311e704e3bb0654e4f/connect.c#L1303

I just flew transatlantic so pretty tired, but will have another look tomorrow.

The only question I can think of just now is whether the target has any kind of restrictive environment (Docker container, low memory VM/host machine, weird ulimits set, etc)

@alexrussu

This comment has been minimized.

alexrussu commented Jul 19, 2018

Hi , thanks for the feedback. I tried moving around the task and running it from another role; only if I run it as the first thing in the play then it works ( or manually after). I am running on Ubunbtu 16.04 bare metal.

The "unable to fork" is Git indeed but I think the cause is "error: cannot run /tmp/tmpY2pVyC: No such file or directory".

The speed improvement is impressive but I have to stick to "low speed" at least for a while.

dw added a commit that referenced this issue Jul 19, 2018

dw added a commit that referenced this issue Jul 19, 2018

@dw

This comment has been minimized.

Owner

dw commented Jul 19, 2018

The problem turned out to be really simple -- a regression introduced in d8e0c9e in v0.2.1 due to a missing docstring, led to not resetting the process environment after a module runs, if no custom environment variables were set in the playbook.

1 line fix, +50 lines of tests :)

The change will be on master in about 30 minutes once the tests finish. Sorry for the inconvenience, and thanks a ton for reporting it!

@dw dw changed the title from ansible: "error: cannot run /tmp/tmpY2pVyC: No such file or directory\nfatal: unable to fork" to ansible: environment is not restored after module execution in v0.2.1 Jul 19, 2018

@dw dw closed this in #312 Jul 19, 2018

dw added a commit that referenced this issue Jul 19, 2018

Merge pull request #312 from dw/issue309
issue #309: fix environment cleanup regression.

@dw dw reopened this Jul 20, 2018

@dw

This comment has been minimized.

Owner

dw commented Jul 20, 2018

Hi Alex,

Can you please ensure you're definitely running from the latest copy of master branch. Are you using a fresh Git checkout?

@alexrussu

This comment has been minimized.

alexrussu commented Jul 20, 2018

Hi David,

Yeah, here is my git log with #312 already merged:

commit 422c6e0b994f7f02b054b966360ff86ca3035c6e
Merge: def848f 15d68b3
Author: dw <dw@botanicus.net>
Date:   Thu Jul 19 11:23:20 2018 -0400

    Merge pull request #312 from dw/issue309
    
    issue #309: fix environment cleanup regression.

commit 15d68b3c326d0150fbdcc5b811f6fe64e4ad4a52
Author: David Wilson <dw@botanicus.net>
Date:   Thu Jul 19 10:58:38 2018 -0400

    issue #309: fix environment cleanup regression.
    
    Closes #309.

commit def848f118d0f1466bfc0da3f78e66ec6542bb6e
Merge: fd72720 09d077e
Author: dw <dw@botanicus.net>
Date:   Tue Jul 17 21:34:43 2018 +0100

    Merge pull request #308 from dw/dmw
    
    Dmw

I also checked the code to make sure I got the changes, here is the ansible_mitogen/runner.py file which has the changes from the merge:

290     def revert(self):
291         """
292         Revert changes made by the module to the process environment. This must
293         always run, as some modules (e.g. git.py) set variables like GIT_SSH                                                                   
294         that must be cleared out between runs.
295         """
296         os.environ.clear()
297         os.environ.update(self.original)
298 

Besides using master and including the config with the strategy plugins in Ansible should I do anything else maybe ? Thanks !

@dw

This comment has been minimized.

Owner

dw commented Jul 20, 2018

Nope, that should be all that's required! Let me play with creating another reproduction for your issue -- the old one doesn't work any more :)

Would it be possible to share a description of any Git-related tasks in your playbook.

My reproduction used a 'git pull' step (which internally sets GIT_SSH) followed by the command task mentioned above, but perhaps there is another pattern that also triggers the behaviour.

@alexrussu

This comment has been minimized.

alexrussu commented Jul 25, 2018

Sure ! Thanks for all the support in debugging this !

I have a sequence of 6 tasks which involve git in one yaml file . The intention is to add some content to an empty git repo:

  • name: Git init
    command: git init
    args:
    chdir: "{{ content_location }}"

  • name: add Git remote
    command: git remote add origin ssh://git@...
    args:
    chdir: "{{ content_location }}"

  • name: add content to git
    command: git add .
    args:
    chdir: "{{ content_location }}"

  • name: commit
    command: git commit -m "foo"
    args:
    chdir: "{{ content_location }}"

  • name: add Git branch
    command: git checkout -b {{ branch }}
    args:
    chdir: "{{ content_location }}"

  • name: push Git branch
    command: git push -u origin "{{ branch }}"
    args:
    chdir: "{{ content_location }}"

@alexrussu

This comment has been minimized.

alexrussu commented Aug 7, 2018

Hi. Did you have time so far to check this ? Thanks.

@alexrussu

This comment has been minimized.

alexrussu commented Aug 7, 2018

Just checked again with master@898c06f1b9f1417b9f7c18465bee78eda7df2ec0 and it works fine ! I guess this can be closed . thanks ! note: tag v0.2.2 does not work.

@alexrussu alexrussu closed this Aug 7, 2018

@dw

This comment has been minimized.

Owner

dw commented Aug 7, 2018

Hi Alex!

I'm going to reopen this just so I know to recheck prior to next release -- since we didn't find a root cause for your issue, I'm not sure what fixed it yet :)

Really busy with other work just now, but will be back to the stable branch in a couple more days. Thanks again!

@dw dw reopened this Aug 7, 2018

@alexrussu

This comment has been minimized.

alexrussu commented Aug 7, 2018

No problem. Good luck !

@dw dw added the target:v0.2 label Aug 11, 2018

@dw

This comment has been minimized.

Owner

dw commented Aug 19, 2018

This is now on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://www.freelists.org/list/mitogen-announce

Thanks for reporting this!

@dw dw closed this Aug 19, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment