Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

local_action running on jump host when connection delegation is enabled? #251

Closed
Aeon opened this issue May 18, 2018 · 7 comments
Closed

local_action running on jump host when connection delegation is enabled? #251

Aeon opened this issue May 18, 2018 · 7 comments

Comments

@Aeon
Copy link

@Aeon Aeon commented May 18, 2018

Hi, and thank you for the awesome project

Trying to get it set up for testing with mitogen_via option, and immediately running into the following when I run my playbook against a small group of hosts:

fatal: [morty2]: UNREACHABLE! => {"changed": false, "msg": "error occurred on host bastion1: Child start failed: [Errno 2] No such file or directory. Command was: \"/webapps/xyzzy/venv/bin/python2.7\" \"-c\" \"import codecs,os,sys;_=codecs.decode;exec(_(_(\\\"eNpd0FFLwzAQB/Bn+yl8u4SF0nRjaKGgFPG9D/ZBRbr0psGsCWm7OD+9Nzo09S2/+1/uSGrRlHZInXbIeOJFiKT314S99Z+MF8kVnbvJ5SwTMsv4n2sRy1MqZytjB2R1DB+jiRFmYH/U3vbPcF8/PmXwWg4nKn6hmsZ2Z3DuIRq2DMSSK2AHPdp37ItBfWA3GfR3rj39Ih3UMW2dM5gqeyhyudnecuAJjQ9ej8ikgIcqe+mpVpVv7PwRnXXY0/PB74CnHtuOSSlvtlzAt3bz5UtTIyAA/ctlWBVnYZnBJpfrNS1aVf/Xy/P6Hzv/gWU=\\\".encode(),\\\"base64\\\"),\\\"zip\\\"))\"", "unreachable": true}
fatal: [morty7]: UNREACHABLE! => {"changed": false, "msg": "error occurred on host bastion1: Child start failed: [Errno 2] No such file or directory. Command was: \"/webapps/xyzzy/venv/bin/python2.7\" \"-c\" \"import codecs,os,sys;_=codecs.decode;exec(_(_(\\\"eNpd0FFLwzAQB/Bn+yl8u4SF0nRjaKGgFPG9D/ZBRbr0psGsCWm7OD+9Nzo09S2/+1/uSGrRlHZInXbIeOJFiKT314S99Z+MF8kVnbvJ5SwTMsv4n2sRy1MqZytjB2R1DB+jiRFmYH/U3vbPcF8/PmXwWg4nKn6hmsZ2Z3DuIRq2DMSSK2AHPdp37ItBfWA3GfR3rj39Ih3UMW2dM5gqeyhyudnecuAJjQ9ej8ikgIcqe+mpVpVv7PwRnXXY0/PB74CnHtuOSSlvtlzAt3bz5UtTIyAA/ctlWBVnYZnBJpfrNS1aVf/Xy/P6Hzv/gWU=\\\".encode(),\\\"base64\\\"),\\\"zip\\\"))\"", "unreachable": true}
fatal: [morty4]: UNREACHABLE! => {"changed": false, "msg": "error occurred on host bastion1: Child start failed: [Errno 2] No such file or directory. Command was: \"/webapps/xyzzy/venv/bin/python2.7\" \"-c\" \"import codecs,os,sys;_=codecs.decode;exec(_(_(\\\"eNpd0FFLwzAQB/Bn+yl8u4SF0nRjaKGgFPG9D/ZBRbr0psGsCWm7OD+9Nzo09S2/+1/uSGrRlHZInXbIeOJFiKT314S99Z+MF8kVnbvJ5SwTMsv4n2sRy1MqZytjB2R1DB+jiRFmYH/U3vbPcF8/PmXwWg4nKn6hmsZ2Z3DuIRq2DMSSK2AHPdp37ItBfWA3GfR3rj39Ih3UMW2dM5gqeyhyudnecuAJjQ9ej8ikgIcqe+mpVpVv7PwRnXXY0/PB74CnHtuOSSlvtlzAt3bz5UtTIyAA/ctlWBVnYZnBJpfrNS1aVf/Xy/P6Hzv/gWU=\\\".encode(),\\\"base64\\\"),\\\"zip\\\"))\"", "unreachable": true}
...

Of course this is because /webapps/xyzzy/venv/bin/python2.7 does not exist on the jump host. However, since it's local_action, shouldn't it still run on localhost?

For reference, the local action is a initial step to ensure that a network-based disk image is mounted before we copy data off it to remote hosts. Looks something like this:

  - name: mount latest source.dmg
    local_action:
      module: shell
      args: |
        if [[ -d /tmp/Project ]]; then
          hdiutil detach /tmp/Project
        fi
        hdiutil attach /Corp/Source/Images/Current/Project.dmg -nobrowse -mountpoint /tmp/Project
    tags:
      - preflight

But I am pretty sure that the content of the action is irrelevant, the issue is that it's trying to execute on jumphost. It would actually even work there, probably, if the interpreter path was correct.

I tried setting ansible_python_interpreter config var for the jumphost, but that did nothing.

Thanks again
Anton

@dw
Copy link
Member

@dw dw commented May 18, 2018

Great catch! That's super broken :) Give me 15 minutes

dw added a commit that referenced this issue May 18, 2018
This needs more work -- pretty certain that python_path and suchlike are
coming from the wrong place. Possibly we need another config_from_..()
specialized for delegate_to.
@dw
Copy link
Member

@dw dw commented May 18, 2018

Hi there,

I've pushed a partial fix to issue251 branch, but it looks like it'll need more work to correctly derive variables in every case. My head is swimming in issues with mainline Ansible this week, so possibly things will just work already (Mitogen local connections should inherit the interpreter used to start Ansible, but not certain of that)

@dw dw added the user-reported label Jun 10, 2018
@ayaz
Copy link

@ayaz ayaz commented Jun 29, 2018

Removed my comment. Created #291 instead.

dw added a commit that referenced this issue Aug 10, 2018
This needs more work -- pretty certain that python_path and suchlike are
coming from the wrong place. Possibly we need another config_from_..()
specialized for delegate_to.
@dw dw added the target:v0.2 label Aug 11, 2018
dw added a commit that referenced this issue Aug 18, 2018
This needs more work -- pretty certain that python_path and suchlike are
coming from the wrong place. Possibly we need another config_from_..()
specialized for delegate_to.
@dw
Copy link
Member

@dw dw commented Aug 19, 2018

This is now on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://networkgenomics.com/mail/mitogen-announce/

Thanks for reporting this!

@dw dw closed this Aug 19, 2018
@michalmedvecky
Copy link

@michalmedvecky michalmedvecky commented Sep 13, 2018

I came here via google, I have a similar problem with 0.2.2:

error occurred on host 55a78418c587: Child start failed: [Errno 2] No such file or directory. Command was: sudo -u root -H /usr/bin/python2 -c "import codecs,os,sys;_=codecs.decode;exec(_(_(\"eNqFkEtrwzAQhM/xr/BtJSISOQ/iGgQtOZQeSsGU+tCG4ofcijqSkJ246a/vxgnETg+97cfO7gwTs0SYemKVlYR6jrU9UqWPUBr3RWjkjXAudnZGOAs4pxeOWZ8cboMT55WpJYn74PqQ9KFFQMP6gPZV2qDr1hfChyJ1rdLgp7rolvJb5rsmzSrZrae72k0zpaf20HwaDZhzdCUbi+5wL12tjH6N5pvOVuq9cshwF9+/cNiI4dlJg1iR4YINcQxkqxrzIXXkjGlul8t0FS6CMF+Gqyi84RSoh59apxpJAgaPD89PnPM3DZggNwUWTb21eCfHqgtjpcaCwWVAJ06mBQlmizmnDH6UxU+lFRddwqDN4Nh+ac8G624+NXqlbv9T/00ZDFL+AoQMrM4=\".encode(),\"base64\"),\"zip\"))"

This happens when my playbook runs a local_task with become: true but already runs under root (in a container)

@dw
Copy link
Member

@dw dw commented Jan 21, 2019

This 'fix' for this is causing breakage elsewhere (e.g. #434)

dw added a commit that referenced this issue Jan 22, 2019
This refactors connection.py to pull the two huge dict-building
functions out into new transport_transport_config.PlayContextSpec and
MitogenViaSpec classes, leaving a lot more room to breath in both files
to figure out exactly how connection configuration should work.

The changes made in 1f21a30 / 3d58832 are updated or completely removed,
the original change was misguided, in a bid to fix connection delegation
taking variables from the wrong place when delegate_to was active.

The Python path no longer defaults to '/usr/bin/python', this does not
appear to be Ansible's normal behaviour. This has changed several times,
so it may have to change again, and it may cause breakage after release.

Connection delegation respects the c.DEFAULT_REMOTE_USER whereas the
previous version simply tried to fetch whatever was in the
'ansible_user' hostvar. Many more connection delegation variables closer
match vanilla's handling, but this still requires more work. Some of the
variables need access to the command line, and upstream are in the
process of changing all that stuff around.
dw added a commit that referenced this issue Jan 22, 2019
@dw
Copy link
Member

@dw dw commented Jan 22, 2019

@michalmedvecky I added an explicit test for your case in the current tree and can confirm it's gone :) Sorr for the wait!


This should now be fixed on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://www.freelists.org/list/mitogen-announce

If you are still experiencing the problem on the current master branch, please do not hesitate to reopen this issue.

Thanks for reporting this!

@dw dw closed this Jan 22, 2019
dw added a commit that referenced this issue Jan 22, 2019
* origin/dmw:
  issue #404: add to Changelog.
  issue #251: readd to Changelog.
  tests: add exact test for issue 251; closes #251.
  issue #412: pad out debugging docs, add get_stack to changelog.
  issue #412: force-verbose output for mitogen_get_stack.
  issue #412: promote "mitogen_get_stack" to the main extension.
  issue #412: add docstrings/boilerplate to transport_config.py.
  issue #251, #412, #434: fix connection configuration brainwrong
  issue #434: tests: set a default remote_user in ansible.cfg.
  tests: CI should symlink all contents of ansible/hosts/
  ansible: fix test failure during process exit.
  tests: use assert_equal in more places.
  tests: make assert_equal work on newer Ansibles.
  tests: convert stack_construction.yml to assert_equal.
  tests: make fork_histogram optional
  tests: use assert_equal in delegate_to_template.yml.
  tests: import assert_equal action.
  tests: rename 'delegation/' to 'connection_delegation/'
  core: replace ancient YOLO loop in fire().
  tests: some more utility function tests + flake8.
  tests: clean up / deduplicate Ansible inventory.
  tests: add some more helper function tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants