Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pipeline-ish method using dd for file transfer over SSH (#18642) #18643

Merged
merged 1 commit into from Jan 19, 2017

Conversation

Projects
None yet
7 participants
@agaffney
Copy link
Contributor

commented Nov 28, 2016

ISSUE TYPE
  • Feature Pull Request
COMPONENT NAME

ssh connection plugin

ANSIBLE VERSION
ansible 2.3.0 (file_transfer_pipeline 44cf3bb916) last updated 2016/11/27 21:46:18 (GMT -600)
SUMMARY

This commit adds a new dd-based method for pushing/pulling files over SSH. It also introduces a new config option (transfer_method) that overrides scp_if_ssh if set.

This new transfer method is useful in cases where neither sftp or scp are functional, or where you're pulling dirty tricks that make paths not line up between the ssh commands and the file transfer commands.

@agaffney

This comment has been minimized.

Copy link
Contributor Author

commented Nov 28, 2016

The failed tests seem to only be on Windows and unrelated to this change

@abadger

This comment has been minimized.

Copy link
Member

commented Nov 28, 2016

I had suggested this at one time but @bcoca pointed out that scp and sftp are more robust in the face of errors than piping would be. OTOH, I think I was talking about whole-sale replacing all of the scp and sftp hackery with this method (whereas you're making it one of several options) so it might be okay this way. I'll ping bcoca about it and if it's controversial we can discuss it in a meeting.

@agaffney

This comment has been minimized.

Copy link
Contributor Author

commented Nov 28, 2016

Yeah, I'd considered doing it as a replacement, but I didn't think that would fly. This way, it can be explicitly enabled or serve as a fallback when sftp and scp have already failed.

@agaffney

This comment has been minimized.

Copy link
Contributor Author

commented Nov 28, 2016

I'm considering changing the method name from dd to pipeline. Thoughts on that?

@alikins

This comment has been minimized.

Copy link
Contributor

commented Nov 28, 2016

What systems have ssh but don't have functional sftp or scp?

or where you're pulling dirty tricks that make paths not line up between the ssh commands and the file transfer commands

Are there examples of when this would be useful?

@agaffney

This comment has been minimized.

Copy link
Contributor Author

commented Nov 28, 2016

Normally, a system with sshd would also have scp or sftp. However, if you delete the "helper" binaries for those two, neither are functional. The "dirty tricks" I'm referring to is a hack I put together to do management of a chroot on a remote host using a wrapper script with chroot /mnt/gentoo /bin/sh "$@" and setting ansible_shell_executable=/root/chroot_wrapper.sh.

@alikins

This comment has been minimized.

Copy link
Contributor

commented Nov 28, 2016

Normally, a system with sshd would also have scp or sftp. However, if you delete the "helper" binaries for those two, neither are functional.

Right, but are there cases where that happens for a reason? Is this a problem that needs to be solved?

The "dirty tricks" I'm referring to is a hack I put together to do management of a chroot on a remote host using a wrapper script with chroot /mnt/gentoo /bin/sh "$@" and setting ansible_shell_executable=/root/chroot_wrapper.sh

That seems pretty esoteric to me. Is this something that could be done with a custom connection plugin?
(or custom shell plugin?)

@agaffney

This comment has been minimized.

Copy link
Contributor Author

commented Nov 28, 2016

To be honest, the bit about scp/sftp not working was my attempt to justify this outside of the needs of my crazy hack, but it is legitimate. It's entirely possible that someone could put together an "embedded" system that has neither helper binary.

A custom connection plugin could potentially be used in the case of my crazy remote chroot hack, but it seems silly when all the pieces are already there to allow this with built-in bits. It would also complicate things, since I'd need to select a different connection plugin on a per-task basis. The only thing that doesn't work right now with my hack is copying files, simply because scp wants to put them outside the chroot.

@abadger

This comment has been minimized.

Copy link
Member

commented Nov 28, 2016

The need is legitimate even if not very common. Random users have asked me on IRC about doing this at their site which doesn't have scp or sftp available.

@abadger

This comment has been minimized.

Copy link
Member

commented Nov 28, 2016

@agaffney, yeah.. dd isn't the best name for it. pipeline somewhat collides with pipelining mode, though. Maybe "piped"?

@agaffney

This comment has been minimized.

Copy link
Contributor Author

commented Nov 28, 2016

I have no strong feelings about the naming. "piped" could work, but I think we can do better :)

@bcoca

This comment has been minimized.

Copy link
Member

commented Nov 29, 2016

i'm fine with it as an option, my objection, as stated above, would be as the only system as it removes a lot of the consistency scp/sftp have built in.

@alikins

This comment has been minimized.

Copy link
Contributor

commented Dec 1, 2016

To be honest, the bit about scp/sftp not working was my attempt to justify this outside of the needs of my crazy hack, but it is legitimate. It's entirely possible that someone could put together an "embedded" system that has neither helper binary.

The need is legitimate even if not very common. Random users have asked me on IRC about doing this at their site which doesn't have scp or sftp available.

I mainly just looking for specific scenarios where this feature would be used. Use cases or user stories (to use verboten terminology). Something that is impossible now that would possible after adding it. Something to base a test case on.

@alikins

This comment has been minimized.

Copy link
Contributor

commented Dec 1, 2016

Some vague questions about potential design/impl...

Would it be accurate to say that there are cases where a connection plugin can and should support multiple file transfer mechanisms? (ssh has sftp/scp, so I assume yes)

Is the goal to generically support multiple file transfer mechanisms?

How 'pluggable' do they need to be?

How will the mechanism be selected?

Should file transfer api be split into a separate plugin? (The piped/dd file transfer approach sounds like it could potentially be used from other connection plugins)

@agaffney

This comment has been minimized.

Copy link
Contributor Author

commented Dec 1, 2016

The piped/dd method is already used by other connection plugins. I copied the concept and some of the code from the docker and chroot connection plugins.

@agaffney agaffney force-pushed the agaffney:file_transfer_pipeline branch Dec 1, 2016

@agaffney

This comment has been minimized.

Copy link
Contributor Author

commented Dec 1, 2016

I rebased my branch to resolve a merge conflict

@bcoca

This comment has been minimized.

Copy link
Member

commented Dec 1, 2016

I'm thinking that this should also be host specific ansible_transfer/ansible_ssh_transfter?
overriding the main config (nicer if not ssh specific)

@agaffney

This comment has been minimized.

Copy link
Contributor Author

commented Dec 1, 2016

Yeah, I'd considered that. However, since scp_if_ssh isn't able to be used that way (I didn't see a mapping in the play context class), I didn't add that for this, either. I can easily add it if you think this new option should be overrideable via the inventory file (I'm assuming this magic happens via the play context class).

@bcoca

This comment has been minimized.

Copy link
Member

commented Dec 1, 2016

yes, that is an example i wish to depart from. In play context we'll need to update the magic vars dict

@agaffney

This comment has been minimized.

Copy link
Contributor Author

commented Dec 1, 2016

I guess the connection plugin would then need to check both the play context attribute and C.DEFAULT_... to determine the behavior to have?

@agaffney agaffney force-pushed the agaffney:file_transfer_pipeline branch Dec 2, 2016

@agaffney

This comment has been minimized.

Copy link
Contributor Author

commented Dec 2, 2016

The code now supports setting the transfer method via a variable. I tested with:

$ ansible nuc1 -e ansible_ssh_transfer_method=piped -m copy -a 'src=/etc/fstab dest=/tmp/foo' -vvv

Note: I also changed dd to piped

@agaffney agaffney force-pushed the agaffney:file_transfer_pipeline branch Dec 6, 2016

@agaffney

This comment has been minimized.

Copy link
Contributor Author

commented Dec 6, 2016

Rebased from devel after repo merge

@mattclay mattclay force-pushed the ansible:devel branch from a236cbf to c709b22 Dec 8, 2016

@agaffney agaffney force-pushed the agaffney:file_transfer_pipeline branch Dec 9, 2016

@agaffney

This comment has been minimized.

Copy link
Contributor Author

commented Dec 9, 2016

Rebased again after repo merge

@mattclay

This comment has been minimized.

Copy link
Member

commented Jan 19, 2017

@agaffney Are you going to update the connection tests to test the new transfer method? It should be easy now that I've updated the connection test script to support it.

@agaffney

This comment has been minimized.

Copy link
Contributor Author

commented Jan 19, 2017

Yes, I've just been lazy.

@mattclay
Copy link
Member

left a comment

Looks good, just needs the updated tests.

@jctanner

This comment has been minimized.

Copy link
Member

commented Jan 19, 2017

bot_status

@ansibot

This comment has been minimized.

Copy link
Contributor

commented Jan 19, 2017

waiting_on: agaffney

component: None
supported_by: core
needs_info: False
shippable_status: success
mergeable_state: clean
maintainer_shipits: False
community_shipits: False
ansible_shipits: False

click here for bot help

@agaffney agaffney force-pushed the agaffney:file_transfer_pipeline branch to ae7dcfe Jan 19, 2017

@bcoca bcoca merged commit ac51266 into ansible:devel Jan 19, 2017

1 check passed

Shippable Run 10022 status is SUCCESS.
Details

@agaffney agaffney deleted the agaffney:file_transfer_pipeline branch Jan 19, 2017

@mattclay

This comment has been minimized.

Copy link
Member

commented Jan 19, 2017

@agaffney I verified that the new tests are running correctly:

ansible-test integration connection/connection_ssh -vvv --docker 2>/dev/null | grep 'dd '

Yields the following ansible output showing dd being used:

<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866452.15-247321708972467/source bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd if=/tmp/ansible-remote-汉语/汉语.txt bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866453.5-108808061542992/file.py bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866453.76-170399041276255/file.py bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866454.02-12882446438043/stat.py bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866454.26-276255720588566/source bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866454.26-276255720588566/copy.py bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866454.75-233261733152283/stat.py bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd if=/tmp/ansible-remote-汉语/汉语.txt bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866455.22-25863813271812/file.py bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866456.95-11103622012736/source bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd if=/tmp/ansible-remote-??/??.txt bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866458.28-35281426781343/file.py bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866458.54-112837234143683/file.py bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866458.81-255503860861843/stat.py bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866459.05-39824807938061/source bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866459.05-39824807938061/copy.py bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866459.54-143323119482494/stat.py bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd if=/tmp/ansible-remote-??/??.txt bs=65536'
<localhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r localhost 'dd of=/root/.ansible/tmp/ansible-tmp-1484866460.03-205028058966366/file.py bs=65536'
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.