Slow file transference

[rfhayashi]

Every time I do a converge, it takes a long time until all cookbooks are transferred. I think this is more of a kitchen + my internet connection issue but it shows up when using a remote driver like kitchen-ec2. As I could see, each file is copied one by one. Perhaps if everything was compressed in just one file we could improve on that. I'm not familiar with kitchen codebase, so I'd just like to know which component is exactly responsible for that, so I could try to work on it. Or could you suggest another way of solving that?

[iiro]

👍

Bit of an issue... Wish I had even suggestions...

This saved me for now: https://github.com/coderanger/kitchen-sync

[iiro]

This became an issue again after Test-Kitchen 1.4 was released - as kitchen-sync does not work anymore.

I left a comment here: coderanger/kitchen-sync#8

Is this an issue of Test-Kitchen really (the slow sync) of kitchen-ec2?

[iiro]

@tyler-ball any chance to get something done for milestone 1.0.0 on this one - or is this an issue within
Test-Kitchen?

update Test-Kitchen's transport I assume...

[tyler-ball]

Yeah, I think this would be an update to (or brand new) transport added to Test Kitchen. Paging @fnichol for his opinion, but I think this seems completely reasonable. We should probably re-file this against Test Kitchen and just reference back to this bug so we know a use case is for slow AWS uploads.

We do some compressing/uncompressing for WinRM now so we could use that transport as a pattern. I also see that @coderanger updated kitchen-sync so it should be working with TK 1.4.

[jkerry]

The discussion might be dead, but just so there's another data point: the transfer time to windows guests from a windows host over a vsphere cloud is also incredibly long ( even on the same domain ). The same files transfered over windows remote directory copy is an order of magnitude faster.

[iiro]

Hopefully it's not completely dead :)

On linux perspective: Test-Kitchen 1.4+ works using kitchen-sync only with sftp; the initial sync is still very slow (for us - pretty unusable). Gotta use ChefDK 0.4.0 with kitchen-sync and rsync for best results still.

The compressing/uncompressing probably solves this yes...

[iiro]

There is a PR available to address this on Test-Kitchen (as this is a TK issue, not kitchen-ec2 itself): test-kitchen/test-kitchen#845

[cheeseplus]

In that PR we re-iterated that any alternate transfer methods should start as plugins like kitchen-sync and if they become popular/widely used we may bring them into core. We are trying to keep TK as lean as possible until we can triage the existing mountain of issues. As this discussion is already duplicated I'm closing this one.

[brettcave]

Without opening a new issue, I wanted to jump in on this one to discuss a slightly different approach that may be feasible. Some of our testing to EC2 happens over slow links, and it's super unproductive to wait for our full chef repo to sync. e.g. with no files changed:

~/chef-repo $ time rsync -rz -e "ssh -i /path/to/rsa_key.pem" --rsync-path="sudo rsync" cookbooks notrooot@ec2host:/tmp/kitchen/cookbooks/
real    0m5.995s

vs

[2016-02-26T07:14:50.844501 #35]  INFO -- my-test-target-ubuntu-1404: Transferring files to <my-test-target-ubuntu-1404>
[2016-02-26T07:43:43.673175 #35]  INFO -- my-test-target-ubuntu-1404: Starting Chef Client, version 12.7.2

6 seconds vs 39 minutes.

Would an acceptable approach within EC2 to be that instead of hard wiring a specific provider or framework for syncing, use the current method but allow for parameterized overriding of transfer mechanism.... having a look at ruby-rsync it uses IO.open with a provided command. My thinking is something like this in the kitchen config yml:

transport:
  sync_command: rsync
  sync_options: '-rz -e "ssh -i #{ssh_key}" --rsync-path="sudo rsync"' # can drop in excludes here if i need.
  sync_source_glob: # the default glob on this could be something similar to what kitchen already uses
sync_target_format: #{target.username}@#{target_host}:/tmp/kitchen

I haven't delved into the code and not sure if this sort of approach is feasible, just throwing out an idea. Also, not sure if this might be more appropriate for TK, or something specifically aimed at TK-ec2. The idea behind this approach is to not introduce anything that isn't well established into the core but still allow hooks into alternatives.

Thoughts / feedback?

[coderanger]

@brettcave kitchen-sync includes an Rsync-based transport.

[jkerry]

Do we have any sensible windows guest solution for this? I can't seem to find one ( or a functioning ssh implementation for windows that'll support kitchen-sync )

[coderanger]

@jkerry Nope. I probably won't bother until we reify TK vs. Train given that they have slightly different WinRM file transfer implementations. Also notable that WinRM doesn't actually include a file transfer API, TK (and Train) use the equiv of cat file | base64 | ssh "unbase64 > file" so making that faster might be "fun".

[mwrock]

Just an FYI that as of TK 1.6 (still pending release) Test-Kitchen and Train will be using the same underlying file copy gem winrm-fs. Also, @sneal and I are currently working to get a productionized PSRP implementation inside of the WinRM gem. This has the potential to likely dramatically improve remote file transfer performance over windows and as the implementation matures, we will port it into winrm-fs.

[cheeseplus]

That sounds dope @mwrock