New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filtering non-cookbook files leave empty directories that are still scp-ed #429

Closed
ryansouza opened this Issue May 5, 2014 · 2 comments

Comments

Projects
None yet
2 participants
@ryansouza
Contributor

ryansouza commented May 5, 2014

We use test-kitchen & kitchen-ec2 as part of our CI testing on travis. By default travis will bundle in to vendor/bundle/ in the cloned repository.

We were getting builds that would appear to hang randomly while transferring files for the cookbook being tested. I SSH-ed in to the box while it was transferring files, ran strace against the scp process, and there I saw that it was copying bunch of "files" in the installed bundle directory. Investigating further, I found that they were actually all empty directories.

Looking at the code it looks like all of the resolved cookbooks are copied to a temp directory and then ignorable files are removed. However, directories that these files were in are left behind, due to filtering non-files from the removal list.

Uploading files to the instance happens in Kitchen::SSH using Net::SCP. This will happily upload those "empty" directories that we don't care about anymore. It also doesn't yield to the progress block for these directories, so it is not obvious this is happening.

The issue is caused by the ~2000 directories that get created by a pretty reasonable set of installed gems, namely berkshelf, foodcritic, chefspec, & test kitchen. For now I've worked around this issue by having bundle install to another location, outside of the work directory.

I'm not sure what the best way to "fix" this would be, so this is just a bug report for now. I think it would make sense to figure out a way to avoid having these directories end up in the upload at all, whether that means a second round of deleting or switching to a white-list copy.

It appears some discussion happened a while back about a better way to upload cookbooks, which might also solve the slowness issue even if these ignorable directories were still uploaded

@englishm

This comment has been minimized.

Show comment
Hide comment
@englishm

englishm Jun 14, 2014

We are also encountering this issue—slowness due to many subdirectories in vendor/bundle being transferred. I'd be interested in hearing about workarounds anyone has found.

englishm commented Jun 14, 2014

We are also encountering this issue—slowness due to many subdirectories in vendor/bundle being transferred. I'd be interested in hearing about workarounds anyone has found.

@ryansouza ryansouza closed this Jun 14, 2014

@ryansouza ryansouza reopened this Jun 14, 2014

@ryansouza

This comment has been minimized.

Show comment
Hide comment
@ryansouza

ryansouza Jun 14, 2014

Contributor

oops

englishm: Our workaround is to avoid bundling in to vendor by overriding travis' bundle command to not include a path

Contributor

ryansouza commented Jun 14, 2014

oops

englishm: Our workaround is to avoid bundling in to vendor by overriding travis' bundle command to not include a path

@sethvargo sethvargo closed this in #530 Jan 21, 2015

@test-kitchen test-kitchen locked and limited conversation to collaborators Nov 16, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.