New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop uploading empty directories #530

Merged
merged 1 commit into from Jan 21, 2015

Conversation

Projects
None yet
3 participants
@whiteley

whiteley commented Oct 23, 2014

Fixes #429

@whiteley

This comment has been minimized.

Show comment
Hide comment
@whiteley

whiteley Jan 21, 2015

kitchen-sync is a great plugin for speeding up transfers, but why are we compressing or pipelining empty directories when this would help all upload methods?

whiteley commented Jan 21, 2015

kitchen-sync is a great plugin for speeding up transfers, but why are we compressing or pipelining empty directories when this would help all upload methods?

sethvargo pushed a commit that referenced this pull request Jan 21, 2015

Seth Vargo
Merge pull request #530 from whiteley/rmdirs
Stop uploading empty directories

@sethvargo sethvargo merged commit 492104a into test-kitchen:master Jan 21, 2015

1 check passed

continuous-integration/travis-ci The Travis CI build passed
Details
@sethvargo

This comment has been minimized.

Show comment
Hide comment
@sethvargo

sethvargo Jan 21, 2015

Contributor

Thanks!

Contributor

sethvargo commented Jan 21, 2015

Thanks!

@L2G

This comment has been minimized.

Show comment
Hide comment
@L2G

L2G Jan 24, 2015

This change is causing new errors for me. See the example below. The path ending with test/integration/joyent/bats is one of the directories that was pruned, but the error suggests that something somewhere else in the code didn't get the message that it was pruned.

-----> Starting Kitchen (v1.3.2.dev)
-----> Converging <prod-redis28-master-centos-66>...
       Preparing files for transfer
       Preparing dna.json
       Resolving cookbook dependencies with Berkshelf 3.1.5...
       Removing non-cookbook files before transfer
       Preparing data_bags
       Preparing environments
       Preparing roles
       Preparing validation.pem
       Preparing client.rb
-----> Chef Omnibus installation detected (install only if missing)
       Transferring files to <prod-redis28-master-centos-66>
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #converge action: [No such file or directory @ rb_file_s_stat - /var/folders/b_/y337by3540l885cbfn8d85ww0000gp/T/prod-redis28-master-centos-66-sandbox-20150123-98432-1736qnt/cookbooks/itoys-tsung/test/integration/joyent/bats]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

L2G commented Jan 24, 2015

This change is causing new errors for me. See the example below. The path ending with test/integration/joyent/bats is one of the directories that was pruned, but the error suggests that something somewhere else in the code didn't get the message that it was pruned.

-----> Starting Kitchen (v1.3.2.dev)
-----> Converging <prod-redis28-master-centos-66>...
       Preparing files for transfer
       Preparing dna.json
       Resolving cookbook dependencies with Berkshelf 3.1.5...
       Removing non-cookbook files before transfer
       Preparing data_bags
       Preparing environments
       Preparing roles
       Preparing validation.pem
       Preparing client.rb
-----> Chef Omnibus installation detected (install only if missing)
       Transferring files to <prod-redis28-master-centos-66>
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #converge action: [No such file or directory @ rb_file_s_stat - /var/folders/b_/y337by3540l885cbfn8d85ww0000gp/T/prod-redis28-master-centos-66-sandbox-20150123-98432-1736qnt/cookbooks/itoys-tsung/test/integration/joyent/bats]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
@whiteley

This comment has been minimized.

Show comment
Hide comment
@whiteley

whiteley Jan 24, 2015

@L2G very interesting, I've been using this for months with cookbooks containing different types of test files including bats without issue. Obviously, you're seeing a problem though and I'd like to help figure it out. Would you be able to post a listing of the contents of your test/ directory? Something like:

$ find test -ls
4355099        0 drwxr-xr-x    4 mwhiteley        wheel                 136 Jan 23 18:20 test
4355100        0 drwxr-xr-x    3 mwhiteley        wheel                 102 Jan 23 18:20 test/fixtures
4355101        0 drwxr-xr-x    3 mwhiteley        wheel                 102 Jan 23 18:20 test/fixtures/test-kitchen
4355102        0 drwxr-xr-x    3 mwhiteley        wheel                 102 Jan 23 18:20 test/fixtures/test-kitchen/scripts
4355103        8 -rw-r--r--    1 mwhiteley        wheel                 384 Jan 23 18:20 test/fixtures/test-kitchen/scripts/install-smartos.sh
4355104        0 drwxr-xr-x    3 mwhiteley        wheel                 102 Jan 23 18:20 test/integration
4355105        0 drwxr-xr-x    3 mwhiteley        wheel                 102 Jan 23 18:20 test/integration/default
4355106        0 drwxr-xr-x    3 mwhiteley        wheel                 102 Jan 23 18:20 test/integration/default/bats
4355107        8 -rw-r--r--    1 mwhiteley        wheel                  50 Jan 23 18:20 test/integration/default/bats/install_a_package.bats

Are you using kitchen-sync or any other kitchen-* plugins? Which driver are you using?

Thanks!

whiteley commented Jan 24, 2015

@L2G very interesting, I've been using this for months with cookbooks containing different types of test files including bats without issue. Obviously, you're seeing a problem though and I'd like to help figure it out. Would you be able to post a listing of the contents of your test/ directory? Something like:

$ find test -ls
4355099        0 drwxr-xr-x    4 mwhiteley        wheel                 136 Jan 23 18:20 test
4355100        0 drwxr-xr-x    3 mwhiteley        wheel                 102 Jan 23 18:20 test/fixtures
4355101        0 drwxr-xr-x    3 mwhiteley        wheel                 102 Jan 23 18:20 test/fixtures/test-kitchen
4355102        0 drwxr-xr-x    3 mwhiteley        wheel                 102 Jan 23 18:20 test/fixtures/test-kitchen/scripts
4355103        8 -rw-r--r--    1 mwhiteley        wheel                 384 Jan 23 18:20 test/fixtures/test-kitchen/scripts/install-smartos.sh
4355104        0 drwxr-xr-x    3 mwhiteley        wheel                 102 Jan 23 18:20 test/integration
4355105        0 drwxr-xr-x    3 mwhiteley        wheel                 102 Jan 23 18:20 test/integration/default
4355106        0 drwxr-xr-x    3 mwhiteley        wheel                 102 Jan 23 18:20 test/integration/default/bats
4355107        8 -rw-r--r--    1 mwhiteley        wheel                  50 Jan 23 18:20 test/integration/default/bats/install_a_package.bats

Are you using kitchen-sync or any other kitchen-* plugins? Which driver are you using?

Thanks!

@L2G

This comment has been minimized.

Show comment
Hide comment
@L2G

L2G Jan 26, 2015

@whiteley: I will see what I can do. I need to make sure I'm not revealing any sensitive info.

Are you interested in the "main" test/ directory, or the one that is called out in that error message (which is different)?

L2G commented Jan 26, 2015

@whiteley: I will see what I can do. I need to make sure I'm not revealing any sensitive info.

Are you interested in the "main" test/ directory, or the one that is called out in that error message (which is different)?

@L2G

This comment has been minimized.

Show comment
Hide comment
@L2G

L2G Jan 26, 2015

Oh, indeed, this turned up something interesting:

11418104        8 lrwxr-xr-x    1 larrygilbert     staff                  15 Oct 31 17:40 test/integration/joyent/bats -> ../default/bats

Maybe both the symlink and its target are being collected by that Dir.glob, the target directory is getting removed, and then when it gets to the symlink (which previously looked like a directory), the symlink doesn't point to anything so an error gets thrown.

L2G commented Jan 26, 2015

Oh, indeed, this turned up something interesting:

11418104        8 lrwxr-xr-x    1 larrygilbert     staff                  15 Oct 31 17:40 test/integration/joyent/bats -> ../default/bats

Maybe both the symlink and its target are being collected by that Dir.glob, the target directory is getting removed, and then when it gets to the symlink (which previously looked like a directory), the symlink doesn't point to anything so an error gets thrown.

@test-kitchen test-kitchen locked and limited conversation to collaborators Nov 16, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.