New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add switch for "bundle clean" on "store build cache" #2518

Closed
apepper opened this Issue Jul 11, 2014 · 11 comments

Comments

Projects
None yet
7 participants
@apepper

apepper commented Jul 11, 2014

Background

I'm using multiple Gemfiles in one travis run, e.g. ./Gemfile, ./folder_a/Gemfile and ./folder_b/Gemfile. To improve the speed of one run, caching for bundle install is advised. Using directory caching (see http://docs.travis-ci.com/user/caching/#With-a-custom-install-step ) this is already possible.

Limits to this approach

Sadly right before uploading the cache (store build cache), a bundle clean is executed. This means, that all gems, which are not listed in ./Gemfile are removed from the bundle cache. So all gems from ./folder_a/Gemfile and ./folder_b/Gemfile which are not in ./Gemfile are removed. This also implies, that all Gems from folder_a and folder_b need to be installed on every travis run, thus defeating the purpose of a cache.

Workaround

It's possible to use two different bundle cache folders. One for ./Gemfile and the other for all the other Gemfiles. This implies, that whenever something is called from folder_a or folder_b an additional ENV variable needs to be set.

This is cumbersome and error prune, since forgetting to set the correct ENV:

  • installing gems to the wrong cache and therefore being removed from the cache, once bundle clean will run
  • or bundle exec can not be executed, because the gems where installed into another cache folder.

Wish

Please provide a way to disable bundle clean during store build cache. This would allow me to use multiple Gemfiles, while using one cache folder.

I understand, that bundle clean is there for a reason, to ensure, that the cache will not contain unneeded or obsolete files. But it's in our (the customers) interest to keep the cache folder small to improve the speed of a travis run. So a manual travis cache --delete once in awhile would be enough.

@rkh

This comment has been minimized.

Show comment
Hide comment
@rkh

rkh Jul 14, 2014

Member

So, just to verify, using our gemfile directive is not possible in your case? Because then you'd have a separate cache per Gemfile: https://github.com/travis-ci/travis-build/blob/master/lib/travis/build/script/ruby.rb#L15-L18

Member

rkh commented Jul 14, 2014

So, just to verify, using our gemfile directive is not possible in your case? Because then you'd have a separate cache per Gemfile: https://github.com/travis-ci/travis-build/blob/master/lib/travis/build/script/ruby.rb#L15-L18

@apepper

This comment has been minimized.

Show comment
Hide comment
@apepper

apepper Jul 14, 2014

No, it's not an option, because at least in our case in one travis job multiple Gemfile's are needed. E.g. one Gemfile for packaging a gem (with asset precompile and other stuff) and the other Gemfile to use the packaged gem without all the packaging dependencies for an integration test.

apepper commented Jul 14, 2014

No, it's not an option, because at least in our case in one travis job multiple Gemfile's are needed. E.g. one Gemfile for packaging a gem (with asset precompile and other stuff) and the other Gemfile to use the packaged gem without all the packaging dependencies for an integration test.

@roidrage

This comment has been minimized.

Show comment
Hide comment
@roidrage

roidrage Aug 20, 2014

Member

@rkh is there something we can/should do here to improve this?

Member

roidrage commented Aug 20, 2014

@rkh is there something we can/should do here to improve this?

@apepper

This comment has been minimized.

Show comment
Hide comment
@apepper

apepper Nov 25, 2014

Any updates on this? We still have to use workarounds with different bundle paths, which is annoying.

apepper commented Nov 25, 2014

Any updates on this? We still have to use workarounds with different bundle paths, which is annoying.

@kikonen-fiksu

This comment has been minimized.

Show comment
Hide comment
@kikonen-fiksu

kikonen-fiksu Jan 16, 2015

I would be curious also to hear about progress on this.

My scenario:

  • main server repo with tests
    • Thus has its' own Gemfile
  • launches server from another repo as embedded server for testing
    • Thus has its' own Gemfile

=> Enforced "bundle clean" caused that direct built in bundle caching logic doesn't work, and I ended up simply using

cache:
  directories:
  - vendor/bundle

which works partially (but I presume that it will grow in size when gems change?!?)

kikonen-fiksu commented Jan 16, 2015

I would be curious also to hear about progress on this.

My scenario:

  • main server repo with tests
    • Thus has its' own Gemfile
  • launches server from another repo as embedded server for testing
    • Thus has its' own Gemfile

=> Enforced "bundle clean" caused that direct built in bundle caching logic doesn't work, and I ended up simply using

cache:
  directories:
  - vendor/bundle

which works partially (but I presume that it will grow in size when gems change?!?)

@kostia

This comment has been minimized.

Show comment
Hide comment
@kostia

kostia Feb 2, 2015

Any progress on this? Using different directories is annoying, since you have sometimes to manually clear the caches, because the suite is broken for no reason.

kostia commented Feb 2, 2015

Any progress on this? Using different directories is annoying, since you have sometimes to manually clear the caches, because the suite is broken for no reason.

@joshk

This comment has been minimized.

Show comment
Hide comment
@joshk

joshk Feb 23, 2015

Member

@rkh what are your thoughts on this?

Member

joshk commented Feb 23, 2015

@rkh what are your thoughts on this?

@BanzaiMan

This comment has been minimized.

Show comment
Hide comment
@BanzaiMan

BanzaiMan Jul 14, 2015

Member

I think it is reasonable.

cache:
  skip_bundle_clean: true

for example.

Member

BanzaiMan commented Jul 14, 2015

I think it is reasonable.

cache:
  skip_bundle_clean: true

for example.

@rkh

This comment has been minimized.

Show comment
Hide comment
@rkh

rkh Jul 14, 2015

Member

The

cache:
  directories:
  - vendor/bundle

version should work just fine.

but I presume that it will grow in size when gems change?!?

That is true. But it would also be true if we'd implement skip_bundle_clean: true, since bundle clean is used to keep the cache from growing indefinitely.

Member

rkh commented Jul 14, 2015

The

cache:
  directories:
  - vendor/bundle

version should work just fine.

but I presume that it will grow in size when gems change?!?

That is true. But it would also be true if we'd implement skip_bundle_clean: true, since bundle clean is used to keep the cache from growing indefinitely.

@apepper

This comment has been minimized.

Show comment
Hide comment
@apepper

apepper Jul 29, 2015

The

cache:
  directories:
  - vendor/bundle

version should work just fine.

It does. Thanks for the pointer.

apepper commented Jul 29, 2015

The

cache:
  directories:
  - vendor/bundle

version should work just fine.

It does. Thanks for the pointer.

@BanzaiMan

This comment has been minimized.

Show comment
Hide comment
@BanzaiMan
Member

BanzaiMan commented Feb 1, 2017

@BanzaiMan BanzaiMan closed this Feb 1, 2017

i2000s added a commit to i2000s/en that referenced this issue May 9, 2017

y-yagi added a commit to y-yagi/rails that referenced this issue Feb 2, 2018

Avoid bundle clean before caching
I'm not sure cause, but due to the influence of `gem clean`, the expected
gem seems not to be installed correctly.
In order to avoid a test failure due to this, I fixed that `gem clean` not be executed.
Ref: travis-ci/travis-ci#2518 (comment)

This is a workaround. If Travis fixes something, please revert this.

y-yagi added a commit to rails/rails that referenced this issue Feb 2, 2018

Avoid bundle clean before caching
I'm not sure cause, but due to the influence of `gem clean`, the expected
gem seems not to be installed correctly.
In order to avoid a test failure due to this, I fixed that `gem clean` not be executed.
Ref: travis-ci/travis-ci#2518 (comment)

This is a workaround. If Travis fixes something, please revert this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment