Conservative Bundle Updates #122

Closed
chrismo opened this Issue Aug 23, 2016 · 42 comments

Projects

None yet

7 participants

@chrismo
Member
chrismo commented Aug 23, 2016 edited

Conservative Bundle Updates

Starting with 1.13.0.rc.2 (released on 2016 Aug 21), bundle update received some new options to support conservative updates (bundler/bundler#4676), as previously implemented in the bundler-patch plugin. The plan is to leave the new options undocumented and unsupported in 1.13 to give it a chance to be used and flush out bugs and iron out some design decisions.

Part of that work before 1.14 (or maybe a later version) will be to properly document bundle update as well as the Conservative Updating section of bundle install, but right now we need a placeholder document showing the new stuff so folks know how to use it. Posting it here also allows for comments & discussion.

Usage

The default bundle update behavior remains untouched. Use of the new --patch or --minor options will invoke the new conservative update behavior.

"Conservative" means it will sort all available versions to prefer the latest releases from the current version, then the latest minor releases and then the latest major releases. (Examples below.)

"Prefer" means that no available versions are removed from consideration, to help ensure a suitable dependency graph can be reconciled. This does mean some gems cannot be upgraded or may be upgraded to unexpected versions. NOTE: There is a --strict option which will remove versions from consideration, see below.

Gem requirements as defined in the Gemfile will still define what versions are available. The new conservative behavior controls the preference order of those versions.

For example, if gem 'foo' is locked at 1.0.2, with no gem requirement defined in the Gemfile, and versions 1.0.3, 1.0.4, 1.1.0, 1.1.1, 2.0.0 all exist, the default order of preference will be "1.0.4, 1.0.3, 1.0.2, 1.1.1, 1.1.0, 2.0.0".

In the same example, if gem 'foo' has a requirement of '~> 1.0', version 2.0.0 will be removed from consideration as always.

With no gem names provided on the command line, all gems will be unlocked and open for updating.

$ bundle update --patch 

A list of gem names can be passed to restrict to just those gems.

$ bundle update --patch foo bar
  • --patch option will give preference for release/patch versions, then minor, then major.
  • --minor option will give preference for minor versions over release versions, then major versions.
  • --major option will give preference for major versions over minor or release versions. This is the default behavior currently, so this flag is cosmetic for now. Bundler 2.0 will likely make --patch the default behavior.
  • --strict option will actually remove from consideration versions outside either the current release (or minor version if --minor specified). This increases the chances of Bundler being unable to reconcile the dependency graph and in some cases could even raise a VersionConflict.

Examples

Single Gem

Requirements Locked Available Option Result
foo 1.4.3 1.4.4, 1.4.5, 1.5.0, 1.5.1, 2.0.0 --patch 1.4.5
foo 1.4.3 1.4.4, 1.4.5, 1.5.0, 1.5.1, 2.0.0 --minor 1.5.1
foo 1.4.3 1.4.4, 1.4.5, 1.5.0, 1.5.1, 2.0.0 --major 2.0.0

Two Gems

Given the following gem specifications:

  • foo 1.4.3, requires: ~> bar 2.0
  • foo 1.4.4, requires: ~> bar 2.0
  • foo 1.4.5, requires: ~> bar 2.1
  • foo 1.5.0, requires: ~> bar 2.1
  • foo 1.5.1, requires: ~> bar 3.0
  • bar with versions 2.0.3, 2.0.4, 2.1.0, 2.1.1, 3.0.0

Gemfile:

gem 'foo'

Gemfile.lock:

foo (1.4.3)
  bar (~> 2.0)
bar (2.0.3)
# Command Line Result
1 bundle update --patch 'foo 1.4.5', 'bar 2.1.1'
2 bundle update --patch foo 'foo 1.4.4', 'bar 2.0.3'
3 bundle update --minor 'foo 1.5.1', 'bar 3.0.0'
4 bundle update --minor --strict 'foo 1.5.0', 'bar 2.1.1'
5 bundle update --patch --strict 'foo 1.4.4', 'bar 2.0.4'

In case 1, bar is upgraded to 2.1.1, a minor version increase, because the dependency from foo 1.4.5 required it.

In case 2, only foo is unlocked, so foo can only go to 1.4.4 to maintain the dependency to bar.

whoops, case 2 is wrong. Here's the bundler spec demonstrating it. This is a documentation glitch with the root cause being my brain

In case 3, bar goes up a whole major release, because a minor increase is preferred now for foo, and when it goes to 1.5.1, it requires 3.0.0 of bar.

In case 4, foo is preferred up to a 1.5.x, but 1.5.1 won't work because the --strict flag removes bar 3.0.0 from consideration since it's a major increment.

In case 5, both foo and bar have any minor or major increments removed from consideration because of the --strict flag, so the most they can move is up to 1.4.4 and 2.0.4.

Troubleshooting

First, make sure the current bundle command itself runs to completion on its own without any problems.

The most frequent problems involve expectations around what gems should or shouldn't be upgraded. This can quickly get complicated as even a small dependency tree can involve many moving parts, and Bundler works hard to find a combination that satisfies all of the dependencies and requirements.

NOTE: the requirements in the Gemfile trump anything else. The most control you have is by modifying those in the Gemfile, in some circumstances it may be better to pin your versions to what you need instead of trying to diagnose why Bundler isn't calculating the versions you expect with a broader requirement. If there is an incompatibility, pinning to desired versions can also aide in debugging dependency conflicts.

You can get a (very verbose) look into how Bundler's resolution algorithm is working by setting the DEBUG_RESOLVER environment variable. While it can be tricky to dig through, it should explain how it came to the conclusions it came to.

In particular, grep for 'Unwinding for conflict' in the debug output to isolate some key issues that may be preventing the outcome you expect.

To get additional Bundler debugging output, enable the DEBUG env variable. This will include all of the details of the downloading the full dependency data from remote sources.

At the end of all of this though, again, the requirements in the Gemfile trump anything else, and the most control you have is by modifying those in the Gemfile.

TODOs

Should

  • bundler/bundler#4934 - Inconsistency with bundle update (major vs minor/patch) - [PR: https://github.com/bundler/bundler/issues/4977]
  • bundler/bundler#4912 - Update bundle lock with conservative update behavior/options - [PR: https://github.com/bundler/bundler/pull/5045]
  • bundler/bundler#4772 - Change outdated to use the new bundle update flags and conservative resolution code - [PR: https://github.com/bundler/bundler/pull/5061]
  • chrismo/bundler-cases#1 & chrismo/bundler-cases#3 - --conservative flag for bundle update - [PR: https://github.com/bundler/bundler/pull/4980]
  • bundler/bundler#4775 - Conservative update documentation updates [PR: https://github.com/bundler/bundler/pull/5108]

Nice

  • bundler/bundler#4776 - Add a --dry-run option to installer (available to bundle install and bundle update)
  • bundler/bundler#4774 - Add warning to bundle update when an unlocked gem doesn't move
  • bundler/bundler#4773 - Add warnings on bundle update when dependent updates go past --patch or --minor level

(not gonna do for now)

@chrismo
Member
chrismo commented Aug 23, 2016 edited

from @jkeiser: "hmm, I'm not sure I see an option to say "update foo, but leave bar at its current version unless you absolutely have to change it"? I'm not sure how this would prevent a wholesale update of bar in all cases where foo is updated.

I still would like a way for bundle install to do the non-surprising thing, but I'll get around to making a patch for that."

Off the top of my head, I'm not sure there's a way to capture the nuance of "unless you absolutely have to". Right now a gem can be locked or unlocked, bundle update foo means bar won't be changed at all, which denies the "unless you have to" request.

There is the option of specifying both --patch --strict flags ... which wouldn't allow foo or bar to be updated to a minor version, but could update bar when it didn't have to be...

Ok ... I guess perhaps this is the heuristic you want?

  • attempt update with 'foo' unlocked and 'bar' locked.
  • if 'foo' cannot be moved with 'bar' locked, now re-attempt update with 'foo' and 'bar' unlocked (and have it obey the desired option [patch/minor/major]) but only update 'bar' minimally.

I can see that. I wonder if we then have to worry about weighting one over the other.

For example ... well, lemme come up with an example and i'll update this.

This case does relate to this issue, methinks: bundler/bundler#4774 "Add warning to bundle update when an unlocked gem doesn't move"

@jkeiser
jkeiser commented Aug 23, 2016

@chrismo honestly, what I really want is for bundle install to do this. I can totally see how bundle update foo updating foo's dependencies is unsurprising :) But when bundle install notices that foo's specification in the Gemfile has changed, it's not immediately obvious that bar should be updated as well--in general, I had always assumed that bundle install would only ever change things which absolutely had to and go to great lengths not to change anything else. So it could be off topic for this?

OTOH, if a type of update were made for this, the same support could be added to bundle install or at least bundle install --super-conservative if a flag is necessary.

@chrismo
Member
chrismo commented Aug 23, 2016

I think it's on-topic enough for this. Internally I don't think there's too much separation in the version resolution process that's too specific to install over update.

@jkeiser
jkeiser commented Aug 23, 2016

@chrismo the actual use case for me is, when I bump the version of my source code (which has a Gemfile with gemspec in it to read the local gemspec), I run bundle install to update the Gemfile.lock (since it has to include the version).

However, even if I don't modify my Gemfile or any actual dependencies, and just bump my version for unrelated reasons, Bundler will update all my dependencies, because it feels that my source code's gem is "unlocked" since its version changed. This is super unexpected.

@chrismo
Member
chrismo commented Aug 23, 2016

Gotchya. Is this a correct use case?

Given the following gem specifications:

  • foo 1.4.3, requires: ~> bar 2.0
  • foo 1.5.1, requires: ~> bar 2.1
  • foo 1.5.2, requires: ~> bar 3.0

Gemfile (before):

gem 'foo', '~> 1.4'

Gemfile.lock (before):

foo (1.4.3)
  bar (~> 2.0)
bar (2.0.3)

Gemfile (change):

gem 'foo', '~> 1.5'

bundle install you'd expect foo 1.5.1 with bar 2.1.0, not foo 1.5.2 with bar 3.0.0.

@jkeiser
jkeiser commented Aug 23, 2016 edited

Yep. I think the use case is a lot clearer when foo is retrieved via gemspec so the Gemfile itself doesn't even change, but that is indeed the use case.

I also wonder, if you changed gem 'foo', '~> 1.4' to gem 'foo', '~> 1.4.3', if the same thing would happen. i.e. if you don't actually change foo, but change what it looks like in the gemspec, bar may be upgraded anyway? Haven't tested this, and it's slightly different anyway.

@marcandre

In the example given in the original post, third one says:

bundle update --patch foo => 'foo 1.4.4', 'bar 2.0.3'

That would be fantastic; does this means the current behavior (which is --major, right?) is also changing? I'm asking because if one does bundle update foo, at a time where 1.4.4 just came out, it currently also updates bar to 2.1.1, but from that example (and given the fact that it doesn't update bar to 2.0.4) I'd deduce that now it wouldn't update it?

@chrismo
Member
chrismo commented Aug 24, 2016 edited

Current behavior --major is untouched and no plans to change it. It'll be the current level of un-conservative.

The following is currently live in 1.13.0.rc.2:

[chrismo@momac bundler]$ cat Gemfile*
source 'https://rubygems.org' do
  gem 'rack'
  gem 'addressable'
end
GEM
  remote: https://rubygems.org/
  specs:
    addressable (2.1.1)
    rack (1.4.1)

PLATFORMS
  ruby

DEPENDENCIES
  addressable (= 2.1.1)!
  rack (= 1.4.1)!

BUNDLED WITH
   1.13.0.rc.2

[chrismo@momac bundler]$ bundle outdated
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/.
Resolving dependencies...

Outdated gems included in the bundle:
  * addressable (newest 2.4.0, installed 2.1.1) in group "default"
  * rack (newest 2.0.1, installed 1.4.1) in group "default"

[chrismo@momac bundler]$ bundle update rack --patch
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/.
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/.
Resolving dependencies...
Using addressable 2.1.1
Installing rack 1.4.7 (was 1.4.1)
Using bundler 1.13.0.rc.2
Bundle updated!

[chrismo@momac bundler]$ bundle update --minor
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/.
Resolving dependencies...
Using addressable 2.4.0 (was 2.1.1)
Using rack 1.6.4 (was 1.4.7)
Using bundler 1.13.0.rc.2
Bundle updated!
@indirect
Member

@chrismo @jkeiser I think your example revealed a very interesting detail of what "conservative update" means, and how it is probably impossible to meet everyone's expectations.

Given the setup that @chrismo laid out, I absolutely would expect changing the Gemfile to gem "foo", "~> 1.5" to update to foo-1.5.2 and bar-3.0.0. Since you explicitly asked to update foo, you are (implicitly) telling bundler that it is fine to update any of foo's dependencies as much as is needed to get the newest version of foo possible. My idea of conservative updating is that the currently-locked version of a gem is always tried first, but other versions are allowed.

Here is an example of when I would expect bar to not update: if foo-1.5.2 instead still depended on bar (~> 2.0) (the same as existing locked foo-1.4.3), then I would expect bundle update foo to first try foo-1.5.3, select it, and then first try bar-2.0.3 (because it is locked). At that point, all requirements would be satisfied and the update would end with foo-1.5.2 and bar-2.0.3.

My understanding is that the behavior I just described is the way bundle install has always been intended to work, although I honestly have no idea if that is how it actually works today.

Does that make sense? Does that conform to all of your expectations?

@chrismo
Member
chrismo commented Aug 25, 2016

I should clarify, that use case I called out was for the kind of 'conservative' version of bundle install that I thought @jkeiser was requesting. I'd expect your explanation @indirect to be the way it is now (and if so should stay put for backward compat). If so, then @jkeiser's case would perhaps require some sort of additional conservative options passed to install in the same vein as what we've added to bundle update -- and if there's enough overlap with the new code so far for update then it could be worth figuring out.

I do agree it's likely we'll hear conflicting expectations within the community on even the same exact use case ... so in fact, I'm going to try and come up with a real example for @jkeiser's case (that I can execute with code) so we can talk more directly to the differences.

@jkeiser
jkeiser commented Aug 25, 2016

Yes, that's more like what I expect. Basically, the way I look at it, bundle install should update any gems that do not match current constraints in the gemfile; then, if any other gems no longer match constraints, update those. I don't really care what it chooses when it determines it has to update a gem--i don't mind if it picks the latest available at that point. But the point is, I don't expect bundle install to ever update a dependency that matches constraints.

Consider this. Lockfile points at:

  • foo 5.0.2 depends on bar >= 4.0
  • bar 4.0.1

Gemfile updates to point at foo 5.0.3, which depends on bar >= 4.0.1.

Afterwards, I expect foo to update to 5.0.3 and bar to stay the same, even if bar 4.0.2 exists.

@jkeiser
jkeiser commented Aug 25, 2016

@chrismo @indirect bundler/bundler#4690 has a pretty minimal case in it.

@chrismo
Member
chrismo commented Aug 25, 2016

right! thx for reminding me to read back up on that issue. I'd made a sweep through old issues who might care about this issue here and briefly linked here ... I'd lost touch with your specifics there.

Re-reading this one phrase from our Jun conversation brought it back: "If you want 'bar' to stay put, you gotta declare it as a dependency in the Gemfile."

Your expectation would be it shouldn't matter if the 'bar' gem is listed in the Gemfile or not, don't move it if you don't have to, right?

@jkeiser
jkeiser commented Aug 25, 2016

Yep. That was always my assumption about it. I use bundle update when I
want things to change, bundle install when I don't :)

On Wed, Aug 24, 2016, 9:41 PM chrismo notifications@github.com wrote:

right! thx for reminding me to read back up on that issue. I'd made a
sweep through old issues who might care about this issue here and briefly
linked here ... I'd lost touch with your specifics there.

Re-reading this one phrase from our Jun conversation brought it back: "If
you want 'bar' to stay put, you gotta declare it as a dependency in the
Gemfile."

Your expectation would be it shouldn't matter if the 'bar' gem is listed
in the Gemfile or not, don't move it if you don't have to, right?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#122 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAn-IqRS_YtmJ6dw5VbLYh7E9bnNV4H_ks5qjR0PgaJpZM4JrLV8
.

@chrismo
Member
chrismo commented Aug 25, 2016

Ok, got it. Changing this behavior I'm sure is not in the cards, question for this thread/my brain then is there room/overlap with what has been done so far to add a flag or something to bundle install to communicate this preference in this case.

@chrismo
Member
chrismo commented Aug 25, 2016

Random note to myself concerning this earlier heuristic comment, which isn't pertinent at all to this point of the conversation with John...

  • attempt update with 'foo' unlocked and 'bar' locked.
  • if 'foo' cannot be moved with 'bar' locked, now re-attempt update with 'foo' and 'bar' unlocked (and have it obey the desired option [patch/minor/major]) but only update 'bar' minimally.

This could probably be done in one shot by internally flagging all gems as unlocked, doing the normal version ordering for the changing gem, and doing a --minimal (bundler-patch behavior) sort on all other gems, including the current version as the first preference. Sort of a 'prefer locked version' for locked gems, rather than actually locking gems. It sort of would be the first case where we're declaring a combination of preferences. Currently --patch or --minor' apply to all unlocked gems. This possible case would be--major for unlocked, --minimal for locked`. (I know this language is terribly confusing, but should capture enough for me for now).

@marcandre
marcandre commented Aug 25, 2016 edited

Sorry if I'm insistent, but I could imagine two different behaviors for bundle update --patch foo in @chrismo's original example. Clearly, foo is updated to 1.4.4. bar could remain at 2.0.3 or be updated at 2.0.4. While I personally much prefer it remains at 2.0.3, why does it not update to 2.0.4? Or is --patch not consistent with --major by design?

@jkeiser
jkeiser commented Aug 25, 2016

I think I responded wrongly at first; he was suggesting that the update
move dependencies forward absolutely as little as possible to satisfy
dependencies. My personal feeling is that for install, if existing
constraints are satisfied by a dependency, it shouldn't move. But if it has
to move, I'm OK with it updating to the latest. It is the number of the
changes (and which ones) rather than their magnitude that matters in that
case. I like the patch and minor options, but they seem orthogonal to me.

On Thu, Aug 25, 2016, 7:46 AM Marc-André Lafortune notifications@github.com
wrote:

Sorry if I'm insistent, but I could imagine two different behaviors for bundle
update --patch foo in your original example. Clearly, foo is updated to
1.4.4. bar could remain at 2.0.3 or be updated at 2.0.4. While I
personally much prefer it remains at 2.0.3, why does it not update to
2.0.4? Or is --patch not consistent with --major by design?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#122 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAn-IrUJONtgay32-ruwWAZzfhZntvdhks5qjarTgaJpZM4JrLV8
.

@marcandre

Thanks @jkeiser , that doesn't quite answer my question though, which was more aimed at @chrismo . I'm still trying to understand the feature!

@jrochkind
jrochkind commented Aug 25, 2016 edited

I thought that legacy behavior was on install, dependencies only moved if they had to (to meet changes in the Gemfile, themselves or other things that specify requirements for them).

Is this not the way it always worked sans bugs?

The difficulty here is we don't seem to even have a consensus on existing behavior.

Here's the existing docs: http://bundler.io/man/bundle-install.1.html#CONSERVATIVE-UPDATING

They seem to me to suggest that bundle install should not change any dependencies unless it has to to meet changes in Gemfile since last Gemfile.lock.

This is the behavior I expected, and seems right to me, and I'd consider anything else a bug.

When we talk about 'conservative updating' in bundle update, a new feature -- this is what I wanted too. This does not seem to be what the patches offered do, however. What I want has nothing to do with limiting to minor or patch levels -- just as previous "conservative updating" used in bundle install did not, so far as I know? What I want, and what i think others wanted, is an option to bundle update that will make it use the same "conservative updating" algorithm that already existed and was used for install, but for the gems mentioned on the command line. Exactly what this looks like starts to get a bit confusing when you get into the details, but the simple naive case is bundle update --conservative widget would update widget to the latest version available (just as bundle update widget does, right?), but then update any other dependencies with the same "conservative updating" algorithm used by bundle install -- not updating them unless they need to be (unlike existing bundle update widget that updates all widget dependencies aggressively).

@indirect
Member

@jrochkind thanks for writing that up! that's a great summary, and I think I agree with you on both points:

I would expect install to not unlock any versions (even child dependencies) that meet all requirements. I think that is how install worked, once, and it is how I would like it to work again one day.

It also sounds helpful to add a --conservative flag to update. Having that behavior available is useful, but I worry that it will encourage Gemfiles with little or no version requirements. Over time, that builds up into Gemfiles that take hours to resolve because there are just too many possibilities. That said, it sounds like a useful option--I would probably use it.

@jrochkind
jrochkind commented Aug 26, 2016 edited

Thanks @indirect!

I don't think --conservative along the lines I'm imagining makes Gemfiles without any versions any better of an idea, for reasons other than bundler efficiency. But that doesn't mean people won't do it -- people do it now, people don't seem to understand how bundler is supposed to work.

However, the fact that bundler doesn't always work 'right' or as originally intended now is actually a barrier to people understanding how Gemfiles are meant to be used.

For instance, not to bring up water under the bridge, but in bundler#3089, I reported an issue where bundle update did not arrive at the same conclusion that bundle install would have done (and did not use the most recent dependencies that it could have) -- and you told me this was not a bug, bundle update made no such guarantees. But, to my perspective, that makes it very difficult to use bundler the way it was intended, and if one's exposure to bundler is a broken bundler, that makes it harder to understand how bundler is meant to be used -- as I understand it, ideally, all gems would use semver and would be specified to major versions. (or would make some other commitment about backwards compat, and would be specified accordingly; for instance, Rails needs to be locked to a minor version).

I have definitely needed an imagined bundle --conservative before, despite my best efforts to use bundler this way, but I'm having trouble remembering the use case now -- I'll think on it, or see if I can encounter it again, and try to explain it.

@chrismo
Member
chrismo commented Aug 26, 2016

Thx for all the feedback! I've been too busy to thoughtfully digest, but I will get caught up.

@chrismo
Member
chrismo commented Aug 28, 2016

Have a little side-project I put together the last few days, a tool that should help us have clearer discussions about the various cases we'd like to see (and demonstrate current behavior).

https://github.com/chrismo/bundler-cases

It'll still need a lot more tweaking, but it currently demonstrates the Conservative Updating section of the bundle install docs.

@chrismo
Member
chrismo commented Aug 29, 2016

:facepalm: this is already paying off. I just finished implementing cases for the single gem and two gem examples in the docs here in the original post, and one of the 5 cases doesn't work as-is.

I sort of knew whenever I was writing up the foo/bar cases above that while I had similar unit/integration tests in the codebase, I was still not going verbatim from existing test cases.

So ... anyway, this commit details the flaw.

@marcandre

So, was I right to point out that bundle update --patch foo (without the strict) would update bar to 2.0.4?
I don't understand the --strict option, and the fact that when doing update --patch --strict foo and bundle update --patch --strict (without the foo) have different versions of bar (2.0.3 and 2.0.4) isn't clear at all for me. Maybe the "documentation" above could be reworded?

@chrismo
Member
chrismo commented Aug 29, 2016 edited

--strict means, "Don't let ANYTHING get updated past --patch or --minor level." It can result in no update being allowed (or a version conflict) in the right circumstances.

I need to research more, but it could be the existing tests in all my code still don't take into account the proper locking/unlocking behavior of dependencies of unlocked gems, in the same vein as documented in bundle install's Conservative Updating section.

The rack and addressable cases shown in an earlier comment don't have any dependency on each other like the foo/bar relationship in my samples.

Some current specs are here and here

Oh, hell ... this exact test IS in there :)

https://github.com/bundler/bundler/blob/master/spec/commands/update_spec.rb#L517-L521

Well, I'm not a total moron.

@chrismo chrismo added a commit to chrismo/bundler-cases that referenced this issue Aug 29, 2016
@chrismo chrismo jkeiser use case about conservative bundle install
See [this
comment](bundler/bundler-features#122 (comment)).

The only way to get his desired behavior is to either declare 'bar' as a
dependency in the Gemfile, or have another gem that also depends on
'bar' declared as a dependency in the Gemfile.
818176f
@chrismo chrismo referenced this issue in chrismo/bundler-cases Aug 29, 2016
Closed

jkeiser use case about conservative bundle install #1

@marcandre

lol, sorry for my confusion. So there was an error in your original post, but not how I thought it was. I understand that now (at last!)

I still don't understand how come these two tests pass. More precisely, I get the last one, but I don't get how come the first one doesn't result in the same versions too. Even with the strict and patch options, bar could go to 2.0.4, no?

@chrismo
Member
chrismo commented Aug 29, 2016

No worries, if we start apologizing for confusion, we'll have to start a whole separate Issue (or Repo) to contain them all.

I THINK (not sure) the magic difference between bundle update --patch --strict foo which LEAVES bar alone, and bundle update --patch foo which DOESN'T, is that the requirement from foo to bar changes in the second case.

In both cases, we told bundler to just update foo - which in general means bar is locked. But when foo is allowed to go up to 1.4.5, and now 1.4.5 has a new requirement on bar, then bar is essentially unlocked.

But - I need to find the code in Bundler (which I have a hazy memory of at the moment) that does this.

@chrismo
Member
chrismo commented Aug 29, 2016

@jrochkind and @indirect -> made a case PR demoing your part of the conversation: chrismo/bundler-cases#3

@chrismo chrismo referenced this issue in chrismo/bundler-cases Aug 29, 2016
Merged

Cases demo-ing @jrochkind desire for cons. update. #3

@marcandre

I've looked into the inconsistency I perceived and I can confirm, there is no explanation that makes sense (at least to be). I opened this issue

@chrismo chrismo added a commit to chrismo/bundler that referenced this issue Sep 14, 2016
@chrismo chrismo WIP: replicate cons. bundle install in update cmd
In the discussion on new 1.13 [Conservative Bundle
Updates](bundler/bundler-features#122), some
users would like to have the same [Conservative
Updating](http://bundler.io/v1.12/man/bundle-install.1.html#CONSERVATIVE-UPDATING)
behavior available in `bundle install` when a declared dependency is
altered in the Gemfile, also available in the `bundle update` command.

This commit may be most of the way there.

The interesting (and risky) bit is extracting out the eager_unlock code
from the Definition initializer into its own bits of code that can be
re-used from the Bundler::CLI::Update class. The `expand_dependencies`
method needed extraction to a neutral place (I've briefly chosen a new
DependencyList class, but that certainly could use some more research).

The good news about this change is now the new CLI option can be dealt
with strictly in the Bundler::CLI::Update class, which is already mostly
responsible for determining what gems are unlocked, otherwise a new
option might have be drilled all the way through into the Definition
class.

A few things to do or consider:

- [ ] What else (`bundle lock`?) passes in @unlock[:gems] and is now
  broken?

- [ ] Is the `--lock-shared-deps` flag a good name. Some proposed the
  name `--conservative`, which is better in that it's probably easier to
  remember, but I also fear is too vague. I'd be open to them being
  synonyms. And/or a `-c` type flag.

- [ ] Needs more tests around the interactions of the new `--patch` and
  `--minor` flags when combined with `--lock-shared-deps` -- code should
  be orthogonal, but tests need to prove that out.

- [ ] DependencyList, good name? Good location? Separate file?
688c627
@chrismo
Member
chrismo commented Sep 14, 2016

Just submitted this PR with some code that may do what some of y'all have requested: bundler/bundler#4980

@chrismo chrismo added a commit to chrismo/bundler that referenced this issue Oct 19, 2016
@chrismo chrismo WIP: replicate cons. bundle install in update cmd
In the discussion on new 1.13 [Conservative Bundle
Updates](bundler/bundler-features#122), some
users would like to have the same [Conservative
Updating](http://bundler.io/v1.12/man/bundle-install.1.html#CONSERVATIVE-UPDATING)
behavior available in `bundle install` when a declared dependency is
altered in the Gemfile, also available in the `bundle update` command.

This commit may be most of the way there.

The interesting (and risky) bit is extracting out the eager_unlock code
from the Definition initializer into its own bits of code that can be
re-used from the Bundler::CLI::Update class. The `expand_dependencies`
method needed extraction to a neutral place (I've briefly chosen a new
DependencyList class, but that certainly could use some more research).

The good news about this change is now the new CLI option can be dealt
with strictly in the Bundler::CLI::Update class, which is already mostly
responsible for determining what gems are unlocked, otherwise a new
option might have be drilled all the way through into the Definition
class.

A few things to do or consider:

- [ ] What else (`bundle lock`?) passes in @unlock[:gems] and is now
  broken?

- [ ] Is the `--lock-shared-deps` flag a good name. Some proposed the
  name `--conservative`, which is better in that it's probably easier to
  remember, but I also fear is too vague. I'd be open to them being
  synonyms. And/or a `-c` type flag.

- [ ] Needs more tests around the interactions of the new `--patch` and
  `--minor` flags when combined with `--lock-shared-deps` -- code should
  be orthogonal, but tests need to prove that out.

- [ ] DependencyList, good name? Good location? Separate file?
7e74abd
@chrismo chrismo added a commit to chrismo/bundler that referenced this issue Oct 19, 2016
@chrismo chrismo WIP: replicate cons. bundle install in update cmd
In the discussion on new 1.13 [Conservative Bundle
Updates](bundler/bundler-features#122), some
users would like to have the same [Conservative
Updating](http://bundler.io/v1.12/man/bundle-install.1.html#CONSERVATIVE-UPDATING)
behavior available in `bundle install` when a declared dependency is
altered in the Gemfile, also available in the `bundle update` command.

This commit may be most of the way there.

The interesting (and risky) bit is extracting out the eager_unlock code
from the Definition initializer into its own bits of code that can be
re-used from the Bundler::CLI::Update class. The `expand_dependencies`
method needed extraction to a neutral place (I've briefly chosen a new
DependencyList class, but that certainly could use some more research).

The good news about this change is now the new CLI option can be dealt
with strictly in the Bundler::CLI::Update class, which is already mostly
responsible for determining what gems are unlocked, otherwise a new
option might have be drilled all the way through into the Definition
class.

A few things to do or consider:

- [ ] What else (`bundle lock`?) passes in @unlock[:gems] and is now
  broken?

- [ ] Is the `--lock-shared-deps` flag a good name. Some proposed the
  name `--conservative`, which is better in that it's probably easier to
  remember, but I also fear is too vague. I'd be open to them being
  synonyms. And/or a `-c` type flag.

- [ ] Needs more tests around the interactions of the new `--patch` and
  `--minor` flags when combined with `--lock-shared-deps` -- code should
  be orthogonal, but tests need to prove that out.

- [ ] DependencyList, good name? Good location? Separate file?
b94f242
@homu homu added a commit to bundler/bundler that referenced this issue Oct 21, 2016
@homu homu Auto merge of #4980 - chrismo:lock_shared_deps, r=indirect
replicate cons. bundle install in update cmd

In the discussion on new 1.13 [Conservative Bundle
Updates](bundler/bundler-features#122), some
users would like to have the same [Conservative
Updating](http://bundler.io/v1.12/man/bundle-install.1.html#CONSERVATIVE-UPDATING)
behavior available in `bundle install` when a declared dependency is
altered in the Gemfile, also available in the `bundle update` command.

This commit may be most of the way there.

The interesting (and risky) bit is extracting out the eager_unlock code
from the Definition initializer into its own bits of code that can be
re-used from the Bundler::CLI::Update class. The `expand_dependencies`
method needed extraction to a neutral place (I've briefly chosen a new
DependencyList class, but that certainly could use some more research).

The good news about this change is now the new CLI option can be dealt
with strictly in the Bundler::CLI::Update class, which is already mostly
responsible for determining what gems are unlocked, otherwise a new
option might have be drilled all the way through into the Definition
class.

A few things to do or consider:

- [ ] What else (`bundle lock`?) passes in @unlock[:gems] and is now
  broken?

- [ ] Is the `--lock-shared-deps` flag a good name. Some proposed the
  name `--conservative`, which is better in that it's probably easier to
  remember, but I also fear is too vague. I'd be open to them being
  synonyms. And/or a `-c` type flag.

- [ ] Needs more tests around the interactions of the new `--patch` and
  `--minor` flags when combined with `--lock-shared-deps` -- code should
  be orthogonal, but tests need to prove that out.

- [ ] DependencyList, good name? Good location? Separate file?
0e2665f
@chrismo chrismo added a commit to chrismo/bundler that referenced this issue Oct 21, 2016
@chrismo chrismo Add bundle install conservative updating to update
In the discussion on new 1.13 Conservative Bundle Updates (see
bundler/bundler-features#122), some users
would like to have the same Conservative Updating (see
http://bundler.io/v1.12/man/bundle-install.1.html#CONSERVATIVE-UPDATING)
behavior available in `bundle install` when a declared dependency is
altered in the Gemfile, also available in the `bundle update` command.

This adds a new option called `--conservative` to both `bundle update`
and `bundle lock`. The option only applies on `bundle lock` if the
`--update` option is in use.

The internal flag is more descriptive as to what actually takes place:
It locks any shared dependencies from the gem(s) being updated.

This also promotes the previously added patch level options to being
shown in command-line help, anticipating a 1.14 release, to make these
public and documented.
ad80a92
@homu homu added a commit to bundler/bundler that referenced this issue Oct 21, 2016
@homu homu Auto merge of #4980 - chrismo:lock_shared_deps, r=indirect
replicate cons. bundle install in update cmd

In the discussion on new 1.13 [Conservative Bundle
Updates](bundler/bundler-features#122), some
users would like to have the same [Conservative
Updating](http://bundler.io/v1.12/man/bundle-install.1.html#CONSERVATIVE-UPDATING)
behavior available in `bundle install` when a declared dependency is
altered in the Gemfile, also available in the `bundle update` command.
03c864a
@chrismo
Member
chrismo commented Oct 22, 2016

Just a comment for any participants here - all the code for 1.14 is ready to go. Writing up some updates to the docs. See #4775.

@jrochkind

Thanks for attending to docs!

There's no issue or PR #4775, where did you mean to direct us?

@chrismo
Member
chrismo commented Oct 23, 2016

Sorry, this one => bundler/bundler#4775

@pierre-pretorius
pierre-pretorius commented Nov 25, 2016 edited

If I want to update a gem without updating it's dependencies unless there is no other option I still use the hack below because it consistently does what I want. I was hoping one of these new commands can replace the hack.

bundle update --source some_gem_name

@segiddins
Member

@pierre-pretorius bundle update --conservative --strict gem_name should do what you want

@jrochkind

Does --strict mean it won't update more than a patch version though? Cause I think that's not what we want, or at least not what I want. I want to update a certain gem to the latest version (allowed by restrictions, just like ordinary bundle update gemname does), but without updating any of it's dependencies. The way to replicate that under old bundler for me would be: Look up the latest version available of gemname; change/add a line in Gemfile to require that version; run bundle install (not update); remove/change back the line in the Gemfile I changed.

Is there a way to do that? I still haven't managed to wrap my head around the new options.

@segiddins
Member

Can you test it out? As long as you're not passing --patch, it shouldn't be restricting the set of possible versions for the gem you've explicitly unlocked.

@chrismo
Member
chrismo commented Nov 30, 2016

@pierre-pretorius FYI, that hack doesn't always work: https://github.com/chrismo/bundler-source-hack#why-then-does-the-hack-sometimes-not-work - but if it's still working for you in the cases you need, awesome :) And the --conservative option hasn't been released yet (should go with 1.14).

@jrochkind @segiddins --conservative and the other patch options should be orthogonal. And --strict is only meaningful if --patch, --minor, or --major are used. That said, I see I didn't add any tests that combine those to prove it, probably because I believed them to be unrelated, and those tests are expensive, integration tests in update_spec.rb.

@jrochkind these features I've added get pretty nit-picky and hard to understand for myself sometimes. For my brain, it's best understood when working with them vs. just trying to discuss them.

All that said, bundle update --conservative [gem_name] in 1.14 should do exactly what you want, should mimic the bundle install conservative behavior, without you having to resort to modifying the Gemfile.

@jrochkind

Thanks @chrismo, helpful comments. I'll try to find some time to play with it! I didn't realize --conseravative wasn't released yet, since it was mentioned in the 1.13 release announcement. This stuff does get confusing, I agree!

@chrismo
Member
chrismo commented Dec 19, 2016

This issue is all but done, waiting on a 1.14 release which is waiting on some other stuff. Feel free to continue to comment if you wanna - but I'm going to close this for now.

@chrismo chrismo closed this Dec 19, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment