Skip to content
This repository has been archived by the owner. It is now read-only.

Do not update BUNDLED WITH if no other changes to the lock happened #3697

Closed
jhass opened this issue May 31, 2015 · 70 comments
Closed

Do not update BUNDLED WITH if no other changes to the lock happened #3697

jhass opened this issue May 31, 2015 · 70 comments

Comments

@jhass
Copy link
Contributor

@jhass jhass commented May 31, 2015

Running bundle install (e.g. as a dev on a project) and having a newer version of bundler, writes that newer version into the lock. This makes the lock dirty, leaving you with the option to 1) make a meaningless commit to bump the bundler version referenced in it or 2) constantly run git checkout Gemfile.lock as you switch branches, rebase and so on.

If there were no other changes to the lock, we know that the previous version of Bundler was not only working for the lock and the corresponding Gemfile, in fact we know that there's very likely not even the slightest difference in behaviour for it.

I'm happy to look into a fix if you have some pointers, like where one could decide to not write the lock at all if it wasn't changed.

@ashchan
Copy link

@ashchan ashchan commented May 31, 2015

This is also annoying when you have v1.10, and other team members still have 1.9.

jhass added a commit to diaspora/diaspora that referenced this issue May 31, 2015
@segiddins
Copy link
Member

@segiddins segiddins commented May 31, 2015

Bundler will only ever increase that version, not decrease it. The reason we need to do that, even when there are seemingly no other changes, is to know what version you, the developer, have verified "works" with your project.

@segiddins segiddins closed this May 31, 2015
@jhass
Copy link
Contributor Author

@jhass jhass commented May 31, 2015

Can I have "please do a commit to all your projects on every bundler bugfix release" as an official statement clearly spelled out then?

@segiddins
Copy link
Member

@segiddins segiddins commented May 31, 2015

Well, it's of course up to you if you want to commit those changes or not :)

-Samuel E. Giddins

On May 31, 2015, at 9:56 AM, Jonne Haß notifications@github.com wrote:

Can I have "please do a commit to all your projects on every bundler minor release" as an official statement clearly spelled out then?


Reply to this email directly or view it on GitHub.

@jhass
Copy link
Contributor Author

@jhass jhass commented May 31, 2015

Well, variant 2 that I described in my initial issue is an even bigger pain in the ass.

@jhass
Copy link
Contributor Author

@jhass jhass commented May 31, 2015

I'm just asking what the officially endorsed workflow is here.

Also keep in mind that things like spring constantly write the lock on changes (I didn't investigate why, I just see it very clearly happening).

@SuperTux88
Copy link

@SuperTux88 SuperTux88 commented May 31, 2015

This is no problem, when there is only one developer ... the Gemfile.lock always contains the version from this developer.

But this is a really big issue, when there are multiple developers contributing via pull-requests ... when in the Gemfile.lock on the upstream-branch is e.g. 1.10.0 and one developer has a newer version (e.g. 1.10.2) then he has always an outgoing change on the Gemfile.lock. But he can't commit the Gemfile.lock to his bugfix/feature-branch, because it has nothing to do with the bug/feature he is developing.

So someone needs to commit a new Gemfile.lock to the upstream-branch every time a new bundler version is released? Is this the intended workflow?

@indirect
Copy link
Member

@indirect indirect commented May 31, 2015

Our end goal is to allow both you and us to keep track of what version of Bundler works with your application. At some point in the future, Bundler 2 will introduce breaking changes, and we will need the version information to know whether it is even possible to install your Gemfile or not.

I realize that you're not used to checking in your lock when your version of Bundler changes, but I hope it at least makes sense—you check in your lock when your version of Rake changes, after all. It's not a pointless commit, it's a commit with a big point: letting other developers on this project know that they have a version of Bundler older than the version you used. Their older version definitely might not have features that you used, or their version might have a bug that keeps them from being able to get the same results that you did.

@jhass
Copy link
Contributor Author

@jhass jhass commented May 31, 2015

The difference is that I can decide when I change my rake version for each project, after all I use bundler exactly for that point. This breaks making this decision on a per project basis, that is allow me to use a different version locally because I know what I'm doing without forcing it onto other developers or making my own workflow very annoying.

@jhass
Copy link
Contributor Author

@jhass jhass commented May 31, 2015

Oh and as @SuperTux88 pointed out, it also makes the workflow in the same way annoying for collabs working on feature branches that don't have commit access to the main branch.

@urbanautomaton
Copy link
Contributor

@urbanautomaton urbanautomaton commented Jun 1, 2015

Hi @indirect - thanks for the response.

If breaking changes are planned for a future major version, it absolutely makes sense to note the major version and warn if this doesn't match. At present, however, this checking goes down to the patch level, which is going to generate a lot of warnings and diffs from perfectly safe version differences.

In a large team, it's very likely that devs will have bundler versions with varying minor or patch numbers, and so people will frequently be seeing unexpected diffs or warnings. When there are non-ruby devs working on a bundled project, these warnings and diffs can cause a surprising amount of confusion.

There's also the likelihood that if warnings are generated every time the lockfile patch version changes, people will start tuning them out, and when breaking changes really are introduced, the warning will be ignored.

Would you consider recording only the major version that the gemfile was bundled with, and warning only when this version differs?

@chemica
Copy link

@chemica chemica commented Jun 2, 2015

For some reason I can't even use 'git checkout' to remove the new version number. As soon as I do so (or remove it by hand) Bundler seems to swoop in and change it back again. It must be watching the file system for changes.

The only way I've found to stop it is to uninstall the Bundler gem.

This is beyond annoying and has ground my day to a halt trying to find a sensible workflow with Git...

@jhass
Copy link
Contributor Author

@jhass jhass commented Jun 2, 2015

@chemica are you using spring or guard?

@segiddins
Copy link
Member

@segiddins segiddins commented Jun 2, 2015

Or are you running bundle exec?

-Samuel E. Giddins

On Jun 2, 2015, at 5:54 AM, Jonne Haß notifications@github.com wrote:

@chemica are you using spring or guard?


Reply to this email directly or view it on GitHub.

@abulrim
Copy link

@abulrim abulrim commented Jun 2, 2015

Having the same issue as @chemica

@chemica
Copy link

@chemica chemica commented Jun 2, 2015

Not using spring or guard. Not using bundle exec as the only commands I'm calling are bundle, git and vim.

Uninstalling the bundler gem temporarily solves the issue, so it would appear to be the cause.

I have resolved the issue for myself by temporarily gaining push privileges to the master branch to make the update. However, it's an issue that probably still needs looking at. My case was severe and complicated by some particular git rules around push and pull at this company, but I'm not the only user expressing pain over this.

@chemica
Copy link

@chemica chemica commented Jun 2, 2015

Bundler version: 1.10.2.
OS: Mac OSX Yosemite
ruby 2.2.0p0

@mferrier
Copy link

@mferrier mferrier commented Jun 2, 2015

Wow, this is very brutal for large dev teams. There's really no alternative?

I guess the workaround is to set BUNDLED WITH to 1.99.99 and just forget this feature exists.

@segiddins
Copy link
Member

@segiddins segiddins commented Jun 2, 2015

I really don't see how just not git adding a file is so difficult...

@jhass
Copy link
Contributor Author

@jhass jhass commented Jun 2, 2015

I see you never use git checkout branch or git rebase or git stash

@mferrier
Copy link

@mferrier mferrier commented Jun 2, 2015

@segiddins time to git reset to avoid merge/checkout issues × each unmerged branch × each developer × each bundler point release

Any reason the suggestion above of only introducing breaking changes via major version changes and then only tracking that isn't an option?

@segiddins
Copy link
Member

@segiddins segiddins commented Jun 2, 2015

We're considering different options, but not tracking the minor version isn't on the table, since minor versions can add significant new features.

@segiddins
Copy link
Member

@segiddins segiddins commented Jun 2, 2015

I see you never use git checkout branch or git rebase or git stash

I do all the time. But I also selective stage changes and git reset aggressively, so this change has never been an issue for me

@jhass
Copy link
Contributor Author

@jhass jhass commented Jun 2, 2015

From my side I could accept not tracking the patchlevel/bugfix as a compromise.

@botandrose
Copy link

@botandrose botandrose commented Jun 2, 2015

I understand what @indirect is trying to accomplish here, but I'd suggest this should be held back until bundler 2.0, because until now, all 1.x releases could interop backwards and forwards. You could bundle a 1.0 Gemfile with a 1.9 bundler, and 1.0 bundler would be none the wiser. Maybe this wasn't explicitly acknowledged as a feature of bundler, and thus not eligible for protection under semver, but it was a feature I have relied on, nonetheless.

@segiddins
Copy link
Member

@segiddins segiddins commented Jun 2, 2015

This IS a backwards compatible change.

@jhass
Copy link
Contributor Author

@jhass jhass commented Jun 4, 2015

To repeat another point that has been said previously, the difference between Bundler and your apps other gem dependencies is that Bundler is installed and managed outside/not by Bundler. As such it is a lot more likely to have different versions of it across developer setups, especially in the last component.

To add another point to this that hasn't ben explicitly said yet, another difference is that using multiple versions of Bundler is a lot harder compared to gems managed under Bundler. In fact I could remove gemsets from my workflow entirely due to that and this change will likely force my to reconsider that.

@segiddins
Copy link
Member

@segiddins segiddins commented Jun 4, 2015

You can always use bundler _1.10.2_ to use a specific bundler version (RubyGem’s executable stubs allow specifying a gem version in that manner)

On Jun 3, 2015, at 5:25 PM, Jonne Haß notifications@github.com wrote:

To repeat another point that has been said previously, the difference between Bundler and your apps other gem dependencies is that Bundler is installed and managed outside/not by Bundler. As such it is a lot more likely to have different versions of it across developer setups, especially in the last component.

To add another point to this that hasn't ben explicitly said yet, another difference is that using multiple versions of Bundler is a lot harder compared gems managed under Bundler. In fact I could remove gemsets from my workflow entirely due to that.


Reply to this email directly or view it on GitHub #3697 (comment).

@jhass
Copy link
Contributor Author

@jhass jhass commented Jun 4, 2015

Alright, that reduces that particular issue a bit, but stays that I didn't need to care in the past.

@indirect
Copy link
Member

@indirect indirect commented Jun 4, 2015

Yes, that is literally one of the points of this change. If there is a bugfix release of Bundler, the minor version was broken. Something about it didn't work at all, and needed to be fixed. As a result, it's important to be on the latest bugfix release, especially if your project depends on new features. I can't even describe how many bug reports are opened for problems that have already been fixed by developers who are still using old versions of Bundler.

We're incredibly careful about backwards compatibility, and have put huge amounts of effort into making sure that new versions install bundles the same way that older versions did. Do you object to the idea of getting bugfixes? Do you object to the idea of knowing the bugfixes are available?

If you personally don't want to upgrade your own version of Bundler, that's fine—literally nothing will happen except that you'll still have bugs that have been fixed already. There won't be any changes to the lock. But it's very uncool to demand that bugfixes not be included in the out-of-date version warning.

@jhass
Copy link
Contributor Author

@jhass jhass commented Jun 4, 2015

For all other gems I have to explicitly update my Gemfile.lock to get the bugfixes, I can decide if I need them, I can do the change isolated when I want it. You're still completely neglecting the raised points about contributor workflows.

@indirect
Copy link
Member

@indirect indirect commented Jun 4, 2015

For Bundler you have to explicitly gem install bundler to get the bugfixes. You don't have to upgrade, and nothing will happen if you don't upgrade. It won't even dirty your lock in git if you don't upgrade.

After you have installed 1.10, the only time the lock will dirty is if you personally upgrade your own version of Bundler to a newer one than the one used by the project. At that point, and only at that point, you can commit the newer version to the lock. Once. And then you're done. If someone else on the project uses older versions, that's fine. They won't see a dirty lock. If you get behind, that's also fine. You won't see a dirty lock either.

@botandrose
Copy link

@botandrose botandrose commented Jun 4, 2015

If someone else on the project uses older versions, that's fine. They won't see a dirty lock.

Except for everyone who is using Bundler < 1.10. This change bifurcates all bundler users into two groups, who are going to find it frustrating to collaborate with one another. IMHO, such a bifurcation seems more appropriate for a major version bump of bundler.

@jhass
Copy link
Contributor Author

@jhass jhass commented Jun 4, 2015

You're still completely neglecting the raised points about contributor workflows.

Let's try it with another scenario.

  • I am contributor to two projects. I do not have commit rights to either, I'm just sending pull requests. Both projects use spring, thus working around with the versioned binstub Rubygems provides doesn't work, spring will make lock dirty if I happen to have a newer Bundler version
  • Project A depends on a bugfix in a newly released Bundler version, it's impossible to work on it without it.
  • Project B doesn't depend on said bugfix release of Bundler, additionally the maintainer is a bit slow, so it's unlikely that the Bundler version in their main branch will get updated soon.
  • There are other people contributing to both projects.
  • Project B likes to have a clean commit history, so using git rebase is inevitable at some point.
  • Since Project B's maintainer is a bit slow to respond, I'm working on multiple pull requests concurrently, meaning I often switch branches with git checkout.

With this situation in mind, please reread this thread or at least my initial posting.

@SuperTux88
Copy link

@SuperTux88 SuperTux88 commented Jun 4, 2015

The problem is, when I (a contributor with no direct push permissions to upstream) upgrade my bundler, then my Gemfile.lock is dirty. Then someone with upstream permissions needs to change the version in the upstream Gemfile.lock, or I have to downgrade again.

Another problem: When I want to temporarily checkout an older brach (with an older Gemfile.lock), the old Gemfile.lock gets touched and update to the current bundler-version, and I have to switch back to the last branch with "git checkout -f "

@jhass
Copy link
Contributor Author

@jhass jhass commented Jun 4, 2015

To have something constructive, if I narrow down to a possible compromise I currently see making updating the Bundler version written into the lock explicit, possibly via a new subcommand such as bundle update-itself and raising a warning if the current version is newer than the one written to the lock.

@simi
Copy link
Contributor

@simi simi commented Jun 4, 2015

It will be great if bundle command can try to reload bundler in locked version if specified and installed or print warning that you don't have locked bundler and it is recommended to install it via gem install bundler -v=1.x.

  1. run bundle
  2. parse only locked bundler from Gemfile.lock (not sure if it is possible)
  3. compare current running version and locked version
  4. decide to spawn another bundler (like bundle _x.y.z_ does) and pass all arguments or continue

There will be no need to remember what bundler version is used for specified project to use bundle _1.y.z_. And bonus points to lock your bundler in Gemfile (so it will be installed with classic bundle, whatever version you're running). The question is if it's recommended to lock your bundler in Gemfile. I'm not doing that for example.

This can be also good candidate for 2.0 plugin.

just an initial idea

@jhass
Copy link
Contributor Author

@jhass jhass commented Jun 4, 2015

Just to clarify @simi's idea since I learned you need to be very clear here, that would include to only warn if you run a newer than the locked version.

@simi
Copy link
Contributor

@simi simi commented Jun 4, 2015

But anyway I agree this is very aggressive change for minor release.

@indirect
Copy link
Member

@indirect indirect commented Jun 4, 2015

Alright. I think we've gotten as far as we can get with this particular thread.

Yes, @simi's idea is a possible solution. It includes a huge amount of work, and could easily create a lot of confusion about what version of Bundler is actually being used, but I haven't ruled it out. There are a lot of possible solutions to this problem, including silently changing the running version of Bundler at runtime, reverting this change and waiting until a later version to include the Bundler version in the lock, or even just waiting until most people have upgraded to 1.10 or higher. Which one is the best use of our (very) limited time is still unclear.

Complaints on this ticket have been split between those unhappy that 1.9 and 1.10 have a different idea of what the lock should look like, and those unhappy that future 1.10 versions will put their own version number into the lock. The first problem will simply go away when people upgrade. The second problem is a deliberately designed change, to encourage users of Bundler to install bugfixes, ensure projects track the version of Bundler that they are known to work with, and to start getting ready for the eventual Bundler 2.0 release that will include breaking changes.

It's an aggressive change, yes. However, Bundler 1.10.x has been installed about 435,000 times. There have been less than 10 people who have given feedback about this change so far, including both GitHub and Twitter. We've listened to each of your feedback, and we'll keep it in mind as we think about changes to make in the future.

With that said, I'm going to lock this thread, because +1s and rehashing the same points again is just a waste of everyone's time. Thanks for the discussion, everyone.

@rubygems rubygems locked and limited conversation to collaborators Jun 4, 2015
jhass added a commit to diaspora/diaspora that referenced this issue Jun 4, 2015
kaosf added a commit to kaosf/shirasagi that referenced this issue Jun 4, 2015
indirect added a commit that referenced this issue Jun 5, 2015
When either `bundle check` is run, or any application requires the 
`bundler/setup` file, Bundler will automatically check whether it is 
possible to lock the Bundle. During the lock process, Bundler updates 
the lock if the implicit locking changes the lock file.

Starting with the 1.10 release, Bundler includes a lockfile section 
named BUNDLED WITH that includes the version of Bundler that generated 
the lockfile. In order to minimize git churn, and guarantee that the 
lockfile will only be changed when the user runs an explicit Bundler 
command, Bundler will now only add or update the BUNDLED WITH section 
during commands where the user asks for changes to the lock. This 
includes, but is not limited to, `install`, `update`, `lock`, and 
`package`.

Running the `check` command or loading an application that uses Bundler 
will still now add or update the BUNDLED WITH section if, and only if,
the lockfile has also changed for a different reason (such as a gem 
being updated).

Simply using an application, by running  `bundle exec` commands or by 
running `bin/rails` and the like, will not change the lockfile. As a 
result, the intended workflow with the BUNDLED WITH section is now 
slightly different than it was before:

1. When running `bundle install`, Bundler will update the version in 
   the lockfile if newer than the version present.
2. Then, check in the lockfile change, exactly as you would after 
   running install to change any other gem version.
3. Older versions of Bundler will not change the number in the lock, 
   but will warn the user that they are out of date.

refs rubygems/bundler-features#80
refs #3697
lunks referenced this issue in godunno/dunno Jun 5, 2015
indirect added a commit that referenced this issue Jun 6, 2015
When either `bundle check` is run, or any application requires the 
`bundler/setup` file, Bundler will automatically check whether it is 
possible to lock the Bundle. During the lock process, Bundler updates 
the lock if the implicit locking changes the lock file.

Starting with the 1.10 release, Bundler includes a lockfile section 
named BUNDLED WITH that includes the version of Bundler that generated 
the lockfile. In order to minimize git churn, and guarantee that the 
lockfile will only be changed when the user runs an explicit Bundler 
command, Bundler will now only add or update the BUNDLED WITH section 
during commands where the user asks for changes to the lock. This 
includes, but is not limited to, `install`, `update`, `lock`, and 
`package`.

Running the `check` command or loading an application that uses Bundler 
will still now add or update the BUNDLED WITH section if, and only if,
the lockfile has also changed for a different reason (such as a gem 
being updated).

Simply using an application, by running  `bundle exec` commands or by 
running `bin/rails` and the like, will not change the lockfile. As a 
result, the intended workflow with the BUNDLED WITH section is now 
slightly different than it was before:

1. When running `bundle install`, Bundler will update the version in 
   the lockfile if newer than the version present.
2. Then, check in the lockfile change, exactly as you would after 
   running install to change any other gem version.
3. Older versions of Bundler will not change the number in the lock, 
   but will warn the user that they are out of date.

refs rubygems/bundler-features#80
refs #3697
indirect added a commit that referenced this issue Jun 8, 2015
When either `bundle check` is run, or any application requires the 
`bundler/setup` file, Bundler will automatically check whether it is 
possible to lock the Bundle. During the lock process, Bundler updates 
the lock if the implicit locking changes the lock file.

Starting with the 1.10 release, Bundler includes a lockfile section 
named BUNDLED WITH that includes the version of Bundler that generated 
the lockfile. In order to minimize git churn, and guarantee that the 
lockfile will only be changed when the user runs an explicit Bundler 
command, Bundler will now only add or update the BUNDLED WITH section 
during commands where the user asks for changes to the lock. This 
includes, but is not limited to, `install`, `update`, `lock`, and 
`package`.

Running the `check` command or loading an application that uses Bundler 
will still now add or update the BUNDLED WITH section if, and only if,
the lockfile has also changed for a different reason (such as a gem 
being updated).

Simply using an application, by running  `bundle exec` commands or by 
running `bin/rails` and the like, will not change the lockfile. As a 
result, the intended workflow with the BUNDLED WITH section is now 
slightly different than it was before:

1. When running `bundle install`, Bundler will update the version in 
   the lockfile if newer than the version present.
2. Then, check in the lockfile change, exactly as you would after 
   running install to change any other gem version.
3. Older versions of Bundler will not change the number in the lock, 
   but will warn the user that they are out of date.

refs rubygems/bundler-features#80
refs #3697
indirect added a commit that referenced this issue Jun 8, 2015
When either `bundle check` is run, or any application requires the 
`bundler/setup` file, Bundler will automatically check whether it is 
possible to lock the Bundle. During the lock process, Bundler updates 
the lock if the implicit locking changes the lock file.

Starting with the 1.10 release, Bundler includes a lockfile section 
named BUNDLED WITH that includes the version of Bundler that generated 
the lockfile. In order to minimize git churn, and guarantee that the 
lockfile will only be changed when the user runs an explicit Bundler 
command, Bundler will now only add or update the BUNDLED WITH section 
during commands where the user asks for changes to the lock. This 
includes, but is not limited to, `install`, `update`, `lock`, and 
`package`.

Running the `check` command or loading an application that uses Bundler 
will still now add or update the BUNDLED WITH section if, and only if,
the lockfile has also changed for a different reason (such as a gem 
being updated).

Simply using an application, by running  `bundle exec` commands or by 
running `bin/rails` and the like, will not change the lockfile. As a 
result, the intended workflow with the BUNDLED WITH section is now 
slightly different than it was before:

1. When running `bundle install`, Bundler will update the version in 
   the lockfile if newer than the version present.
2. Then, check in the lockfile change, exactly as you would after 
   running install to change any other gem version.
3. Older versions of Bundler will not change the number in the lock, 
   but will warn the user that they are out of date.

refs rubygems/bundler-features#80
refs #3697
FriedSock added a commit to ministryofjustice/defence-request-service that referenced this issue Jun 12, 2015
As long as everyone is running a version of Bundler
above 1.10, bundler versions will not decrease this
BUNDLED WITH version, but it can possibly increase

rubygems/bundler#3697
Koronen added a commit to Koronen/hound that referenced this issue Jun 13, 2015
jwworth added a commit to hashrocket/hr-til that referenced this issue Jun 24, 2015
@indirect
Copy link
Member

@indirect indirect commented Jun 24, 2015

We've added workarounds for BUNDLED WITH issues in 1.10.4 and 1.10.5, and the BUNDLED WITH section will no longer be added or updated during bundle install if there are no other changes to the lock. (Anyone who explicitly wants to update the BUNDLED WITH section can run bundle update --source bundler --local.) Read more about it on the Bundler blog.

fxn added a commit to rails/rails-docs-server that referenced this issue Jun 28, 2015
See the rationale in the comment present in the patch.

Related to rubygems/bundler#3697.
@fxn
Copy link
Contributor

@fxn fxn commented Jun 28, 2015

The use case in the commit referenced above is as follows: Rails docs generation is automated and designed to need as little maintenance as possible.

In particular, system dependencies are upgraded only if totally needed, and Bundler as of today is 1.7.x there. While 1.7 works, it won't be upgraded. Gemfile.lock in master has started to come up with BUNDLE WITH which is removed from Gemfile.lock by 1.7, thus leaving a modified Gemfile.lock in place that prevents pulling again later.

@segiddins
Copy link
Member

@segiddins segiddins commented Jun 28, 2015

@fxn at the moment, the only release series we're actively supporting are 1.9.x and 1.10.x, both of which can (at their latest version) handle BUNDLED WITH. The team's recommendation is to update bundler to either 1.9.10, or preferably, 1.10.5.

@fxn
Copy link
Contributor

@fxn fxn commented Jun 28, 2015

@segiddins thanks, yes in that case I'll upgrade.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.