Updating gems from git + branch #2165

Closed
jlecour opened this Issue Nov 15, 2012 · 36 comments

Comments

Projects
None yet

jlecour commented Nov 15, 2012

Hi,

I'm maintaining a gem and using it in a project.

During the development phase, I want to try the gem's code in the project, without releasing many beta builds. So I use the :git feature of Bundler to checkout the gem's source code from Git.

In the gem repository, I've created a topic branch (and an alpha version number).
In the project Gemfile, I've set this : gem "my_gem", :git => "git@example.com:my_gem.git", :branch => 'my_branch'

The first time I bundle it's fine ; Bundler fetches the repository, checks out the right branch.

But if I update the gem's source code, push it on the remote repository, and updates the project's bundle, nothing changes. I've verified with bundle open my_gem that the code is not updated.

I don't know if I'm missing something obvious here or if this is a "bug".

Thanks for the collective effort of making Bundler more and more awesome.

jlecour commented Nov 15, 2012

If I delete the "cache" directory, and bundle update my_gem it continues to checkout the wrong Git revision (expected, since it did not change in Gemfile.lock).

If I change the :branch with a :ref I get the correct revision, but I need to change it everytime. I thought a bundle update my_gem would force an update of the Git repository and the revision pointed by the specified branch.

crohr commented Nov 15, 2012

Weird, I actually don't use :branch but directly :ref (I forgot :branch was available), as in:

gem "my_gem", :git => "git@example.com:my_gem.git", :ref => 'my_branch'

and it correctly updates the Gemfile.lock when I run bundle update my_gem and the branch has a new revision. You could try that, and if it doesn't work it might help to know your bundler version.

I'm using Bundler 1.2.x and:

  • run this in my project directory:

    bundle config --local local.my_gem /path/to/my_gem

  • have this in my Gemfile:

    gem 'my_gem', git: 'some_git_repo_somewhere', branch: 'anything'

The local override will do the rest. Does it help?

jlecour commented Nov 15, 2012

@crohr Indeed, it works this way, but it's strange.

@franckverrot It doesn't help since I want to deploy, so the "local" config seems a little hardcore.

Thanks guys for your answers.

Something strange with the "put the branch name in the :ref attribute" trick ; the branch name is truncated to 7 chars, just as if it's a SHA1.
And given that it works with this tricks, I think even more that my issue is related to a bug.

Oh shoot, yeah OK. :)

Owner

indirect commented Nov 15, 2012

Try running "bundle update gemname". That will update to the latest push in your branch. If that doesn't work, please include the output from "bundle config".

On Nov 15, 2012, at 3:04 AM, Jérémy Lecour notifications@github.com wrote:

Hi,

I'm maintaining a gem and using it in a project.

During the development phase, I want to try the gem's code in the project, without releasing many beta builds. So I use the :git feature of Bundler to checkout the gem's source code from Git.

In the gem repository, I've created a topic branch (and an alpha version number).
In the project Gemfile, I've set this : gem "my_gem", :git => "git@example.com:my_gem.git", :branch => 'my_branch'

The first time I bundle it's fine ; Bundler fetches the repository, checks out the right branch.

But if I update the gem's source code, push it on the remote repository, and updates the project's bundle, nothing changes. I've verified with bundle open my_gem that the code is not updated.

I don't know if I'm missing something obvious here or if this is a "bug".

Thanks for the collective effort of making Bundler more and more awesome.


Reply to this email directly or view it on GitHub.

@ghost

ghost commented Nov 19, 2012

I am experiencing the same issue.

"bundle update my_gem" does not grab the latest revision from the repo.

# Gemfile
gem 'my_gem', git: 'git@my:private-repo.git', branch: 'master'
$ bundle config
Settings are listed in order of priority. The top value will be used.

bin
  Set for your local app (/home/ben/Projects/ic/api/.bundle/config): "bin"

EDIT:
bundler 1.2.2

Owner

indirect commented Nov 20, 2012

@jlecour, I haven't been able to reproduce this yet. When I run bundle update gitgemname, it updates to the latest commit on github for the branch that I have put in the :branch option. :\ Can you provide more information?

@ronaldmcdollars, please see ISSUES for troubleshooting instructions.

@ghost

ghost commented Nov 20, 2012

@indirect, thanks. I've seen the issues section. I was commenting here because my issue is the same as @jlecour has described. I can open an additional ticket if that is preferred, but I would consider it a duplicate of this. I don't mean to hijack @jlecour's ticket, but the issue is causing my team a lot of grief.

Additional information...

We have a private git repo with three gems in it, structured like so...

example.git
/
|-- gem1/
|   |-- gem1.gemspec
|-- gem2/
|   |-- gem2.gemspec
|-- gem3/
|   |-- gem3.gemspec

When we run "bundle update" it updates the gems from our repo to their latest push (along with all the other gems). But when we run what is supposed to be the more targeted "bundle update gem1" it does not update the specified gem to the latest push.

Also, we noticed that when making any change to the Gemfile, like say changing from using branch: 'master' to ref: 'master' as suggested by @crohr will work one time, but then the issue reappears. Plus, I don't think that is how ref is meant to be used.

Owner

indirect commented Nov 20, 2012

@ronaldmcdollars, it sounds like there's some sort of bug in the update unlocking code, but I haven't been able to reproduce this issue. Whenever I have used bundle update gemname, it has updated that gem to the latest commit. Can you provide reproduction instructions?

jlecour commented Nov 20, 2012

@indirect I have the same situation as @ronaldmcdollars ; my Git repository contains multiple gems, organized in sub-folders.

I've not tried to reproduce with a simple/single gem.

jlecour commented Nov 20, 2012

I should explicit the structure :

internal_tools/
|-- core/
|   |-- lib/
|   |   |-- ns-core/
|   |   |   \-- version.rb
|   |   \-- ns-core.rb
|   \-- ns-core.gemspec
|
|-- adapters/
|   |-- lib/
|   |   |-- ns-adapters/
|   |   |   \-- version.rb
|   |   \-- ns-adapters.rb
|   \-- ns-adapters.gemspec
|
\-- services/
    |-- lib/
    |   |-- ns-services/
    |   |   \-- version.rb
    |   \-- ns-services.rb
    \-- ns-services.gemspec

Here is a more accurate inclusion in the Gemfile : gem 'ns-adapters', :git => 'git@example.comt:internal_tools.git', :branch => 'my_branch'.

@ghost

ghost commented Nov 20, 2012

Here are some lengthy steps to reproduce the issue in isolation. I hope this helps!

mkdir git-branch-issue
cd git-branch-issue

# Set up a gem repository
# ==================================================
mkdir repo
cd repo
bundle gem gem1
bundle gem gem2
bundle gem gem3

# BSD/OSX
sed -i '' 's/TODO: //g' */*.gemspec # Remove TODO: from the gemspecs
# LINUX
sed -i 's/TODO: //g' */*.gemspec # Remove TODO: from the gemspecs

git init
git add .
git commit -m "Initial commit"

# Unfortunately, you'll have to create a test github repo for this
# If you'd like me to do that for you I can create one and add you as a collaborator

git remote add origin YOUR_TEST_GITHUB_REPO
git push -u origin master
cd ..

# Set up our project that uses bundler
# ==================================================
mkdir project
cd project
bundle init
vim Gemfile

#--- GEMFILE ---
source "https://rubygems.org"

gem 'gem1',
  git: 'YOUR_TEST_GITHUB_REPO',
  branch: 'master'

gem 'gem2',
  git: 'YOUR_TEST_GITHUB_REPO',
  branch: 'master'

gem 'gem3',
  git: 'YOUR_TEST_GITHUB_REPO',
  branch: 'master'
#--- /GEMFILE ---

bundle
ack revision Gemfile.lock #--> Make note of revision
cd ..

cd repo
# BSD/OSX
sed -i '' 's/VERSION = "0.0.1"/VERSION = "0.0.2"/g' */lib/*/version.rb   # Bump the versions
# LINUX
sed -i 's/VERSION = "0.0.1"/VERSION = "0.0.2"/g' */lib/*/version.rb   # Bump the versions

git add .
git commit -m "Bumping version numbers"
git push origin master
cd ..

cd project
bundle update gem1
# Note that it does not say "Updating [my repo name]"
#--- OUT ---
Fetching gem metadata from https://rubygems.org/..
Using gem1 (0.0.1) from https://github.com/ronaldmcdollars/git-branch-issue.git (at master) 
Using gem2 (0.0.1) from https://github.com/ronaldmcdollars/git-branch-issue.git (at master) 
Using gem3 (0.0.1) from https://github.com/ronaldmcdollars/git-branch-issue.git (at master) 
Using bundler (1.2.2) 
#--- /OUT ---
ack revision Gemfile.lock  #--> Revision is the same as noted earlier

bundle update
#--- OUT ---
Updating https://github.com/ronaldmcdollars/git-branch-issue.git
Username: 
Password: 
Unpacking objects: 100% (14/14), done.
Fetching gem metadata from https://rubygems.org/..
Using gem1 (0.0.2) from https://github.com/ronaldmcdollars/git-branch-issue.git (at master) 
Using gem2 (0.0.2) from https://github.com/ronaldmcdollars/git-branch-issue.git (at master) 
Using gem3 (0.0.2) from https://github.com/ronaldmcdollars/git-branch-issue.git (at master) 
Using bundler (1.2.2) 
#--- /OUT ---
ack revision Gemfile.lock  #--> Revision is updated
Owner

indirect commented Nov 22, 2012

Thanks for the reproduction steps. Sounds like it's probably a bug in the way that repos with multiple gems are handled.

jlecour commented Nov 22, 2012

@ronaldmcdollars Wow, thanks for the detailed steps.

As a side note, unrelated to the bug, Git being a DVCS you should be able to point to your local repository and spare the GitHub repository part, Bundler won't even notice the difference.

In my case, my gems repository is not on GitHub, but on an internal server

@ghost

ghost commented Nov 23, 2012

@indirect @jlecour No problem. @jlecour Good point about repo location.

Contributor

xaviershay commented Aug 18, 2013

I edited your repro to a script, removed need for github, and confirmed that gem1 does not update until a full bundle update is run.

Expected behaviour is for gem1 to be updated to latest version, but for gem2 and gem3 to stay on the old revision. My guess is that we try to optimize by sharing a git cache for the three gems, but that is only valid if they're all on the same sha. Given that using single gems from a single repo isn't that common a use case, it makes sense to just have a different git repo checkout for each. (If someone wants to optimize that later, that's fine.)

#!/bin/bash

set -ex

ISSUE=2165
ROOT_DIR=/tmp/repro-$ISSUE
REPO_DIR=$ROOT_DIR/repo
PROJECT_DIR=$ROOT_DIR/project

mkdir -p $ROOT_DIR
rm -rf $REPO_DIR
rm -rf $PROJECT_DIR
mkdir -p $REPO_DIR
mkdir -p $PROJECT_DIR
cd $ROOT_DIR

# Set up a gem repository
# ==================================================
cd $REPO_DIR
bundle gem gem1
bundle gem gem2
bundle gem gem3

# BSD/OSX
sed -i '' 's/TODO: //g' */*.gemspec # Remove TODO: from the gemspecs
# LINUX
# sed -i 's/TODO: //g' */*.gemspec # Remove TODO: from the gemspecs

git init
git add .
git commit -m "Initial commit"

# Unfortunately, you'll have to create a test github repo for this
# If you'd like me to do that for you I can create one and add you as a collaborator

# Set up our project that uses bundler
# ==================================================
cd $PROJECT_DIR
bundle init

cat > Gemfile <<RUBY
source "https://rubygems.org"

gem 'gem1',
  git: 'file://$REPO_DIR',
  branch: 'master'

gem 'gem2',
  git: 'file://$REPO_DIR',
  branch: 'master'

gem 'gem3',
  git: 'file://$REPO_DIR',
  branch: 'master'
RUBY

bundle
grep revision Gemfile.lock #--> Make note of revision

cd $REPO_DIR
# BSD/OSX
sed -i '' 's/VERSION = "0.0.1"/VERSION = "0.0.2"/g' */lib/*/version.rb   # Bump the versions
# LINUX
# sed -i 's/VERSION = "0.0.1"/VERSION = "0.0.2"/g' */lib/*/version.rb   # Bump the versions

git add .
git commit -m "Bumping version numbers"

cd $PROJECT_DIR
bundle update gem1
# Note that it does not say "Updating [my repo name]"
# Fetching gem metadata from https://rubygems.org/..
# Using gem1 (0.0.1) from https://github.com/ronaldmcdollars/git-branch-issue.git (at master) 
# Using gem2 (0.0.1) from https://github.com/ronaldmcdollars/git-branch-issue.git (at master) 
# Using gem3 (0.0.1) from https://github.com/ronaldmcdollars/git-branch-issue.git (at master) 
# Using bundler (1.2.2) 

grep revision Gemfile.lock  #--> Revision is the same as noted earlier

bundle update
#--- OUT ---
# Updating https://github.com/ronaldmcdollars/git-branch-issue.git
# Username: 
# Password: 
# Unpacking objects: 100% (14/14), done.
# Fetching gem metadata from https://rubygems.org/..
# Using gem1 (0.0.2) from https://github.com/ronaldmcdollars/git-branch-issue.git (at master) 
# Using gem2 (0.0.2) from https://github.com/ronaldmcdollars/git-branch-issue.git (at master) 
# Using gem3 (0.0.2) from https://github.com/ronaldmcdollars/git-branch-issue.git (at master) 
# Using bundler (1.2.2) 
#--- /OUT ---
grep revision Gemfile.lock  #--> Revision is updated

dvdplm commented Nov 12, 2013

I have the same issue.

ameketa commented Jan 10, 2014

👍

mhenry commented Jan 17, 2014

This is still broken for me and the workaround described by @jlecour does not work.

dpehrson commented Feb 7, 2014

Chiming in with a +1. Even with a repo that is just a standalone gem, referenced by branch, does not appear it wil look for new commits to said branch when installing or updating the bundle.

I seem to have the exact same issue:

Bundler 1.6.1
gem "my_gem", :git => "git@my-private-repo.git", branch: "master"

When bundle update my_gem, the gem is not updated from git.

Workaround is to e.g. remove the gem from gemfile, bundle, add it again, bundle

Another workaround is to bundle update, but that updates all gems, which I might not want.

Owner

indirect commented Apr 15, 2014

isn't this fixed in 1.6.3?

On Tue, Apr 15, 2014 at 5:05 AM, Lasse Skindstad Ebert
notifications@github.com wrote:

Another workaround is to bundle update, but that updates all gems, which I might not want.

Reply to this email directly or view it on GitHub:
#2165 (comment)

Thanks @indirect. It seems to have been solved in 1.6.2.

I know this issue is closed, but I just realize that I am running into this error with Bundler 1.7.2. I am loading two rubygems from different branches in the same (private) Github repository.

Sorry that I cannot share the code, but perhaps somebody can confirm this with most recent Bundler AND the examples posted in comments above?

Also, please guide me, would it be proper to reopen this issue, or open an new one?

Owner

TimMoore commented Sep 8, 2014

@jesperronn please do open a new issue and include the output from bundle env if you can.

vbklv commented Dec 7, 2015

The issue persists with the latest Bundler.

Owner

indirect commented Dec 7, 2015

@vbklv please follow ISSUES.

Tuuleh referenced this issue in SumOfUs/Champaign Jun 13, 2016

Merged

Force update the share progress gem. #552

skylarmb commented Aug 25, 2016 edited

Hi, its 2016 now and still having this issue on the latest bundler (1.12.5) as of writing this.

An easy solution I came across that works for me is to just comment the line out of the Gemfile, run bundle install, uncomment it, run bundle install again. Bundler then fetches the latest ref from github and updates Gemfile.lock with the correct ref. Better solution than bundle update as you really (most of the time) dont want to update all your gems, and avoids you having to manually update / define a ref in your Gemfile.

Owner

indirect commented Aug 26, 2016

@skylarmb please follow ISSUES.

rcrogers commented Jan 6, 2017

Still seeing this in 2017 for Bundler version 1.13.7

Using bundler (1.13.7) I can observe same issue:

In the Gemfile:

...
gem 'custom_gem', git: 'git@github.com:organization/custom_gem.git'
...

After first bundle install:

...
Using custom_gem 0.1.0 from git@github.com:organization/custom_gem.git (at master@743a980)
...

Which is fine, because latest commit on the master has the same SHA eq to 743a980.

After second bundle install with new existing commits on the master branch for custom_gem:

...
Using custom_gem 0.1.0 from git@github.com:organization/custom_gem.git (at master@743a980)
...

bundle update custom_gem - helps, and I can see bundle install uses latest changes for custom_gem:

Using custom_gem 0.1.0 from git@github.com:organization/custom_gem.git (at master@34hfe13)

However, shouldn't bundle install fetch the latest master for the gem from the github?

Member

andremedeiros commented Mar 29, 2017

bundle install will simply meet the dependencies in the state that is described by Gemfile.lock. This is what guarantees reproducible builds.

In other words, bundle install should never install gem versions that your code has never run against.

@andremedeiros Thank You for clear answer. @andremedeiros Do You consider use of ref along with github link to the gem in Gemfile, as a valid option to get latest sources from latest changes on the master? Or there is another better way to do it, besides using tags and released versions?

Member

andremedeiros commented Mar 29, 2017

This is all personal opinion, so take it with a grain of salt.

I wouldn't release to production anything that pointed to a git source with the ref master. Typically master branches have the latest, and potentially breaking changes that might affect how your application interacts with that particular dependency.

Using a version (or a tag) at least guarantees that no bundle update operations will break for that one dependency.

In terms of keeping it updated, you need to explicitly run bundle update <gem> (and do check out one of the latest improvements, --conservative.)

Again, this gives you the guarantee that the code you ran on your development environment, and that you checked works and passes the tests on your CI is exactly the same that will run in your production environment. In no case will bundle install change versions that you've used and that are recorded in your Gemfile.lock.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment