Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

subpackage recognised as independent package? #244

Closed
stroborobo opened this issue Feb 9, 2016 · 26 comments
Closed

subpackage recognised as independent package? #244

stroborobo opened this issue Feb 9, 2016 · 26 comments

Comments

@stroborobo
Copy link

Hi,

I'm using glide for the first time today, so maybe I just misunderstood something, but here's the problem:

After glide create, fixing(?) a private repo entry, and glide install the latter always outputs warnings that it's "Unable to checkout" a (sub)package. Glide however did put the package into the vendor dir, just as the other subpackages of the same repo (that didn't get a warning).

Here's the glide.yaml import entry I edited:

- package: src.ybit.eu/ybit/yamcha
  repo: git@src.ybit.eu:ybit/yamcha.git
  vcs: git
  subpackages:
  - /apibase
  - /helpers
  - /server
  - /storage

The one from glide.lock:

- name: src.ybit.eu/ybit/yamcha
  version: d7880c188617d3d3bbdaa991b0a597809f8d0135
  repo: git@src.ybit.eu:ybit/yamcha.git
  vcs: git
  subpackages:
  - /apibase
  - /helpers
  - /server
  - /storage
- name: src.ybit.eu/ybit/yamcha/helpers
  version: ""

The glide install --debug output (snippet):

[...]
[DEBUG] ----> Scanning src.ybit.eu/ybit/yamcha
[DEBUG] Looking in /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor/src.ybit.eu/ybit/yamcha/Godeps/ for a Godeps.json file.
[DEBUG] Looking in /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor/src.ybit.eu/ybit/yamcha/vendor/ for a manifest file.
[INFO] Scanning src.ybit.eu/ybit/yamcha for dependencies.
[DEBUG] => Scanning github.com/didip/tollbooth
[DEBUG] ✨☆ GOPATH dependency: github.com/didip/tollbooth
[DEBUG] => Scanning github.com/gorilla/mux
[DEBUG] => Scanning github.com/gorilla/sessions
[DEBUG] ✨☆ GOPATH dependency: github.com/gorilla/sessions
[DEBUG] => Scanning github.com/jinzhu/gorm
[DEBUG] => Scanning github.com/facebookgo/grace/gracehttp
[DEBUG] ✨☆ GOPATH dependency: github.com/facebookgo/grace
[DEBUG] => Scanning github.com/gorilla/handlers
[DEBUG] ✨☆ GOPATH dependency: github.com/gorilla/handlers
[DEBUG] => Scanning src.ybit.eu/ybit/yamcha/helpers
[DEBUG] ✨☆ GOPATH dependency: src.ybit.eu/ybit/yamcha/helpers
[DEBUG] => Scanning github.com/lib/pq
[DEBUG] ----> Updating all dependencies for "src.ybit.eu/ybit/yamcha" (5)
[DEBUG] ----> Already updated gopkg.in/gemnasium/logrus-airbrake-hook.v2
[DEBUG] Getting project github.com/gorilla/handlers (/Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor/github.com/gorilla/handlers)
[INFO] Fetching updates for github.com/gorilla/handlers.
[DEBUG] Attempting to find current branch for https://github.com/gorilla/handlers
[DEBUG] Saving default branch for https://github.com/gorilla/handlers
[DEBUG] ----> Already updated github.com/Sirupsen/logrus
[DEBUG] ----> Already updated golang.org/x/crypto
[DEBUG] ----> Already updated src.ybit.eu/ybit/yamcha
[DEBUG] ----> Already updated gopkg.in/airbrake/gobrake.v2
[DEBUG] ----> Already updated github.com/jinzhu/inflection
[DEBUG] ----> Already updated github.com/lib/pq
[DEBUG] Getting project github.com/didip/tollbooth (/Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor/github.com/didip/tollbooth)
[INFO] Fetching updates for github.com/didip/tollbooth.
[DEBUG] Attempting to find current branch for https://github.com/didip/tollbooth
[DEBUG] Saving default branch for https://github.com/didip/tollbooth
[DEBUG] ----> Already updated github.com/jinzhu/gorm
[DEBUG] Getting project github.com/gorilla/sessions (/Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor/github.com/gorilla/sessions)
[INFO] Fetching updates for github.com/gorilla/sessions.
[DEBUG] Attempting to find current branch for https://github.com/gorilla/sessions
[DEBUG] Saving default branch for https://github.com/gorilla/sessions
[DEBUG] Getting project src.ybit.eu/ybit/yamcha/helpers (/Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor/src.ybit.eu/ybit/yamcha/helpers)
[INFO] Fetching updates for src.ybit.eu/ybit/yamcha/helpers.
[WARN] src.ybit.eu/ybit/yamcha/helpers appears to be a vendored package. Unable to update. Consider the '--update-vendored' flag.
[WARN] Problem setting version on src.ybit.eu/ybit/yamcha/helpers: Cannot detect VCS
[DEBUG] ----> Already updated github.com/gorilla/mux
[DEBUG] ----> Already updated github.com/gorilla/context
[DEBUG] Getting project github.com/facebookgo/grace (/Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor/github.com/facebookgo/grace)
[INFO] Fetching updates for github.com/facebookgo/grace.
[...]

Also this warning right before "Project relies on X dependencies":

[WARN] Problem setting version on src.ybit.eu/ybit/yamcha/helpers: Cannot detect VCS (flatten)

Did I misunderstand something or is this a bug?

@technosophos
Copy link
Member

We're checking into this. Out of curiosity, does everything still build okay?

@stroborobo
Copy link
Author

Yes, looks fine after glide install, just tested it with an empty GOPATH dir just to make sure.

@mattfarina
Copy link
Member

@stroborobo what version of Glide are you using (glide --version) and how did you get Glide?

@stroborobo
Copy link
Author

Latest release: glide version 0.8.3

@stroborobo
Copy link
Author

Oh, via brew :)

@mattfarina
Copy link
Member

@stroborobo can you try Glide 0.9.0 RC1? It's available at https://github.com/Masterminds/glide/releases/tag/0.9.0-rc1.

The section of code that does this work was rewritten between 0.8.3 and the release candidate. It may already be fixed.

@stroborobo
Copy link
Author

Tried it, didn't fix the problem unfortunately, but the output changed:

[INFO] Lock file (glide.lock) does not exist. Performing update.
[INFO] Downloading dependencies. Please wait...
[INFO] Fetching updates for github.com/jinzhu/gorm.
[INFO] Fetching updates for github.com/Sirupsen/logrus.
[INFO] Fetching updates for golang.org/x/crypto.
[INFO] Fetching updates for src.ybit.eu/ybit/yamcha.
[INFO] Fetching updates for github.com/gorilla/mux.
[INFO] Resolving imports
[INFO] Fetching github.com/gorilla/context into /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor
[INFO] Fetching github.com/jinzhu/inflection into /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor
[INFO] Fetching github.com/lib/pq/hstore into /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor
[INFO] Fetching github.com/didip/tollbooth into /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor
[INFO] Fetching github.com/gorilla/sessions into /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor
[INFO] Fetching github.com/facebookgo/grace/gracehttp into /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor
[INFO] Fetching github.com/gorilla/handlers into /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor
[INFO] Found Godeps.json file.
[INFO] Fetching github.com/gorilla/securecookie into /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor
[INFO] Fetching github.com/facebookgo/httpdown into /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor
[INFO] Fetching github.com/juju/ratelimit into /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor
[INFO] Setting version for github.com/juju/ratelimit to aa5bb718d4d435629821789cb90970319f57bfe5.
[INFO] Fetching github.com/facebookgo/clock into /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor
[INFO] Fetching github.com/facebookgo/stats into /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor
[INFO] Downloading dependencies. Please wait...
[INFO] Fetching updates for src.ybit.eu/ybit/yamcha/helpers.
[INFO] Fetching updates for src.ybit.eu/ybit/yamcha/storage.
[WARN] src.ybit.eu/ybit/yamcha/helpers appears to be a vendored package. Unable to update. Consider the '--update-vendored' flag.
[INFO] Fetching updates for src.ybit.eu/ybit/yamcha/apibase.
[WARN] src.ybit.eu/ybit/yamcha/storage appears to be a vendored package. Unable to update. Consider the '--update-vendored' flag.
[WARN] src.ybit.eu/ybit/yamcha/apibase appears to be a vendored package. Unable to update. Consider the '--update-vendored' flag.
[INFO] Fetching updates for src.ybit.eu/ybit/yamcha/server.
[WARN] src.ybit.eu/ybit/yamcha/server appears to be a vendored package. Unable to update. Consider the '--update-vendored' flag.
[INFO] Setting references for remaining imports
[WARN] Failed to set version on src.ybit.eu/ybit/yamcha/apibase to : Cannot detect VCS
[WARN] Failed to set version on src.ybit.eu/ybit/yamcha/helpers to : Cannot detect VCS
[WARN] Failed to set version on src.ybit.eu/ybit/yamcha/server to : Cannot detect VCS
[WARN] Failed to set version on src.ybit.eu/ybit/yamcha/storage to : Cannot detect VCS
[INFO] Project relies on 21 dependencies.

I quadro-checked the repo path and import path now, definitely no typo.

This is what the yamcha repo looks like, if that helps:

yamcha/
├── Makefile
├── README.md
├── apibase
│   ├── README.md
│   ├── apibase.go
│   ├── context.go
│   └── errors.go
├── helpers
│   ├── README.md
│   ├── errcaller.go
│   ├── helpers.go
│   ├── math.go
│   ├── pidfile.go
│   ├── reader.go
│   ├── runes.go
│   └── seoshop.go
├── server
│   ├── README.md
│   └── server.go
└── storage
    ├── README.md
    └── storage.go

(The Makefile is only for putting godoc into the readmes, so no fancy building mechanism.)

@mattfarina
Copy link
Member

@stroborobo I think I've figured out the problem.

Glide follows the same path conventions Go itself does when looking at remote packages and the repos they live in. Go expects patterns like src.ybit.eu/ybit/yamcha.git/helpers. Notice the .git in there. In Go you can trace that back to a regular expression used inside go get. Glide follows this same behavior.

This has to do with Glide, or go get, walking the import statements as much as it does for the glide.yaml file.

I'm open to suggestions to handle this differently. Right now we are following what Go itself does.

@stroborobo
Copy link
Author

Now that you're saying that I've never go get'ed one of our private repos before, that really doesn't work: unrecognized import path "src.ybit.eu/ybit/seoshop/cmd/seoshop". Adding the ".git" after "yamcha" makes it fail complaining about an "insecure protocol", but it works when I enforce SSH like this:

git config --global url."git@src.ybit.eu:".insteadOf "https://src.ybit.eu/"

So I changed the package to src.ybit.eu/ybit/yamcha.git, leaving the subpackages, that didn't do it. Then I made the subs independent packages with the added .git, but also not the desired result. So what should I put in my glide.yaml then?

For all four in both cases:

[WARN] Unable to checkout src.ybit.eu/ybit/yamcha/helpers
[WARN] Skipped getting src.ybit.eu/ybit/yamcha/helpers: Cannot detect VCS

Btw, using the 0.9.0 RC1 I got this in both cases:

[...]
[INFO] Fetching updates for src.ybit.eu/ybit/yamcha.git.
[INFO] Resolving imports
[INFO] Fetching github.com/gorilla/context into /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor
[INFO] Fetching github.com/jinzhu/inflection into /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor
[INFO] Fetching github.com/lib/pq/hstore into /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor
[ERROR] Error scanning src.ybit.eu/ybit/yamcha/apibase: open /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor/src.ybit.eu/ybit/yamcha/apibase: no such file or directory
[ERROR] Error scanning src.ybit.eu/ybit/yamcha/helpers: open /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor/src.ybit.eu/ybit/yamcha/helpers: no such file or directory
[ERROR] Error scanning src.ybit.eu/ybit/yamcha/server: open /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor/src.ybit.eu/ybit/yamcha/server: no such file or directory
[ERROR] Error scanning src.ybit.eu/ybit/yamcha/storage: open /Users/bo/src/go/src/src.ybit.eu/ybit/seo360/vendor/src.ybit.eu/ybit/yamcha/storage: no such file or directory
[INFO] Downloading dependencies. Please wait...
[INFO] Fetching updates for src.ybit.eu/ybit/yamcha/apibase.
[INFO] Fetching updates for src.ybit.eu/ybit/yamcha/server.
[INFO] Fetching updates for src.ybit.eu/ybit/yamcha/helpers.
[INFO] Fetching updates for src.ybit.eu/ybit/yamcha/storage.
[WARN] Unable to checkout src.ybit.eu/ybit/yamcha/server
[WARN] Update failed for src.ybit.eu/ybit/yamcha/server: Cannot detect VCS
[WARN] Unable to checkout src.ybit.eu/ybit/yamcha/apibase
[WARN] Update failed for src.ybit.eu/ybit/yamcha/apibase: Cannot detect VCS
[WARN] Unable to checkout src.ybit.eu/ybit/yamcha/storage
[WARN] Update failed for src.ybit.eu/ybit/yamcha/storage: Cannot detect VCS
[WARN] Unable to checkout src.ybit.eu/ybit/yamcha/helpers
[WARN] Update failed for src.ybit.eu/ybit/yamcha/helpers: Cannot detect VCS
[ERROR] Could not update packages: Cannot detect VCS
Cannot detect VCS
Cannot detect VCS
Cannot detect VCS

However I still wonder why that's important when I'm specifically saying where the repo is and what kind of VCS to use.

@davidzhao
Copy link

I'm having the same issue here with a private git repo. How does glide fetch private/internal repos?

@elan100cs
Copy link

Hey i am having the same issues. I suggest that if it is already specified within the glide.yaml as an import package, then it shouldn't try to update it subpackages.

I imagine to implement this, requires a key value pair which stores packages that's already been updated during the 'glide update' and do a check whether a package which about to be updated is a subpackage of that.

@itscaro
Copy link

itscaro commented Feb 29, 2016

@mattfarina

Is it possible to append to vcsList (util.go) something like following so that GetRootFromPackage() return correctly the root package in this case:

    {
        pattern: `^(?P<rootpkg>[A-Za-z0-9.\-]+/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[A-Za-z0-9_.\-]+)*$`,
    },

This regex is based on my case: (same naming as github.com/user/repo)

- package: domain/first_level/package
  vcs: git
  repo: git@domain/first_level/package.git

@mattfarina
Copy link
Member

@itscaro This is something we're going to have to really think about. To change the patterns is to break compatibility with go get naming conventions. That's why we do what we do.

We've gotten enough requests we will talk about it.

@davidzhao
Copy link

@mattfarina I think this is a fairly important issue as it relates to the ability to use private packages. (hosted on GH or elsewhere). Go get works fine, but the VCS host detection fails to interpret git@github.com:user/repo pattern. We really like the design of glide, but are unable to use it until we could properly reference our private packages.

@mattfarina
Copy link
Member

@davidzhao A couple things that may help...

First, you can use Glide with private packages using something like this in your glide.yaml file...

- package: github.com/user/repo
  repo: git@github.com:user/repo
  vcs: git

This will tell Glide to use the Git repo at that location for a package with that path. If you're going off GitHub or using GitHub Enterprise as your own domain the package name may need a mild variation.

Second, I'm working to make repo detection for URLs like git@github.com:user/repo easier.

@davidzhao
Copy link

@mattfarina Thanks for the tip! your workaround solved that problem. Tho I ran into a related problem with subpackages.

Our glide config looks like this

- package: davidzh/privaterepo
  repo: git@github.com:davidzh/privaterepo
  vcs: git
  subpackages:
  - subpackage1
  - subpackage2

While it fetches the privaterepo just fine, it gave me the following errors at the end:

[ERROR] Failed to set version on davidzh/privaterepo/subpackage1 to : Cannot detect VCS
[ERROR] Failed to set version on davidzh/privaterepo/subpackage2 to : Cannot detect VCS

Is this related to private repo?

@rschmukler
Copy link

I am also running into the same issue that @davidzhao is experiencing. Worth noting that I'm using an https+git endpoint instead of git via ssh

@mattfarina
Copy link
Member

@davidzhao I think I know what happened. Glide expects the package to be a go compatible path. That means the package should be github.com/davidzh/privaterepo. The glide.yaml docs for package touch on this. If you want to dig into the Go code look here and here. Does that help?

@rschmukler
Copy link

@mattfarina for me the repo is a Go getable path myurl.com/my-package/path. It is worth noting that it is a Gogs private repo, so perhaps it could be an issue with gogs lacking go-import metadata on private repos (See gogits/gogs/#2825)?

edit: Looking into the source code it looks like it attempts to ping the repo... So I will bet that for me, the issue will be resolved with go-import metadata being provided (but I only just skimmed the source)

@bryanpaluch
Copy link

bryanpaluch commented Apr 20, 2016

I had a similar issue where a private repo that had 3 subpackages wasn't being created properly in the glide.yaml. I fixed the issue the same way @stroborobo does in this first comment by adding the submodules by had and removing the extra packages. I hit the same issue when I was trying to do a glide update/install and get the can not detect VCS type. It doesn't make sense to me that glide would try to look in the subpackages folder for the VCS info as I think it is. I created a work around which adds packages to the alreadySeen map if they are subpackages in dependency/resolver.go

for _, dep := range r.Config.Imports {
  for _, sub := range dep.Subpackages {
    alreadySeen[dep.Name+"/"+sub] = true
  }
}

bryanpaluch@7deada1

I'm not entirely sure if this breaks the purpose of subpackages but it produces a valid glide.lock and my project builds fine now.

I don't think this is a good workaround but maybe it helps identify the problem a little better.

@jmatosp
Copy link

jmatosp commented Jul 18, 2016

Hi Guys I'm having a related issue

Using a package
"github.com/Sirupsen/logrus"

and a sub package
"github.com/Sirupsen/logrus/formatters/logstash"

When doing a glide install says OK
but glide update says

...[ERROR] Error scanning github.com/Sirupsen/logrus/formatters/logstash: open /Users/jose.pinto/projects/go/src/github.com/jmatosp/hawkeye/vendor/github.com/Sirupsen/logrus/formatters/logstash: no such file or directory...

my glide.yaml looks similar to

package: github.com/jmatosp/hawkeye
import:
- package: github.com/Sirupsen/logrus
  subpackages:
  - formatters/logstash
- package: github.com/hashicorp/consul
  subpackages:
  - api
- package: github.com/julienschmidt/httprouter
- package: golang.org/x/net
  subpackages:
  - context

any help is very much appreciated

@goloveychuk
Copy link

goloveychuk commented Aug 4, 2016

@mattfarina
How to reproduce problem.
go version go1.6 darwin/amd64
glide version 0.11.1

- package: github.com/golang/protobuf
  subpackages:
  - proto

when running glide install first time, when no glide.lock exists, it's ok.
Now we're running glide install - everthing ok.
But after removing rm -rf vendor
while running glide install it produces

[WARN] Unable to checkout github.com/golang/protobuf/proto
[ERROR] Update failed for github.com/golang/protobuf/proto: Unable to get repository

@goloveychuk
Copy link

The solution could be having

- name: github.com/golang/protobuf
  version: c3cefd437628a0b7d31b34fe44b3a7a540e98527
  subpackages:
  - proto
- name: github.com/golang/protobuf/proto
  version: c3cefd437628a0b7d31b34fe44b3a7a540e98527

in your glide.lock file, remove subpackage lines

- name: github.com/golang/protobuf
  version: c3cefd437628a0b7d31b34fe44b3a7a540e98527
  subpackages:
  - proto

this will work nicely.

@mattfarina
I think when glide.lock is generating it shouldn't write subpackages into lock file. (also they same commit number as parent, so it don't have any additional information)

@marioluan
Copy link

Any news about this issue being fixed? :)

@leninmehedy
Copy link

leninmehedy commented Aug 9, 2016

@mattfarina similar to public repository, it should not try to set version for subpackages of a private repo in glide.lock file.

When I remove the lines (i.e. name and version) related to subpackages from glide.lock file, it works nicely.

@AnoopPutta
Copy link

@mattfarina @stroborobo @technosophos @davidzhao @elan100cs

Even i am facing this issue just with glide update. For any packages which has dependent sub pacakges while resolving imports it prompts for password. I can understand removing the sub package stuff from glide.lock before I execute glide install. But in my case glide update is also prompting for password. I Have gone through the code ,when the dependency struct is initialised repo literal are not set as per its parent repo details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests