New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: get -u fails with git submodules #7764

Closed
gopherbot opened this Issue Apr 11, 2014 · 15 comments

Comments

Projects
None yet
@gopherbot

gopherbot commented Apr 11, 2014

by daniel.fanjul.alcuten:

What does 'go version' print?
go version go1.2.1 linux/amd64
It happens also with go 1.2.

What steps reproduce the problem?
1. Create a git non-bare repository.
2. Create a git submodule on that repository with some go code.
3. Make GOPATH point to that git submodule
4. Run 'go get -u' to update the git submodule

What happened?
It fails with the error:
package <project>: directory "<absolute path of the project>" is
not using a known version control system

What should have happened instead?
It should have detected the git repository of the submodule and successfully updated it.

Please provide any additional information below.

It is probably because the submodule has a .git file that is not a folder:
$ cat $GOPATH/src/<project>/.git
gitdir: <superproject>/.git/modules/<submodule path>

It works after replacing the file .git by a proper fully-fledged folder.
@cznic

This comment has been minimized.

Contributor

cznic commented Apr 11, 2014

Comment 1:

Ad "Make GOPATH point to that git submodule". GOPATH is a directory path. How it
supposed to point to a file?
The go tool works with packages (in the first approximation) and supports the convention
package == folder. If GOPATH is /foo/bar then package's file with import path "baz/qux"
are sought for in _folder_ /foo/bar/src/baz/qux.
IMO #WontFix
@gopherbot

This comment has been minimized.

gopherbot commented Apr 11, 2014

Comment 2 by daniel.fanjul.alcuten:

Oh, sorry. I meant a completely different thing:
Make GOPATH point to the proper parent folder.
If the submodule is in:
<superproject>/<path>/src/<project>
With its file:
<superproject>/<path>/src/<project>/.git
Then GOPATH should be or include:
GOPATH=<superproject>/<path>
And the go get command is:
go get -u <project>
@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented May 9, 2014

Comment 3:

Labels changed: added repo-main, release-none.

@gopherbot

This comment has been minimized.

gopherbot commented Sep 17, 2014

Comment 4:

CL https://golang.org/cl/142180043 mentions this issue.

@gopherbot gopherbot added new labels Sep 17, 2014

@bradfitz bradfitz removed the new label Dec 18, 2014

@rsc rsc added this to the Unplanned milestone Apr 10, 2015

@rsc rsc removed release-none labels Apr 10, 2015

@gopherbot

This comment has been minimized.

gopherbot commented May 6, 2015

CL https://golang.org/cl/9815 mentions this issue.

@adg

This comment has been minimized.

Contributor

adg commented May 8, 2015

@tamird sent the above change. Being unsure that fetching git submodules on "go get" is the right thing to do, I asked for some background; they said:

Sure, here are a bunch of repos that would use submodules:
https://github.com/cockroachdb/c-rocksdb
https://github.com/cockroachdb/c-lz4
https://github.com/cockroachdb/c-protobuf
https://github.com/cockroachdb/c-snappy
Currently, each of these repositories maintains a local copy of the upstream underlying native library - we would much rather use a submodule, because it would save us substantial git-noise. Does that sound reasonable?

I'm not sure if this is reasonable or not. @dsymonds @bradfitz @rsc @davecheney thoughts?

@dsymonds

This comment has been minimized.

Member

dsymonds commented May 8, 2015

This feels like a use case that is niche enough (and git submodules are relatively rare) that it is not worth changing go get to support it. The VCS support in go get is already complex enough. I vote for not doing this.

@tamird

This comment has been minimized.

Contributor

tamird commented May 15, 2015

The absence of submodule support makes it impossible to conveniently wrap non go-getable repos to make them go-getable. This is a common need whenever a non-go project is used in a go project (see above).

Also, the added complexity to go get is trivial, it is just replacing a string with a slice and iterating.

@nightlyone

This comment has been minimized.

Contributor

nightlyone commented May 15, 2015

I agree with @dsymonds that this stuff might be better handled by the various vendoring tools. At least for git.

Rationale:
@tamird CL https://golang.org/cl/9815 is not the whole updating story at least for for git. Conversion between a sub-directory tree and submodule and back is not handled by this change.

Those conversions are very error prone. People also keep forgetting to commit/refresh the git submodule reference while working with a repository containing submodules.

I have even seen missing git submodule checkouts used as an advantage by putting an integration test-suite with full of ruby dependencies into a submodule and keep the go gettable sources small and neat that way.

@kylewolfe

This comment has been minimized.

kylewolfe commented Jun 11, 2015

Are there any other open issues around externals (for svn, hg, etc?) I think the need for this and other externals support increases with vendoring being added in to the go tool (https://groups.google.com/forum/#!msg/golang-dev/74zjMON9glU/4lWCRDCRZg0J).

@rsc

This comment has been minimized.

Contributor

rsc commented Jun 18, 2015

I think we should do this. It makes the vendoring proposal much more useful, because it widens the scope of things that 'go get' works for. (And I thought it was already what 'go get' did.)

rsc added a commit that referenced this issue Jun 27, 2015

cmd/go: fetch git submodules in `go get`
Change createCmd, downloadCmd, tagSyncCmd, tagSyncDefault to allow
multiple commands.

When using the vendoring experiment, fetch git submodules in `go get`,
and update them in `go get -u`.

This is a reincarnation of https://codereview.appspot.com/142180043.

For #7764.

Change-Id: I8248efb851130620ef762a765ab8716af430572a
Reviewed-on: https://go-review.googlesource.com/9815
Reviewed-by: Russ Cox <rsc@golang.org>
@akavel

This comment has been minimized.

Contributor

akavel commented Jun 30, 2015

Side note: as an ugly-ish workaround, one could probably create a link to the "sub-gopath" (ln on Unix/OSX?, mklink on Windows) to get a workspace like below:

<superproject>/.git
<superproject>/<path>/src/<project>/.git
sub_link -> <subproject>/<path>

GOPATH=sub_link

in other words, in concrete commands, e.g. below seems to work with Go 1.4:

$ mkdir -p gopath1/src/appA/vendor/src
$ ln -s gopath1/src/appA/vendor gopath_sub
$ export GOPATH=$PWD/gopath_sub:$PWD/gopath1
$ cat <<EOF > gopath1/src/appA/main.go
package main

import (
    "fmt"

    "github.com/bradfitz/iter"
)

func main() {
    for i := range iter.N(10) {
        fmt.Println(i)
    }
}
EOF
$ (cd gopath1/src/appA/ ; git add -A ; git commit -m "initial commit"  )
[master (root-commit) ae2c178] initial commit
 1 file changed, 13 insertions(+)
 create mode 100644 main.go
$ go get -v appA
github.com/bradfitz/iter (download)
github.com/bradfitz/iter
appA
$ go get -v -u github.com/bradfitz/iter
github.com/bradfitz/iter (download)
$ (cd gopath_sub/src ; go get -v -u ./... )
github.com/bradfitz/iter (download)

(although I'm not fully sure yet if that's 100% compatible with git submodules...? sorry.)

EDIT:

uhm; sorry, I may have misunderstood the original issue; for me, above script seems to work ok and as I'd expect with submodules on Go 1.4 even without the link, i.e. with GOPATH=$PWD/gopath1/src/appA/vendor:$PWD/gopath1

@kylewolfe

This comment has been minimized.

kylewolfe commented Aug 20, 2015

@rsc, just curious, when are issues like this supposed to be closed? I believe the fix was merged into 1.5.

@tamird

This comment has been minimized.

Contributor

tamird commented Aug 20, 2015

It's behind the vendor experiment; I'd expect this to stay open until GA.

@tamird

This comment has been minimized.

Contributor

tamird commented Apr 5, 2016

I think this is good to close now that 1.6 is GA.

@adg adg closed this Apr 5, 2016

@golang golang locked and limited conversation to collaborators Apr 6, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.