Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go get downloads all directories except the directory specified on bitbucket #42450

Closed
dwschulze opened this issue Nov 8, 2020 · 21 comments
Closed

Comments

@dwschulze
Copy link

@dwschulze dwschulze commented Nov 8, 2020

What version of Go are you using (go version)?

$ go version
1.14

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env

What did you do?

In my git repo (bitbucket.org) I have

reponame/
	.git/
	integration1/
	integration2/
	integration-common/
	└── util
	    ├── go.mod
	    └── readascii.go

The module directory is a subdirectory of the git repository. The repository is tagged with v0.0.1.

Here's the go.mod

	module bitbucket.org/orgname/reponame/integration-common/util
	go 1.14

When I do go get for this module I get

go get bitbucket.org/orgname/reponame/integration-common/util@v0.0.1
go: downloading bitbucket.org/orgname/reponame v0.0.1
go get bitbucket.org/orgname/reponame/integration-common/util@v0.0.1: module bitbucket.org/orgname/reponame@v0.0.1 found, but does not contain package bitbucket.org/orgname/reponame/integration-common/util

I can see the integration-common/util/ directory in bitbucket if I look with the web browser. The go.mod has the contents shown above.

What is even more odd is that go get does manage to download the other directories in the repository (integration1 and integration2). For some reason it thinks the v0.0.1 tag contains all the other directories except for the integration-common/util directory, even though bitbucket shows it there for that tag.

I download with ssh, not https, because https reqires 2fa on our bitbucket repo. I have this in my ~/.gitconfig:

`[url "git@bitbucket.org:"]insteadOf = https://bitbucket.org/`

I know bitbucket is weird when it comes to using go get. Is there something I missed, or is the a bug with go get?

What did you expect to see?

I wanted to see one directory from a repo pulled down.

What did you see instead?

go get pulled all the other directories instead of the one specified

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Nov 9, 2020

Following the third paragraph of the docs I created the tag

git tag integration-common/util/v0.0.1

and verified that it got pushed to bitbucket. When I go get the tag it shows

go get bitbucket.org/orgname/reponame@integration-common/util/v0.0.1
go: bitbucket.org/orgname/reponame/integration-common/util/v0.0.1 => v0.0.1

but nothing gets downloaded. A silent failure.

I also tried this

go get bitbucket.org/liqid/northbound/common-liqid-integration/lctlutil/@v0.0.1

but it returns immediately and does nothing.

@mvdan
Copy link
Member

@mvdan mvdan commented Nov 9, 2020

Have you tried asking this as a question first? https://golang.org/wiki/Questions

It's likely this has something to do with the way you set up the module(s). You could also try using the latest version of Go.

@mvdan mvdan added the WaitingForInfo label Nov 9, 2020
@bcmills
Copy link
Member

@bcmills bcmills commented Nov 9, 2020

@dwschulze, the argument to go get is in general a module path and version, not a repo and tag.

The command you are probably looking for here is

go get bitbucket.org/orgname/reponame/integration-common/util@v0.0.1

with the desired commit tagged integration-common/util/v0.0.1 (per https://golang.org/ref/mod#vcs-version).

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 9, 2020

That said, in general we recommend only one module per repo. (Is there a particular reason you need to have the util package in a separate module?)

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Nov 9, 2020

I've tried the command

go get bitbucket.org/orgname/reponame/integration-common/util@v0.0.1

with the tag integration-common/util/v0.0.1 and it doesn't download anything.

One module per repo could lead to proliferation of repos, which we could probably handle. But what do groups that put everything into one repo do with modules? (I've heard Google has one huge repo for all of their code which I find hard to believe.)

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 9, 2020

In general it is fine to have all of your packages in one big module in one big repo.

(IMO the only really compelling use-case for multi-module repos is to be able to tag independent releases of packages within that repo. But that seems like a rare use-case, and often isn't worth the overhead of maintaining multiple modules.)

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Nov 10, 2020

@mvdan - I've asked this question on both the golang-nuts forum and stackoverflow. Thanks for asking. I got one response that didn't fix anything.

As far as upgrading to a newer version of Go, was there a bug fix relevant to this issue? If not that sounds like the mindless advice you get by calling an 800 number for tech. support for a childrens computer game.

This is a serious problem and I need to either get a bug fixed or find out if it is a documentation problem.

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 10, 2020

As far as upgrading to a newer version of Go, was there a bug fix relevant to this issue?

There have been fixes in the module system in every release since Go 1.11. Many of the recent improvements are specifically to provide clearer diagnostics.

Also note that the issue template specifically asks “Does this issue reproduce with the latest release?”, and the latest supported releases are go 1.15.4 and go 1.14.11 respectively. An issue posted for go 1.14 (without a minor release) does suggest a reasonable likelihood that something about the issue may have changed in the ~year since the development freeze began for the initial 1.14 release.

Finally, note that our Gopher values include to “be patient” and “avoid … snarking”. We have the common goal of making Go development simple and productive. I understand that you're feeling frustrated, but please keep comments constructive.

@mvdan
Copy link
Member

@mvdan mvdan commented Nov 10, 2020

Thanks for having asked the question elsewhere. A significant portion of modules issues raised here are essentially questions phrased as bug reports, so that's why I tend to ask.

And, like Bryan says, trying the latest stable release is indeed part of the bug reporting process. Go 1.14.x is still supported, but you should still check if your problem has been solved in 1.15.x (or even the upcoming 1.16, once beta1 is out in a few weeks).

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Nov 10, 2020

I added the -v to go get but it gives no output and downloads the same thing (all the subdirectories except the one I'm asking for)

$ go get -v bitbucket.org/orgname/reponame/integration-common/util/@v0.0.1

Something unusual is that that the downloaded dirs/files have permission 500.

The directory I'm trying to get was created on a branch as was the tag v0.0.1. That directory doesn't exist on any other branch. I can see the directory under the tag on the bitbucket UI. My understanding is that tags are independent of branches.

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 10, 2020

@dwschulze, what is the output of go list -versions -m bitbucket.org/orgname/reponame for your repo? The symptom you are describing is consistent with the tag not existing.

You could also try using a totally different version for the nested module to remove any ambiguity. (What happens if you apply the tag integration-common/util/v0.0.2 and then try the same steps with version v0.0.2 instead of v0.0.1?)

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 10, 2020

I could also imagine that you might be running into issue #37438, which is fixed at head but not in any prior release.

(You could confirm that one way or the other by trying the same steps using golang.org/dl/gotip or a go command built from source at head.)

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Nov 10, 2020

I upgraded to go v1.15.4. Same results.

$ go list -versions -m bitbucket.org/orgname/reponame
bitbucket.org/orgname/reponame

I added the tag integration-common/util/v0.0.2 and pushed. I can see it in bitbucket with my directories and files in it. When I do

go get -v bitbucket.org/orgname/reponame/integration-common/util/@v0.0.2

it downloads reponame@v0.0.1 however with all the directories except the one I'm asking for.

I deleted the v0.0.1 tag

git push --delete origin integration-common/util/v0.0.1

and ran the same go get with @v0.0.2 and it still downloads reponame@v0.0.1.

Something is really haywire.

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 10, 2020

When I do

go get -v bitbucket.org/orgname/reponame/integration-common/util/@v0.0.2

it downloads reponame@v0.0.1 however with all the directories except the one I'm asking for.

Ok, that really sounds like #37438. 😞

Try gotip or a go command built from head? I'm hoping it's fixed (I literally just spent the past ~month or more on it), but if not I'd rather know sooner than later.

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 10, 2020

$ go list -versions -m bitbucket.org/orgname/reponame
bitbucket.org/orgname/reponame

That's possibly as expected? What about go list -versions -m bitbucket.org/orgname/reponame/integration-common/util? (If the nested path has its own go.mod file, it is its own module, distinct from bitbucket.org/orgname/reponame.)

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Nov 10, 2020

$ go list -versions -m bitbucket.org/orgname/reponame/integration-common/util
bitbucket.org/orgname/reponame/integration-common/util v0.0.2

Makes sense since the go.mod is in the util/ directory. But v0.0.1 is gone so why should go get download it? Is it pulling the v0.0.1 off of the proxy even though I have bitbucket.org/orgname/* set in GOPRIVATE?

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 10, 2020

0.0.1 is gone so why should go get download it?

Are you sure it is downloading, and not just pulling from the local cache? (That's in either $GOMODCACHE or $GOPATH/pkg/mod.)

If the repo is private, the public proxy can't possibly have the module cached. (The public proxy does not receive credentials or module content from the go command.)

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Nov 10, 2020

I deleted the cache.
I installed gotip as you suggested and ran gotip get with v0.0.2 and got the same results. It downloads v0.0.1 which doesn't even exist in bitbucket.org and downloads all the directories except for the one I asked for.

Later today I'm going to setup a public bitbucket repo and try to reproduce this. I'll send you the repo once I get it set up and if it reproduces.

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Nov 11, 2020

I took out all the require and replace statements in the go.mod and then go get started working. Whatever it was in go.mod that was causing the problem it makes no sense at all that go get would download every directory in the repo except for the one I was asking for.

I don't know if this is a bug or just an FAQ.

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 11, 2020

@dwschulze, you didn't mention any require or replace statements in the original report!

That could certainly explain it, but we'd need more detail to figure out how that maps to the behavior you observed.

@gopherbot
Copy link

@gopherbot gopherbot commented Dec 11, 2020

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this Dec 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants