Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't use modules on a branch other than master #41535

Open
dwschulze opened this issue Sep 21, 2020 · 25 comments
Open

Can't use modules on a branch other than master #41535

dwschulze opened this issue Sep 21, 2020 · 25 comments

Comments

@dwschulze
Copy link

@dwschulze dwschulze commented Sep 21, 2020

What version of Go are you using (go version)?

go version go1.15.1 linux/amd64

What operating system and processor architecture are you using (go env)?

Ubuntu 18.04 with amd64

This code works as long as I'm on the master branch:

main.go:

package main

import (
	datemodlocal "192.168.0.12/gitrepo/go-module-test-dateutil.git"
	stringmodlocal "192.168.0.12/gitrepo/go-module-test-stringutil.git"
	"fmt"
	"github.com/dwschulze/go-module-test-dateutilmod"
	"github.com/dwschulze/go-module-test-stringutilmod"
)

func main() {

	fmt.Println("github:  " + stringmod.ToUpperCase("test"))
	fmt.Println("github:  " + datemod.GetTime().String())
	fmt.Println("local:  " + stringmodlocal.ToUpperCase("test"))
	fmt.Println("local:  " + datemodlocal.GetTime().String())
}

go.mod:

module module-driver

require (
	192.168.0.12/gitrepo/go-module-test-dateutil.git v0.0.1
	192.168.0.12/gitrepo/go-module-test-stringutil.git v0.0.1
	github.com/dwschulze/go-module-test-dateutilmod v0.0.1
	github.com/dwschulze/go-module-test-stringutilmod v0.0.1
)

go 1.15

I need to use the branch dev2 for development. The docs don't show what needs to be done to the import path or the require statement in go.mod. If I change the import statement to:

`datemodlocal "192.168.0.12/gitrepo/go-module-test-dateutil.git@dev2"`

I get:

$ go run main.go
package command-line-arguments
        imports 192.168.0.12/gitrepo/go-module-test-dateutil.git@dev2: can only use path@version syntax with go get

If I move the @dev2 to the require statement in go.mod

`192.168.0.12/gitrepo/go-module-test-dateutil.git@dev2 v0.0.1`

I get

$ go run main.go
go: 192.168.0.12/gitrepo/go-module-test-dateutil.git@dev2@v0.0.1: unrecognized import path "192.168.0.12/gitrepo/go-module-test-dateutil.git@dev2": https fetch: Get "https://192.168.0.12/gitrepo/go-module-test-dateutil.git@dev2?go-get=1": dial tcp 192.168.0.12:443: connect: connection refused

That error message says https which is strange since in my ~/.gitconfig I have

[url "dean@192.168.0.12:"] insteadOf = https://192.168.0.12/

Setting GOPRIVATE has no effect. If I put the @dev2 in both places I get the same error messages.

The docs seem to indicate that modules can be used on branches, but they don't show how the import path or the require statement in go.mod need to be changed.

Go modules need to be able to be used on branches other than master.

@seankhliao
Copy link
Contributor

@seankhliao seankhliao commented Sep 21, 2020

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Sep 21, 2020

Sean - that link doesn't address the use of branches with modules. No combination of adding the branch name to the import path in main.go or the require statement in go.mod works. I'm writing code, not doing go get from the command line.

Has anyone even tried to use modules on a branch other than master?

@bcmills
Copy link
Member

@bcmills bcmills commented Sep 21, 2020

@dwschulze, per https://golang.org/ref/mod#version-queries (emphasis mine):

A version query may be one of the following:

  • A revision identifier for the underlying source repository, such as a commit hash prefix, revision tag, or branch name. If the revision is tagged with a semantic version, this query selects that version. Otherwise, this query selects a pseudo-version for the underlying commit. Note that branches and tags with names matched by other version queries cannot be selected this way. For example, the query v2 selects the latest version starting with v2, not the branch named v2.

Import paths are paths, not versions: they indicate only the broad API in use, not the specific version of the API. (Version information goes in the go.mod file, not the .go source file.)

@bcmills
Copy link
Member

@bcmills bcmills commented Sep 21, 2020

The error message from your second attempt (with the version in the go.mod file) seems to indicate that there was still a @dev2 suffix on the import path and/or module path. The branch name is a version query: it goes in place of the version, not the path.

Please try the import and module path 192.168.0.12/gitrepo/go-module-test-dateutil.git (per https://tip.golang.org/cmd/go/#hdr-Remote_import_paths).

Per https://golang.org/ref/mod#vcs-branch, use the command go get -d 192.168.0.12/gitrepo/go-module-test-dateutil.git@dev2 to update the version to the head of that branch.

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Sep 22, 2020

Do I understand you correctly: In order to use modules on a branch I have to do a go get with the branch name? I thought modules used go mod download ... to get the code from vcs and a manual go get ... was not needed.

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Sep 22, 2020

The blog you pointed to doesn't say anything about pushing your module to a private vcs, let alone what the code should look like in main.go and go.mod. I need code that works, not examples of go get ... or go mod download ....

I'll ask again: Have you verified that this works in code? I think you have a serious bug here.

Please prove or disprove this with a simple main.go and go.mod that uses a branch other than master from a private repo.

@bcmills
Copy link
Member

@bcmills bcmills commented Sep 22, 2020

@dwschulze, most of the tests for module-mode behavior are in the src/cmd/go/testdata/script. You are welcome to read through them, and if you find cases that are under-tested please contribute improvements.

I'll ask again: Have you verified that this works in code? I think you have a serious bug here.

I'm asking you: have you tried the specific commands that I suggested above? If so, please post the output — if you are correct that there is a serious bug, then we need specific steps to reproduce it, ideally as a script test that we can commit to the repo along with a fix.

@beoran
Copy link

@beoran beoran commented Sep 22, 2020

For the record, I have been using go modules on non master branches for more than a year now and it works fine. It takes a while to get used to go modules though, so please take the time needed to learn.

In particular, the version query @Branch will work in a go.mod file, but not in an import statement, and not in combination with a version query. You must use either a branch name or a version number, not both.

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Sep 22, 2020

Let me see if I have this correct. If I use master then I don't have to download anything. I type go run main.go and the modules are downloaded from master and the code works.

Do I understand you correctly that if I want to use a branch other than master that I have to execute go get ... manually? I've tried go get ... and it fails:

$ go get 192.168.0.12/gitrepo/go-module-test-dateutil.git@dev2
go get 192.168.0.12/gitrepo/go-module-test-dateutil.git@dev2: can't request explicit version of path in main module

$ go get 192.168.0.12/gitrepo/go-module-test-dateutil@dev2
go get 192.168.0.12/gitrepo/go-module-test-dateutil@dev2: unrecognized import path "192.168.0.12/gitrepo/go-module-test-dateutil": https fetch: Get "https://192.168.0.12/gitrepo/go-module-test-dateutil?go-get=1": dial tcp 192.168.0.12:443: connect: connection refused

What finally worked was go mod download ...:

$ go mod download 192.168.0.12/gitrepo/go-module-test-dateutil.git@dev2

It downloaded a pseudo version from the branch dev2. So it looks like I have to manually download the code if I use a branch other than master.

But now there's some magic going on that I don't understand. If I delete the directories under GOPATH/pkg/mod where the code was downloaded to I can run the code again and it downloads the code from the dev2 branch. It looks like the cache/ directory is helping out so I delete the cache/ directory along with the directories containing the downloaded module code. But the program still runs and downloads the code again.

I also created a v0.0.2 tag on the dev2 branch and specified that version in my require statement and when I ran the code it automatically downloaded the v0.0.2 code from the dev2 branch without doing the manual go mod download .... Now for some reason it knows about the dev2 branch.

Is there another secret cache somewhere that prevents me from having to run go mod download ... again? This is very confusing. First it doesn't work until I do the manual download with go mod download ..., but after that it's as if it knows how to use the dev2 branch to download code without being told about the dev2 branch (there's nothing in the import and requires statements specifying the dev2 branch).

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Sep 22, 2020

@beoran - How do you use version numbers on your branches, or do you not use version numbers on branches? Omitting version numbers seems like it would defeat one of the biggest features of Go modules.

In my case I will have to use version numbers on non-master branches.

@beoran
Copy link

@beoran beoran commented Sep 23, 2020

What is running on 192.168.0.12? A plain git repository, or a web based solution like gitlab or gitea? Is sshd also running?

You cannot go get a specific version of a main package, but you can go download or go run it. Look at your go.mod file as well as your GOPATH directory to see what is happening.

If you want to use go mod versions with git, you need to create git tags like 0.4.5 or 1.2.3 and go mod will recognize them as versions. You can place such version tags on any commit in any branch.

But as I said before, to modules are relatively complex, please study the documentation well. If the documentation is not clear enough, then maybe there is something we can do about it.

@seankhliao
Copy link
Contributor

@seankhliao seankhliao commented Sep 23, 2020

@dwschulze the module system resolves all versions to commits (which it records as a pseudoversion) or tags (which it assumes will never move). Branches are resolved to the latest commit and recorded at such, there is no way for you to say "always use the latest commit from branch x"

@bcmills
Copy link
Member

@bcmills bcmills commented Sep 23, 2020

@dwschulze, originally you posted an example go.mod file with the declaration module module-driver.

The error message in #41535 (comment) says can't request explicit version of path in main module, which tells me that the go.mod file at that point declares some other module path: either module 192.168.0.12/gitrepo/go-module-test-dateutil.git or module 192.168.0.12/gitrepo or module 192.168.0.12, containing some package at that offset.

The module path declared in your go.mod file is the prefix of the import paths for the packages contained in that module. It should generally match the location of the repository where you are planning to host that module. The module path in your go.mod file must not be the same as the path for any dependency that you plan to use: only one version of each module can be selected, and the selected version of the main module itself is always “the code in the main module”.

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Sep 23, 2020

@beoran - There is no web server running on the git repo. It was created with --bare and is accessed through ssh.

The necessary information is spread throughout the documentation. It needs to be in one place (using modules with private repos). It needs to be complete. Some things like go get not recognizing hostnames unless they include a '.' are not documented at all. There are changes that have to be made to ~/.gitconfig. The issues with using branches that are not master are not covered at all. Modules work just fine as long as you are on master but don't work at all on non-master branches until you do go mod download ... with the branch name. Then they continue to work even after deleting the downloaded code and cache/ directory. I'm still trying to understand what is going on with that last part.

@beoran
Copy link

@beoran beoran commented Sep 23, 2020

We can probably edit https://github.com/golang/go/wiki/Modules and add a chapter on local use, perhaps with your input once you figure it out.

@seankhliao
Copy link
Contributor

@seankhliao seankhliao commented Sep 23, 2020

the new reference page should be https://golang.org/ref/mod with the relevant sections being:

gitconfig is relevant to all private repos, not just module mode, though it might help to put it in there

Then they continue to work even after deleting the downloaded code and cache/ directory. I'm still trying to understand what is going on with that last part.

Indicates you fixed the actual error instead of repeating the same commands with the exact same state

@fizmat
Copy link

@fizmat fizmat commented Sep 23, 2020

At first I liked that go installs dependencies automagically. After reading this and running into similar issues, there is a reason no other language works this way. Stop trying to be too smart with many different things that can break, have a simple and well-documented workflow!

  • Cannot import local from non-local (let me shoot myself in the foot you tin can, I will figure the correct way later! Also, what do you mean non-local, this is my PC with my code in a local directory)
  • Auto-guessing package path is git when it's a local directory named this way because every tutorial insists packages should be named like that. Now it wants to download it from a not-yet-updated repo instead of using the code that's right here

Sorry, learning a new language is annoying...

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Sep 23, 2020

Since this started working with the dev2 branch, but I don't know why it works when I no longer tell it about the branch I thought I'd try this out on a different machine. Using the same main.go and go.mod on the different machine here is what happened:

$ go run main.go
go: 192.168.0.12/gitrepo/go-module-test-dateutil.git@v0.0.3: reading 192.168.0.12/gitrepo/go-module-test-dateutil.git/go.mod at revision v0.0.3: unknown revision v0.0.3

v0.0.3 is only on the dev2 branch, so I repeat the step that got things working on my other machine. From the same directory as my main.go and go.mod are in:

$ go mod download 192.168.0.12/gitrepo/go-module-test-dateutil.git@dev2
go: 192.168.0.12/gitrepo/go-module-test-dateutil.git@v0.0.3: reading 192.168.0.12/gitrepo/go-module-test-dateutil.git/go.mod at revision v0.0.3: unknown revision v0.0.3

This looks like what beoran was talking about. The Go toolchain can't deal with a non-master branch and version numbers. So I changed it to v0.0.1 which is on master as well as dev2 branches and tried again:

$ go mod download 192.168.0.12/gitrepo/go-module-test-dateutil.git@dev2
go: 192.168.0.12/gitrepo/go-module-test-dateutil.git@v0.0.1: reading 192.168.0.12/gitrepo/go-module-test-dateutil.git/go.mod at revision v0.0.1: unknown revision v0.0.1

Tried again leaving off the branch name:

$ go mod download 192.168.0.12/gitrepo/go-module-test-dateutil.git
go: 192.168.0.12/gitrepo/go-module-test-dateutil.git@v0.0.1: reading 192.168.0.12/gitrepo/go-module-test-dateutil.git/go.mod at revision v0.0.1: unknown revision v0.0.1

Try with GOPRIVATE set:

$ export GOPRIVATE=192.168.0.12
$ go run main.go
go: 192.168.0.12/gitrepo/go-module-test-dateutil.git@v0.0.1: reading 192.168.0.12/gitrepo/go-module-test-dateutil.git/go.mod at revision v0.0.1: unknown revision v0.0.1

I've got to conclude that Go modules don't work with non-master branches. The latest attempt doesn't even work with a version that is on master.

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Sep 23, 2020

@fizmat - I agree that Go tries to be too smart when it comes to dependencies. Go mixes concerns. The download IP address is in the source code as an import statement. Having an IP address of a dependency in the source code is an architectural smell, IMO. It's tight coupling as well. Maven avoids this kind of coupling by moving IP addresses to the build.xml so they're not in the source code.

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Sep 23, 2020

@beoran - I'm going to write this up once I get some time (I've got to get some work done - this has been quite a diversion). I'll show all the steps and where it quit working. It worked on one machine, but not on another. I have no idea why it finally started working on one machine.

@beoran
Copy link

@beoran beoran commented Sep 23, 2020

Could you please give us the output of 'git tag' and 'git branch' on the http://192.168.0.12/gitrepo/go-module-test-dateutil.git?

As I said before, the go toolchain can handle versions that are not on the master branch of git perfectly fine, although a go proxy might be needed in some cases.

I would recommend reading the docs that @seankhliao posted, they are new and seem to explain everything in great detail.

@dwschulze
Copy link
Author

@dwschulze dwschulze commented Sep 24, 2020

@beoran:

$ git remote --v
origin dean@ubuntu-18-extssd:gitrepo/go-module-test-dateutil (fetch)
origin dean@ubuntu-18-extssd:gitrepo/go-module-test-dateutil (push)
$ git branch
dev2

  • master
    $ git tag
    v0.0.1
    v0.0.2
    v0.0.3
    $ git checkout dev2
    Switched to branch 'dev2'
    Your branch is up to date with 'origin/dev2'.
    $ git tag
    v0.0.1
    v0.0.2
    v0.0.3

Note that ubuntu-18-extssd has IP address 192.168.0.12.

I've read those docs and they don't address the problems that I'm seeing or the way I need to use modules (use ssh to access git on a host with a private IP address on non-master branch).

@seankhliao
Copy link
Contributor

@seankhliao seankhliao commented Sep 24, 2020

with a remote like:

* d444619 (HEAD -> dev2, origin/dev2) dev2 8 minutes ago <Sean Liao>
* 2cf57c9 (tag: v0.0.3) v0.0.3 52 minutes ago <Sean Liao>
| * 1a19339 (origin/main, main) main 47 minutes ago <Sean Liao>
|/
* 64a4fa3 (tag: v0.0.2) v0.0.2 53 minutes ago <Sean Liao>
* 6862c30 (tag: v0.0.1) v0.0.1 54 minutes ago <Sean Liao>
* b2bff7a init 54 minutes ago <Sean Liao>

your workflow should be something like:

#!/bin/sh

set -ex

# clean environment
export T=$(mktemp -d)
CACHE=$(mktemp -d)
export GOMODCACHE=${CACHE}

# git host access
IP=34.72.69.170
U=arccy
export GOPRIVATE=${IP}
git config --global url."${U}@${IP}:".insteadOf "https://${IP}/"

# develop
cd ${T}
go mod init test/41535

cat << EOF > main.go
package main

import "fmt"
import test41535 "${IP}/repos/test41535.git"

func main() {
    fmt.Println("hello", test41535.Version)
}
EOF

# (optional) show all tagged versions of dependency
go list -m -versions ${IP}/repos/test41535.git
# get a specific version
go get -d ${IP}/repos/test41535.git@dev2
# run
go run .
@dwschulze
Copy link
Author

@dwschulze dwschulze commented Dec 9, 2020

I've written up a long answer covering how to use packages and modules from git repos on private servers. Towards I cover the issue of using branches other than master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants