New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: add support for Azure DevOps Git repos to go get #28236

Open
MrWako opened this Issue Oct 16, 2018 · 18 comments

Comments

Projects
None yet
6 participants
@MrWako

MrWako commented Oct 16, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.11 windows/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOHOSTARCH=amd64
GOHOSTOS=windows
GOOS=windows

What did you do?

I created a few public Azure DevOps repos to illustrate the problems

Then when trying to get the package with the main function I get:

go get -v dev.azure.com/markawakefield/GoGetTest/_git/app
Fetching https://dev.azure.com/markawakefield/GoGetTest/_git/app?go-get=1
Parsing meta tags from https://dev.azure.com/markawakefield/GoGetTest/_git/app?go-get=1 (status code 200)
package dev.azure.com/markawakefield/GoGetTest/_git/app: unrecognized import path "dev.azure.com/markawakefield/GoGetTest/_git/app" (parse https://dev.azure.com/markawakefield/GoGetTest/_git/app?go-get=1: no go-import meta tags ())

I think because go get does not know its a Git repo, and there are no meta tags supplied to indicate that it is. If I try to just get one of the other packages with a .git extension I get:

go get -v dev.azure.com/markawakefield/GoGetTest/_git/stringfmt.git
# cd .; git ls-remote https://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt
Logon failed, use ctrl+c to cancel basic credential prompt.
fatal: could not read Username for 'https://dev.azure.com': terminal prompts disabled
# cd .; git ls-remote git+ssh://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt
ssh: connect to host dev.azure.com port 22: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
# cd .; git ls-remote ssh://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt
ssh: connect to host dev.azure.com port 22: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
package dev.azure.com/markawakefield/GoGetTest/_git/stringfmt.git: cannot download, git://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt uses insecure protocol

I think because the .git extension on the URL is not being passed to the git commands, and therefore it hits an authenticated endpoint.

What did you expect to see?

I expected (hoped) to see the packages add to the GOPATH under the dev.azure.com directory.

What did you see instead?

No files added to the GOPATH

I appreciate that this is not a pure issue with the current Go tool chain, more a miss-match between the conventions of Go and Azure DevOps, and has been noted elsewhere including http://ppanyukov.github.io/2017/02/01/golang-with-vsts-repos.html. Please let me know if a proposal is more appropriate.

For other cloud providers (IBM/Apache/OpenStack) there is logic to accomodate these differences and for Azure DevOps its actually a little worse than just the go get issue as even if this command completed the package ends up under a _git directory and so will be ignored in some instances due to the underscore.

Until now we have used Glide for package management which has allowed us to work around the issue but we would now like to adopt versioned packages so the issue is becoming more pressing.

I'll open a PR with a small revision addressing the issue that can be discussed/reviewed.

Thanks

@MrWako MrWako referenced a pull request that will close this issue Oct 16, 2018

Open

cmd/go/internal/get: fix go get for Azure DevOps Git repos #28240

@gopherbot

This comment has been minimized.

gopherbot commented Oct 16, 2018

Change https://golang.org/cl/142678 mentions this issue: cmd/go/internal/get: fix go get for Azure DevOps Git repos

@FiloSottile FiloSottile changed the title from Cannot go get from Azure DevOps Git repos to cmd/go: add support for Azure DevOps Git repos to go get Oct 16, 2018

@FiloSottile FiloSottile modified the milestones: Unplanned, Go1.12 Oct 16, 2018

@rsc

This comment has been minimized.

Contributor

rsc commented Oct 17, 2018

@FiloSottile, it would be nice to say what the decision is when you add the NeedsDecision label.

@rsc

This comment has been minimized.

Contributor

rsc commented Oct 17, 2018

Note that GitHub generates the <meta> tags that the go command needs, so that GitHub-hosted code works with go get regardless of domain (e.g., GitHub Enterprise, although github.com has the tags too). At this point it would be nice if hosting sites like dev.azure.com would put the tags in and therefore own their meaning, instead of making the go project source code the source of truth. (That made sense early on when no one had heard of Go and not many sites cared.)

@MrWako

This comment has been minimized.

MrWako commented Oct 17, 2018

thanks @rsc - meta tags on the Azure side is definitely another solution route, although due to the .../_git/... in the URLs, it would be more involved than MS just adding these to the existing address. Azure could, or I could, host vanity URLs with the meta tags pointing to the existing repo URLs but this is starting to get more involved and I'd really appreciate a simpler solution.

@MrWako

This comment has been minimized.

MrWako commented Oct 18, 2018

Just adding the links to the issues on the Azure DevOps side:

to indicate that the mismatch between Azure DevOps and Go is a real problem for developers that have no choice but to use Azure DevOps, but want to champion Go.

@FiloSottile

This comment has been minimized.

Member

FiloSottile commented Oct 18, 2018

@FiloSottile, it would be nice to say what the decision is when you add the NeedsDecision label.

Will do. In this instance I'd like an owner of cmd/go to decide if we are still expanding the special case mappings, and if so what transformations should be applied to this one, as it seems to be the only one that magically adds a component to the path.

For the record, I do believe adding <meta> tags at the desired path would serve Azure DevOps best, providing them with more control, flexibility (no issues with adding path components at the right spot) and probably compatibility as there might be tools that extracted the mappings from an existing version of Go (although they should use the go tool instead).

@MrWako

This comment has been minimized.

MrWako commented Oct 18, 2018

@bketelsen perhaps you could add your experience, thanks.

@crsdrw

This comment has been minimized.

crsdrw commented Oct 18, 2018

@rsc Whilst I think it would be a good idea for dev.azure.com to add tags, even if they did, I don't think that would solve the problem. As @MrWako says, because dev.azure.com repository paths have a .../_git/... in them go get results in folders that start with an underscore. One solution to this problem would be to "fix" the go tooling like go list so that it can handle folders that start with an underscore but it seems like an easier solution is to change go get. With the added benefit of getting cleaner import paths for dev.azure.com repositories.

It might be a hard sell to convince MS to host vanity urls for dev.azure.com repositories because of the way go tooling handles folders that start with an underscore, although we are trying.

@FiloSottile

This comment has been minimized.

Member

FiloSottile commented Oct 18, 2018

To be clear I'm not suggesting having import paths with _git in them, but serving meta tags from https://dev.azure.com/markawakefield/GoGetTest/app[/*] pointing to https://dev.azure.com/markawakefield/GoGetTest/_git/app.

@crsdrw

This comment has been minimized.

crsdrw commented Oct 18, 2018

@FiloSottile I doubt MS will be able to serve pages at those exact urls as they might conflict with other pages, for example there already pages like https://dev.azure.com/markawakefield/GoGetTest/_workitems, https://dev.azure.com/markawakefield/GoGetTest/_build.

@FiloSottile

This comment has been minimized.

Member

FiloSottile commented Oct 18, 2018

If they all start with _ they shouldn't ever conflict with Go import paths. Also, the meta tag is not user visible, so it can be served on any page, even unrelated, to describe the import path to repository mapping.

Anyway, I'm not the relevant owner, I was just suggesting that meta tags would be a solution that MS can control, deploy independently, and it would be backwards compatible instead of only working with Go 1.12 and with tools that didn't hardcode the existing set.

@rsc

This comment has been minimized.

Contributor

rsc commented Oct 24, 2018

What is the name space under (say) dev.azure.com/markawakefield/GoGetTest? What kinds of names can appear after the underscore?

@rsc

This comment has been minimized.

Contributor

rsc commented Oct 24, 2018

Unless Go users really want to see _git in their import paths, the fix here should really be on dev.azure.com's side.

@MrWako

This comment has been minimized.

MrWako commented Oct 29, 2018

Hi @rsc names that can appear after the underscore are basically all the kinds that Azure has for its task tracking/development process, so things like _workitems (for you assigned workitems), _sprints (for sprint related views), _boards (for kanban-ish views), etc.

@MrWako

This comment has been minimized.

MrWako commented Oct 29, 2018

Thanks all, this has been a healthy discussion, how do we try and get to a decision?

  • if the PR was to be accepted thats great and would be an easy way forward for Azure Go programmers
  • if not, then I well understand the arguments and at least we've explored and discounted the option. Either Azure can pick up the challenge or communicate that its up to developers to address.

Coming to a decision is probably nearly as important than what the decision is so that I can make plans what to do next. Thanks,

@crsdrw

This comment has been minimized.

crsdrw commented Oct 29, 2018

@rsc, @FiloSottile I think perhaps for historical reasons, they also serve pages (or redirect) if you put the name of a project team after the name of the project. For example: dev.azure.com/crsdrw/GoGetTest/BackendTeam/_git/app

@kaylangan

This comment has been minimized.

kaylangan commented Nov 5, 2018

I'm a Program Manager for Azure DevOps. We've had this request several times over the past few years and I'd love if we could come to a solution for our Go users.

While meta tags don't exist for Azure DevOps, adding those alone will not solve the problem. As others have noted, the crux of the issue is the presence of "_git" in the URLs. We use "_git" as a marker for routing, along with other markers like "_workitems", _dashboards", etc. Our Git URLs specify organization, project, and repo, but have also historically included team name with some of those components being optional (or omitted) if they are identical. Our Git URLs also require "_git" (except for SSH, but I believe that go git doesn't support SSH?).

What is the name space under (say) dev.azure.com/markawakefield/GoGetTest? What kinds of names can appear after the underscore?

@rsc There is a fixed set of names that can come after an underscore (e.g. _git, _workitems, _builds); project names cannot start with an underscore.

To be clear I'm not suggesting having import paths with _git in them, but serving meta tags from https://dev.azure.com/markawakefield/GoGetTest/app[/*] pointing to https://dev.azure.com/markawakefield/GoGetTest/_git/app.

@FiloSottile Unfortunately, this won't work. This scheme matches https://dev.azure.com/{organization}/{project}/{team-name} and redirects to the dashboards directory page: https://dev.azure.com/{organization}/{project}/_dashboards/directory?teamId={team-name}. We've looked at other formats but we run into collisions without "_git".

It sounds like the hesitation is around adding another special case mapping to cmd/go? Is there more information that I could provide that might make a decision on this easier?

@FiloSottile

This comment has been minimized.

Member

FiloSottile commented Nov 5, 2018

This scheme matches https://dev.azure.com/{organization}/{project}/{team-name} and redirects to the dashboards directory page: https://dev.azure.com/{organization}/{project}/_dashboards/directory?teamId={team-name}. We've looked at other formats but we run into collisions without "_git".

I'm not sure I understand. You're in control of the meta tags, can't you just not serve a tag at https://dev.azure.com/{organization}/{project}/_dashboards/...?

If there is no way to map a space without underscores to the correct repositories, then doing the mapping in cmd/go won't help, and you need underscores in import paths, which will break many things we don't have control over. If there is a way to do such a mapping, you can do it by serving meta tags just as well as we can do it in cmd/go, and it will be more backwards compatible, better serving our shared users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment