Skip to content
This repository has been archived by the owner on Jan 16, 2021. It is now read-only.

Packages which normalize casing in import path comment don't show #507

Closed
arp242 opened this issue Aug 26, 2017 · 10 comments
Closed

Packages which normalize casing in import path comment don't show #507

arp242 opened this issue Aug 26, 2017 · 10 comments
Labels

Comments

@arp242
Copy link
Contributor

arp242 commented Aug 26, 2017

Our organisation name is Teamwork, and our packages are github.com/Teamwork/<name>.

For historical reasons, we've always used lowercase import paths: github.com/teamwork/<name>; I recently added some import comments to enforce this to prevent confusion down the line (to prevent e.g the recent logrus confusion):

package test // import "github.com/teamwork/test"

It seems that this confuses godoc.org. Any packages with such an import don't have working documentation pages.

For example for the github.com/teamwork/test example I fixed it with:
Teamwork/test@c3703d0

A still-broken example is https://godoc.org/github.com/Teamwork/mailaddress, which has the import path enforcement, but doesn't show as either https://godoc.org/github.com/Teamwork/mailaddress or https://godoc.org/github.com/teamwork/mailaddress

@arp242
Copy link
Contributor Author

arp242 commented Aug 26, 2017

I got a local development environment running, and I have this error in the logs:

2017/08/26 16:03:51 web   fetch: 411 notfound: Github import path has incorrect case. github.com/teamwork/mailaddress

Which is due to this check: https://github.com/golang/gddo/blob/master/gosrc/github.go#L130

Commenting that check out fixes it.

I'm not sure what the best way to fix it is, as this is before any Go code is parsed.

@dmitshur dmitshur added bug and removed bug labels Aug 28, 2017
@dmitshur
Copy link
Contributor

dmitshur commented Aug 28, 2017

This is a valid issue that we need to find a good solution to.

I simply want to ask this question first. Is it a viable option for you to rename your GitHub organization from Teamwork to teamwork, so that the canonical URL of your repos would match the canonical import path you've chosen to use?

@arp242
Copy link
Contributor Author

arp242 commented Aug 28, 2017

I simply want to ask this question first. Is it a viable option for you to rename your GitHub organization from Teamwork to teamwork

This is good question, but tricky because there is a lot tied in to our GitHub account (not just Go, also other languages/environments).

Predicting what will break is not so easy. Even worse, it's not easy to test if everything works, and failures related to the rename may not be noticed for days, weeks, or even months, and many of these errors would likely be confusing.


Another alternative would be to use Teamwork in the Go import paths; there's no good reason to use the lowercase variant other than that's what someone started using years ago. Renaming this everywhere is also tricky, since it's a bit of an all-or-nothing approach across many packages/repos. It would also break people's local development environments.

It's do-able, but would be fairly time-consuming/costly across the entire company. It would be more cost-effective if I spend time fixing this bug :-)

@dmitshur
Copy link
Contributor

I understand, thank you for providing that information.

I agree, although it's a bit unfortunate, I don't think what you're doing is breaking any rules, and as such, godoc.org should be able to display documentation with the current setup.

The reason some of those checks exist is because we don't want a normal github.com/user/repo import path to be available at many, many different casing variants thereof:

  • github.com/user/repo
  • github.com/User/repo
  • github.com/user/Repo
  • github.com/user/REPO
  • github.com/USER/repo
  • etc.

That would cause an huge artificial increase in the number of packages available.

At the same time, a non-github import path such as example.com/foo may be case sensitive, so example.com/foo and example.com/Foo may be two completely different Go packages. As bad of an idea as that probably is, I don't think Go disallows it... But I think it should be useful to confirm this before proceeding.

@dmitshur dmitshur added the bug label Aug 28, 2017
@arp242
Copy link
Contributor Author

arp242 commented Nov 9, 2017

I'm not entirely sure how to fix this; to get the canonical import path from the comment we need to clone the code and parse it; but to clone the code we first want to normalize the path. It's a bit of a catch-22 :-/

Any suggestions would be appreciated.

arp242 added a commit to Teamwork/gddo that referenced this issue Jun 20, 2018
Our GitHub organisation is named Teamwork, but for various hard to
change reasons we import packages as teamwork (lower case t). To enforce
this we add an import enforcement for our packages.

This works well, but godoc.org enforces users to use the canonical
GitHub path (Teamwork), but with the import statement the Go compiler
enforces a different path, leading to an error.

Fixing this is a bit of a catch-22: to get the canonical import path
from the Go comment need to clone the code and parse it; but to clone
the code we first want to normalize the path.

As a fix, this patch will allow package authors to specify the canonical
import path in the repositories README file as:

	<!-- import "github.com/canonical/path" -->

I have added this to github.com/teamwork/validate as a test:
Teamwork/validate@4b8ee77

It seems to me that using the README for this is the easiest way to
achieve this;. Another way to fix this might be to use a different file
with metadata for gddo (i.e. .gddo.yaml). I don't know if that would
make sense.

Fixes golang#507
arp242 added a commit to Teamwork/gddo that referenced this issue Jun 20, 2018
Our GitHub organisation is named Teamwork, but for various hard to
change reasons we import packages as teamwork (lower case t). To enforce
this we add an import enforcement for our packages.

This works well, but godoc.org enforces users to use the canonical
GitHub path (Teamwork), but with the import statement the Go compiler
enforces a different path, leading to an error.

Fixing this is a bit of a catch-22: to get the canonical import path
from the Go comment need to clone the code and parse it; but to clone
the code we first want to normalize the path.

As a fix, this patch will allow package authors to specify the canonical
import path in the repositories README file as:

	<!-- import "github.com/canonical/path" -->

I have added this to github.com/teamwork/validate as a test:
Teamwork/validate@4b8ee77

It seems to me that using the README for this is the easiest way to
achieve this;. Another way to fix this might be to use a different file
with metadata for gddo (i.e. .gddo.yaml). I don't know if that would
make sense.

Fixes golang#507
@dmitshur
Copy link
Contributor

dmitshur commented Jun 20, 2018

I have an idea for how to deal with the catch-22.

If my memory of how gddo works is right, I recall there's github.com-specific logic that first checks if the user/repo casing is canonical, and redirects to canonical if not. (Then, the canonical casing gives 404 because the import path comment doesn't match.)

To solve this, we can consider delaying the redirect when the casing doesn't match, and instead fetch the package as normal and see if it happens to have a matching import path comment. If yes, display the documentation. If not, then proceed to redirect to canonical casing, as done currently.

Does that make sense? It seems that it would resolve this bug, but not cause the alternative-casing repos to show up unnecessarily.

@arp242
Copy link
Contributor Author

arp242 commented Jun 21, 2018

Thanks for your feedback @shurcooL!

Simply removing this check would implement your suggestion, I think. Since redirection to the canonical import path is already done in doc.newPackage(). It seems to work well in my testing anyway.

I had assumed that this check was added to save disk space, network, or some such, but if that's not an issue, then it's an easy fix.

@arp242
Copy link
Contributor Author

arp242 commented Jun 21, 2018

Oh wait, that's only in cases where an import path is specified.

Perhaps adding a CanonicalPath to &gosrc.Directory would be a solution? Then we can move the logic to doc.newPackage().

@arp242
Copy link
Contributor Author

arp242 commented Jun 21, 2018

I changed the PR. Let me know what you think!

arp242 added a commit to Teamwork/gddo that referenced this issue Jul 12, 2018
Problem:

Our GitHub organisation is named Teamwork, but for various hard to
change reasons we import packages as teamwork (lower case t). To enforce
this we add an import enforcement for our packages:

	package duck // import "github.com/teamwork/duck"

This works well, but godoc.org enforces users to use the canonical
GitHub path (Teamwork), but with the import statement the Go compiler
enforces a different path, leading to an error.

Fix:

Instead of immediately redirecting in the gosrc package, it will now
only store the canonical import path there. This is before the Go source
code is actually scanned, so we don't *actually* know what the real
canonical path should be.

In the doc.newPackage() function it will check for the canonical import
path, taking both the Go import path as well as the reported path from
the gosrc package in to account, and redirect as needed.

This seems to work for all the cases I can think of:

github.com/Teamwork/validate    -> github.com/teamwork/validate
github.com/pkg/ERRORS           -> github.com/pkg/errors
github.com/Carpetsmoker/sconfig -> arp242.net/sconfig
arp242.net/SCONFIG              -> not found (expected)
bitbucket.org/pkg/inflect       -> works
bitbucket.org/pkg/INFLECT       -> works (should probably redirect too)
github.com/docker/docker/client -> works (golang#534)
github.com/moby/moby/client     -> redirects (on master this actually seems to error out?)

It should also be easy to add a similar check for for some other repo
providers, such as BitBucket, GitLab, etc.

Fixes golang#507
@dmitshur
Copy link
Contributor

We've merged the fix (PR #560) and deployed the new version. This is fixed now, and the following page works:

https://godoc.org/github.com/teamwork/validate

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

2 participants