Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: Go 2: cmd/go: allow relative imports #20883

Open
cch123 opened this issue Jul 2, 2017 · 29 comments

Comments

@cch123
Copy link
Contributor

commented Jul 2, 2017

I think this is a useful feature for library writers, sometimes dependency can not be avoided for a lib.
eg. a proxy lib may use a sql parser written by third party.

The library package layout may look like follows:

myproxy: host on github or sth
  main.go
  └── import "github.com/xxx/sqlparser"
  └── import "github.com/yyy/sometcppoolpackage"

If I want to make the dependency invisible to lib endusers, currently I have to move the dependencies to internal package, and manually change the import path to, maybe:

import "github.com/myname/dbproxy/internal/github.com/xxx/sqlparser"

It's quite ugly and inconvenient.

If internal can be imported easily, then import path will be,

import "internal/github.com/xxx/sqlparser"

quite clear

I noticed that some libs also have inner package, and they currently layout their project like the following:

mqtool is hosted on github.com/xxx/mqtool

mqtool/
  main.go
   └─ import "github.com/xxx/mqtool/cli"
  otherfile.go
  └─ import "github.com/xxx/mqtool/cli"
  └─ import "github.com/xxx/mqtool/logutil"
  ... many many file
  otherfileend.go
  └─ import "github.com/xxx/mqtool/bytes2"

If someone wants to fork this mqtool lib to fix bugs or maintain it as his own, then he will meet a lot problems (to modify endless import paths).

Someone may say that he can git clone this project to the correct path, but go get can not support such case, which is really weird (because go get will put this project to the forked path, not the original path).

If internal package can be found in relative path, I think things can go much better :-)

Then the above project may become like this:

mqtool is hosted on github.com/xxx/mqtool

mqtool/
  main.go
   └─ import "internal/cli"
  otherfile.go
  └─ import "internal/cli"
  └─ import "internal/logutil"
  ... many many file
  otherfileend.go
  └─ import "internal/bytes2"

@gopherbot gopherbot added this to the Proposal milestone Jul 2, 2017

@gopherbot gopherbot added the Proposal label Jul 2, 2017

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jul 17, 2017

This is not really about internal at all, since you could have the same situation with non-internal subdirectories you want to import. For better or worse we've decided that having fully-qualified paths is useful for being able to understand imports when you see them. It's not clear we want to break that now.

@rsc rsc changed the title proposal: make internal package can be found in the relative path proposal: cmd/go: allow relative "internal/..." imports Jul 17, 2017

@rsc rsc closed this Jul 17, 2017

@leonklingele

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2017

Can this be reopened as a proposal for Go 2 or should I create another proposal?

I'm especially asking for something similar to this:

mqtool is hosted on github.com/xxx/mqtool

mqtool/
  main.go
   └─ import "internal/cli"
  otherfile.go
  └─ import "internal/cli"
  └─ import "internal/logutil"
  ... many many file
  otherfileend.go
  └─ import "internal/bytes2"

It doesn't make sense to hard-code the platform / domain I'm going to host my code on inside the code itself.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2017

I'll reopen for Go 2.

@ianlancetaylor ianlancetaylor changed the title proposal: cmd/go: allow relative "internal/..." imports proposal: Go 2: cmd/go: allow relative "internal/..." imports Aug 16, 2017

@leonklingele

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2017

While I'm not a fan of internal/.., something like this ("project-relative imports") should be considered for Go 2. internal could be a package name, can it? So this could potentially break existing code.
Instead, I'd prefer import "/x/y" (leading slash indicates project-relative import, / is the root of the project)

Hard-coding the platform / domain one intends to host code under should not be part of the code itself, moving code between different hosting providers (or even renaming the project) should be easier and should not require a.) changes to the source code itself (s/old-domain/new-domain/) b.) the use of a self-hosted go-import-providing server: example which gets the source from diverse hosting platforms but uses a single, static custom domain name.

This proposal basically is a requirement for proposal: incrementally modify the Go toolchain to work without GOPATH

@dlsniper

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2017

I would discourage relative imports to begin with. They can lead to unexpected results. The current model of paths being GOPATH dependent works really well and can be adapted towards a package oriented workflow rather easily.

This proposal basically is a requirement for proposal: incrementally modify the Go toolchain to work without GOPATH

I would disagree about that.

@leonklingele

This comment has been minimized.

Copy link
Contributor

commented Oct 10, 2017

I would discourage relative imports to begin with. They can lead to unexpected results

For example?

I would disagree about that.

Why? How else would the Go compiler toolchain know if an import should be retrieved from $GOPATH / vendor/ (i.e. external packages) or from the current project?

@johnDoe2018

This comment has been minimized.

Copy link

commented Dec 11, 2017

I'd share another example where this could be useful. Let's assume there is a project with the following structure:


github.com/janeRoe/myapp

    • main.go - imports models, handlers, and routes:
import (
	"github.com/janeRoe/myapp/handlers"
	"github.com/janeRoe/myapp/models"
	"github.com/janeRoe/myapp/routes"
)
    • handlers/
    • models/
    • routes/

I want to fork the project and make a few changes to, say, routes. I click GH "Fork" button and get a copy of the project that now lives under github.com/johnAnonDoe/myapp. But the main.go's imports still point to the old "github.com/janeRoe/myapp/...". That means the following requirements cannot be satisfied at the same time:

  1. Changes to the routes/ subpackage must affect my fork (which, in case of absolute import paths, requires update of the main.go).
  2. My fork should be go getable.
  3. It should be easy to merge the changes back to the initial repo.

@ianlancetaylor ianlancetaylor changed the title proposal: Go 2: cmd/go: allow relative "internal/..." imports proposal: Go 2: cmd/go: allow relative imports Feb 27, 2018

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Feb 27, 2018

Because currently Go packages are distinguished by import paths, support for relative import paths would mean that very early on in the build process the relative imports would have to be converted to full paths.

As far as I can see the only time it is useful to have a relative import path is when you are going to move a set of packages around. That seems to happen relatively rarely, and it would be straightforward to write a tool to move a set of packages while rewriting the import paths. I'm not aware of any such tool today, but if we had one would there still be a need for relative import paths?

@cch123

This comment has been minimized.

Copy link
Contributor Author

commented Mar 1, 2018

@ianlancetaylor , if I fork other people's project on github, and want to contribute back to this project.It seems there is still no way to keep the forked project go gettable and the import paths inside this forked project to be same with the original's.

If I want to use it as my own, I have to rewrite the import path, and if I want to contribute, I have to rewrite the import path again. So I think it's not quite friendly to the current github workflow.(but if you can provide a rewriting tool, this annoying procedure can be more convenient than now.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Mar 1, 2018

@cch123 Seems like you may be assuming that the other person's project is already using relative imports, which it most likely is not.

@cch123

This comment has been minimized.

Copy link
Contributor Author

commented Mar 2, 2018

@ianlancetaylor ,for example,I forked etcd project,this file https://github.com/coreos/etcd/blob/master/discovery/discovery.go contains imports of the same project's paths:

	"github.com/coreos/etcd/client"
	"github.com/coreos/etcd/pkg/transport"
	"github.com/coreos/etcd/pkg/types"

After forked to my own repo, the import path should be:

	"github.com/cch123/etcd/client"
	"github.com/cch123/etcd/pkg/transport"
	"github.com/cch123/etcd/pkg/types"

If I don't change the import paths and then execute

go get github.com/cch123/etcd

Then the project will appear at $GOPATH/src/github.com/cch123/etcd

But the import paths inside this project are not correct. I'm not assuming other people is using relative imports, but I mean if we could support some way(eg.relative import? or some way of internal usage) to unbind the import path with the host address, then I will not be forced to change the import paths back and forth.

The original issue is that I want to make use of internal feature to bind dependencies together with my library. Because dependencies on site like github cannot avoid risks to be deleted by original author(like the sad story of leftpad in npm community). So I think feature of this could make my library more safe in such situation. But if I make these dependencies as my library internals , it will also meet the import path problem. I have to change all the import paths of these dependencies if they use the import path like the above case.

@dlsniper

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2018

@cch123 I recommend you read http://blog.sgmansfield.com/2016/06/working-with-forks-in-go/ on how to work with forks if you want to contribute changes to a project. If you want to use your fork instead of the original place, dep supports this (and I think vgo may support this at some point, if not already).

Relative import paths is something I would love to see gone in Go2 as they are a constant source of pain.

@cch123 cch123 closed this Mar 2, 2018

@cch123 cch123 reopened this Mar 2, 2018

@nkovacs

This comment has been minimized.

Copy link

commented Mar 2, 2018

I agree with @cch123 forking packages with subpackages in them is a huge pain.
Changing origins is not a solution (you can't go get them). Dep's support for forks is a band-aid.

You can actually abuse the vendor system to solve this problem. Just put your subpackages into vendor, and you've got project-relative imports, you can fork the package easily without having to rewrite everything. Obviously that's an ugly hack, but I'd like to see something like that as a solution to this problem, maybe not full relative imports.

@dlsniper

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2018

I agree with @cch123 forking packages with subpackages in them is a huge pain.
Changing origins is not a solution (you can't go get them). Dep's support for forks is a band-aid.

@nkovacs this is not a band-aid, nor would relative import paths solve anything in this case. dep allows something that's considered basic functionality for package managers. There is nothing new or different for Go compared to other languages.

Just put your subpackages into vendor, and you've got project-relative imports, [...] Obviously that's an ugly hack

This is not why we have the vendor directory. We have it for reproducible builds. It allows guaranteed that, if committed, everyone gets the same identical source code regardless of environment.

but I'd like to see something like that as a solution to this problem, maybe not full relative imports.

I don't understand how relative imports solve any problem that you have.

@nkovacs

This comment has been minimized.

Copy link

commented Mar 2, 2018

If I fork a php package, I can publish it on packagist without first having to change a bunch of imports inside it.
If I fork a go package with subpackages, I have to change all the imports to its subpackages in it if I want people to go get it, or govendor fetch it, or dep ensure it, or whatever.

Both composer and npm allow you to specify a temporary fork instead of the canonical package. But in both ecosystems you can just publish your forked package without having to rewrite a bunch of imports inside it, because they're all relative. You can't do that in go.

This is not why we have the vendor directory. We have it for reproducible builds. It allows guaranteed that, if committed, everyone gets the same identical source code regardless of environment.

Of course not. I said you can abuse it to fix the problem.

I don't understand how relative imports solve any problem that you have.

I wouldn't have to do this: nkovacs/ifacemaker@c327e8d
Or this: nkovacs/counterfeiter@d0d3dfc

@dominikh

This comment has been minimized.

Copy link
Member

commented Mar 2, 2018

If I fork a go package with subpackages, I have to change all the imports to its subpackages in it if I want people to go get it, or govendor fetch it, or dep ensure it, or whatever.

Which, compared to the effort of forking, modifying and maintaining a project, takes absolutely zero effort.

@dlsniper

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2018

But in both ecosystems you can just publish your forked package without having to rewrite a bunch of imports inside it, because they're all relative. You can't do that in go.

To give you the counter-point for the PHP example. You would have to override the autoload process to tell it that your project provides now the reference for the namespace you want to use, rather than the original package, see https://github.com/Seldaek/monolog/blob/master/composer.json#L46-L48

Now Go being different than PHP is is allowed to behave in a different way. If you ask me, instructing dep to fetch the source from a different place is a lot more cleaner and obvious than digging in the package.json and figuring out where the autoloader will perform the operation for loading certain package.

And even in PHP or in Go world, allowing relative import paths suffer greatly when introducing symlinks into play (which at least for me in the PHP world have been a source of needlessly spent hours of debugging why something doesn't work in a particular setup of a developer / environment or in bug reports that I had to handle).

As far as I can tell, the whole argument is: please allow relative import paths so we don't have to run a search & replace operation for forks we want to use as main source but we don't want to use dep and we don't want to have a commit for that operation and we'll never backport changes to upstream (due to the import path change). Would this be an accurate depiction of the case?

@cch123

This comment has been minimized.

Copy link
Contributor Author

commented Mar 2, 2018

@dlsniper , if the official vgo can support publish and get a forked library without manually replace the imports in src code, I think it's ok.
But before vgo, you cannot control your user's package management tool, maybe some people choose godep or some other tool which doesn't support such compile time import search. They'll be in trouble.

@dlsniper

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2018

@dlsniper , if the official vgo can support publish and get a forked library without manually replace the imports, I think it's ok.

@cch123 that's what dep can do today (and I think glide as well).

But before vgo, you cannot control your user's package management tool, maybe some people choose godep or some other tool which doesn't support such compile time import search. They'll be in trouble.

dep, which is the recommended solution for package management until vgo matures, supports this. Much like gofmt or other tools, people can either use it or not, but if the majority of the community uses that, then the problem is on the users that choose to ignore it, not on the rest.

I hope this helps understanding where my point of view comes from and why I think it's important to address the underlying issue that causes the need for this change as well, not just agreeing to do the change.

@nkovacs

This comment has been minimized.

Copy link

commented Mar 3, 2018

Ok, let's take npm as an example.

You have a library called awesome-package, which contains a subdirectory utils. In your code, you can just use require("awesome-package") or require("awesome-package/utils").
If I fork the package and call it super-awesome-package, you have to change your code to require("super-awesome-package") or require("super-awesome-package/utils"). No confusion there. But I don't have to change the code in super-awesome-package, because it uses require("./utils").

The same thing could work with go. You would import github.com/user/awesome-package or github.com/user/awesome-package/utils, and if you want to use the fork,
github.com/anotheruser/super-awesome-package or github.com/anotheruser/super-awesome-package/utils.

If you ask me, instructing dep to fetch the source from a different place is a lot more cleaner and obvious than digging in the package.json and figuring out where the autoloader will perform the operation for loading certain package.

I don't think it's any better (or worse) to have to dig into dep's lockfile (or wherever it stores this info) to find out that it's not actually importing github.com/user/awesome-package, but github.com/anotheruser/super-awesome-package.
I would much prefer it if my code actually imported github.com/anotheruser/super-awesome-package.

And even in PHP or in Go world, allowing relative import paths suffer greatly when introducing symlinks into play

What do symlinks have to do with relative imports? You can use symlinks with absolute imports too.

As far as I can tell, the whole argument is: please allow relative import paths so we don't have to run a search & replace operation for forks we want to use as main source but we don't want to use dep and we don't want to have a commit for that operation and we'll never backport changes to upstream (due to the import path change). Would this be an accurate depiction of the case?

No, I think that's a bit of a loaded question. Maybe I don't want to use dep because it's slow or not mature yet or because it can't be used for installing tools. Also, instructing it to use a fork is an extra step and makes it not obvious that you're importing the fork (see above).
Maybe I do want to backport changes upstream, but the upstream project is inactive, or they don't agree with the direction I've taken and I have to fork.
Sure, I can run a search & replace, but it feels like I'm working around a deficiency in the language.

@myitcv

This comment has been minimized.

Copy link
Member

commented Apr 27, 2018

@ianlancetaylor - I just happened upon this issue to add my weight to the argument for removing relative imports in Go 2.

As this issue stands, I think the title is somewhat confusing because there doesn't appear to be the context referenced anywhere that the current plan is to remove them (unless I'm missing something).

Can I suggest we tweak the title to proposal: Go 2: cmd/go: continue to allow relative imports? Because:

  • it's currently possible (yet very much discouraged, with good reason) to use relative imports
  • the plan as things stand is to remove them for Go 2
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Apr 27, 2018

Relative imports barely work today. They don't work inside of GOPATH. They only really work when compiling individual files. I think this proposal is about allowing relative imports inside of GOPATH.

I'm not myself aware of any particular plan regarding relative imports in Go 2. Perhaps there has been some discussion of them in the vgo context, I don't know.

@myitcv

This comment has been minimized.

Copy link
Member

commented Apr 27, 2018

Relative imports barely work today. They don't work inside of GOPATH. They only really work when compiling individual files. I think this proposal is about allowing relative imports inside of GOPATH.

👍 - that (a single file via go run) happens to be exactly the scenario I "tested" to see to what extent things worked today. Indeed they don't work within GOPATH - thanks, I hadn't realised that distinction.

Title is fine as is therefore. Apologies for the noise.

@metakeule

This comment has been minimized.

Copy link

commented May 25, 2018

Since the functionality is already available via the vendor directory, it would be sufficient to allow the import of vendor/internal/...'. As of Go2 the vendor` directory could then be renamed to something more matching (maybe just 'r' for 'relative').

@nyetwurk

This comment was marked as off-topic.

Copy link

commented Sep 8, 2018

It is inexcusable that the golang devs didn't think about the impact of forcing absolute imports on the fork/branch/pull-request github workflow.

@dominikh

This comment was marked as off-topic.

Copy link
Member

commented Sep 9, 2018

It is inexcusable that the golang devs didn't think about the impact of forcing absolute imports on the fork/branch/pull-request github workflow.

Not nearly as inexcusable as people not understanding the concept of git remotes: http://blog.campoy.cat/2014/03/github-and-go-forking-pull-requests-and.html

@golang golang locked as too heated and limited conversation to collaborators Sep 9, 2018

@davecheney

This comment was marked as off-topic.

Copy link
Contributor

commented Sep 9, 2018

I’m locking this issue to limit conversation until the proposal committee respond.

@bcmills

This comment was marked as off-topic.

Copy link
Member

commented Jan 8, 2019

The proposal committee doesn't usually respond to Go 2 issues. I'm going to unlock the issue, but folks, please be mindful of our gopher values and keep it constructive!

@sebastianhaberey

This comment has been minimized.

Copy link

commented Aug 17, 2019

I, too, ran into the issues pointed out by @johnDoe2018 and @cch123 recently. I tried the proposed solution involving dep ensure, but I could not get it to work for internal packages. I tend to agree with @cch123 that this dep functionality is a bit of a band-aid. From the documentation:

source rules are generally brittle and should only be used when there is no other recourse.

I am convinced that there are good reasons to completely remove relative paths from Go and so I'll leave that discussion to people more familiar with Go. On the other hand I think that forking repositories, creating pull requests and validating them on CI systems are part of daily developer life. The current need to adress packages inside my own project by their full repository path seems to conflict with that workflow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.