proposal: cmd/go: support local experiments with interdependent modules; then retire GOPATH #44347
Comments
|
I like this proposal more than other alternatives to the problem, hope it get more attention. I used to have And I hesitate to pickup further development of my rusted Go projects at this time, just because go.mod is the right way to go, but it still lacks sufficient support rival to my historical workflows with |
|
I don't understand the significant difference between this proposal and just continuing to use |
Yes it is. Yes — almost: the main and important difference is that it operates in module "units". There are modules under the Same for bootstraping and early experimenting with code structure. When I am done with experiments, simple
The most significant deviation is described above. "Production fuse" is the second, along with builddirs moved inside GOTINKER. Both prevent experimental code leaks to the CI/CD pipelines. It shouldn't happen but it happens — and when it does, it hurts a lot.
It looks at, and if it sees wrong environment it gently refuses to do any work except saying why it won't run "here".
To the developer, probably not much - she already has it set. But deployment people will be less nervous when asked to fire up a field test.
GOPATH destiny, according to previous discussions, is to retire. GOTINKER needs the least code I could think of to get GOPATH benefits back. It boils down to:
...(skip code in vendor)... |
Apparently I'm still not getting it. This doesn't sound very different to me. For example, I currently have So, I really don't see the difference, TBQH. To me, that seems to be exactly the setup and workflow you are describing.
Yes. That's why I'm confused. To me, what you are describing is almost exactly the (for now) still existing |
Minus that with 1.17 the
It is supossed to be almost identical workflow - except for all tools being now in modules mode, so I can work only with, say, three repositories out of thirty or eighty. Modules that I do not plan to tinker with will come from cache (copied or hardlinked to the GOTINKER code will need do a little bit of magic to internal representation of
Yes "we" think we need. See Rationale at top. |
FTR, by "we" I mean "the Go project", not a specific subset of people. It certainly includes you and me and many others, many of whom likely agree with you and many of whom don't. It's possible for "the Go project" to come to a conclusion, even though some or many of its members disagree with it. The proposal process is how these conclusions are reached and are thus what decides what the "we" I was referring to "thinks". So, let me rephrase my questions. If the outcome of this discussion is, that the workflow presented here is important enough to implement and given its extreme similarity (by design) to the existing GOPATH based workflow, wouldn't it be preferable to simply not retire GOPATH? And if, on the other hand, there are good, convincing reasons to retire GOPATH, why would those reasons not apply to this proposal? Surely, whether the environment variable is spelled GOPATH or GOTINKER does not affect those reasons? Alternatively, this proposal is not as similar to GOPATH as it seems to me, which means I misunderstood something. That would certainly make these questions obsolete and provide justification for this proposal. Which is why I tried to understand these differences. So, is it actually the case that the workflow you describe is currently (we are not talking about go 1.17+) captured by setting |
|
@Merovius As I understand it, one very important difference is that while GOPATH is an exclusive choice against go.mod, GOTINKER can work together with go.mod. I for myself especially want the benefits of GOPROXY with go.mod on (to workaround GFW as one reason), I can't get those by choosing GOPATH, while GOTINKER is hopeful to provide equally ideal workflows like with GOPATH, and still have all modern & good things from go.mod. |
What does that mean? Concretely? Like, presumably the intent is to actually use local modifications, ignoring versions specified in FWIW, as far as I can tell so far, the only reason to call it
I don't understand why this would require a new environment variable. To me, this seems to simply mean downloading the zip and unpacking it in the right directory (same as |
I meant specific subset, hence quotation marks.
Probably keeping GOPATH mode would be a least friction solution to the interdependent edits problem in the short term. But, after a period of being in denial, I now second Rob Pike's "GOPATH must go" statement — because the technical debt from keeping GOPATH workflow intact will IMO hurt the ecosystem in the long run. Modules with their clearly, automatically accounted and double-checked versioning mechanics are superior to GOPATH for any software that evolves fast, for long, or both. That said, modules workflow has a blindspot where interdependent edits of two or more modules come with a huge footgun loaded: you need to be very careful editing your replace directives by hand, in many places, then you need to be very careful again when you need to clean your laboratory table after. Citing @bcmills again:
(Just now /04.2021/ I see three discussions about "how to do it with modules" active on the golang-nuts list. Such questions emerge there almost every month since go1.11)
GOPATH is all-or-nothing. When it is on, all used modules repos must go there, and proper (re being investigated code) versions must be checked out by hand. Then vendoring status of each must be consulted too. GOPATH mode is currently, ie. in modules filled world, good only for bootstraping new code, it is unusable for the bughunt purposes.
This proposal keeps only good things from GOPATH mode — all sandboxed and all implemented by already existing mechanics of the modules.
No, not the same files. And almost certainly not the same as being investigated production code uses.
In GOPATH mode tools operate on repo's HEAD. It is up to you to do proper checkouts of everything your, possibly big, app or service uses. In proposed GOTINKER mode tools operate on modules as usual, so anything else but repos you pulled under GOTINKER path is kept at version specified in respective go.mod. All bookkeeping is done for you, nothing will drift apart or leak accidentally. |
Not entirely, I mean, I only want a few modules I put in local filesystem, to override what go tools would find according to go.mod, for vast of the rest dependencies, I would like them managed by the modern
I'm not sure I understand you correctly, but the whole idea is to avoid things local-tinkering-only to go into
I would like all the way |
"Project specific settings" already are addressed in the proposal by the means of local GOENV file inside the GOTINKER path. There is no need to add an another "special" file as a knob. Such file (like the go.mod changes) would be another box to check on the setup/cleanup lists. You can use direnv to set GOTINKER for you. (Even now you can use direnv to set GOPRIVATE/GOPROXY in the WiP tree. Works well if you do not use much external modules). Intended bootstrap workflow using GOTINKER is just to set it to the "project" dir. You then work inside src/import/paths below with all the bells and whistles of module mode. Ie. all external dependencies are resolved as usual, just your code import paths are replaced relative to each other internally by the build tools. |
To make myself clearer, I consider it would be another proposal without the Plain filesystem structure is simpler than that plus env vars, and will be much easier to up-scale without env vars involved. Somewhat like NodeJS uses I mean I think that might be even better than GOTINKER proposed here, but I don't have the necessary funding (of time & energy) to pursuit that idea. So I support your proposal here as far as the situation goes at time of speaking. |
|
I suggest Besides |
It was considered, then refuted. Rationale: GOTINKER is not meant to keep yet another way to build final version of a Go software as explicit tag-based versioning supported by the modules ecosystem is clearly superior to the multi-tree one. (With GOPATH, the only way to keep semi-stable versioning of being used code was to keep multiple code trees — and staging versions there or keeping tinkering under |
|
While I think that #45713 is much overengineered I accept that the work on its implementation already begun, so this much lighter proposal can't stand. Closing. |
Go "tinker mode" proposal.
Last updated: 2021/04/11
Rationale:
Related: #26640, #37755, #26377, #25053.
If an environment variable named GOTINKER is defined and set to an absolute path to the existing location on the local filesystem, and the import path of an include can be found under
$GOTINKER/directory, then build commands treat the$GOTINKER/src/import/pathas a final authoritative source of the import; foregoing bothvendor/and any go.mod directive perpeting to thisimport/path.Ie. build commands like 'go build' and 'go test' will compile modules present in the $GOTINKER directory instead of accessing the network, local module cache, or vendor directory.
Under tinker mode GOBIN, GOCACHE, GOMODCACHE, and GOENV are bound to locations relative to the GOTINKER: $GOTINKER/bin, $GOTINKER/cache, $GOTINKER/pkg/mod, and $GOTINKER/goenv - respectively.
$GOTINKER tree should be populated by the user. For yet some time to come the last version of Go to support GOPATH can be used to ease this task, ie.
GOROOT=/where/go1.16 GO111MODULE=off GOPATH="$GOTINKER" go1.16 getGOTINKER path last element may start with an underscore character so experiments can be kept inside any project tree.
Security considerations
Both object code and the executable built under the tinker mode should not accidentally leak to the production environment. Ie. while objects are built and cached under $GOTINKER, then built executable MUST be amended (by the compiler) to refuse to run in an environment where GOTINKER is not set, or it is set but does not match the last element of the GOTINKER path that was compiled in.
Ie. "tinkered with" executable preserves the last part of the GOTINKER path then matches it to the last part of GOTINKER string where it runs. If these do not match, exectutable exits immediately with "Experimental but GOTINKER is not set or did not match" error message.
How GOTINKER workflow is different to GOPATH''s one?
GOPATH is all-or-nothing regarding versions. Ie. under GOPATH tools operate on code as-is. If we are about debugging or changing interdependent code, it is up to us to do proper checkouts of everything our — possibly big — app or service uses.
In proposed GOTINKER mode tools operate on modules as usual, so anything else but code we pulled under GOTINKER path is kept at version specified in respective go.mod. All bookkeeping is done for us, nothing will drift apart or leak accidentally.
edits:
_tinker/subdirectory.The text was updated successfully, but these errors were encountered: