Create v3 to support Go modules

[raphael]

Goal

We want to keep supporting goa v1 but also want to make goa v2 a clearly separate framework as the differences between the two are fairly fundamental. Today one can use v1 by using the package path github.com/goadesign/goa and v2 with goa.design/goa and we'd like to keep that going forward if only for backwards compatibility. The actual source code for v2 is in the v2 branch in the github.com/goadesign/goa repo. The https://goa.design/goa endpoint redirects go get to gopkg.in/goa/v2 which points to the v2 branch.

Problem

The introduction of Go modules breaks the scheme described above as the go tool now does a local checkout of the branch which is incompatible with the use of gopkg.in.

Proposed solution

The proposal consists of moving the v2 branch to its own repo, e.g. github.com/goadesign/goa2.

Other considered solutions

An alternative would be to create a v2 directory in the existing repo but that does not seem attractive because:

• It would be confusing for newcomers (v1 still leaves under the top level directory).
• It would obfuscate where most of the work will actually happen in the future.
• It would be a breaking changes for v2 as all the import paths would have to be modified to append v2

Notes

This proposal does not address Go modules support in v1.

[raphael]

I have put more thoughts into this and I'm now wondering whether a better approach wouldn't be to simply merge the v2 branch in to the master branch of this repo (github.com/goadesign/goa) and keep v2 up-to-date with master. We would create v2 releases by creating tags which would make it possible to use Go modules by specifying the appropriate version in the module definition file i.e.:

module github.com/you/hello

require goadesign/goa v2.0.0

[cubic3d]

I think merging v2 into master and tagging would be the best way to support modules, but the import would still be require goadesign/goa/v2 v2.0.0 according to https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher and needs to be set in go.mod with the v2.0.0 tag. Golang did a very confusing step towards modules =/

Waiting for the support to finally migrate to v2 and modules on my projects :)

[nitinmohan87]

@raphael We are at a point where we are not adding new features to goa v2. We have been fixing bugs and writing docs to release v2. But without having v2 branch merged to master, we really can't test go modules support. We should probably start doing that.

[k3a]

A separate repo or merge to master would be definitely better than a subdirectory.

I am not sure if merge to master would be the best, though. Goa2 is quite a major overhaul, maybe a separate repo would be clearer option as it would avoid setting tags to go.mod manually and it would let other tools like IDEs pick the correct dependency right away without any tag hacking...

[raphael]

@k3a can you expend a bit on what you mean by "tag hacking"?

[k3a]

Sometimes I create a new project and initiate go modules for it. Then I write a new import in Visual Studio code, save the file and it uses go tools to automatically fetch that new dependency and add it to go.mod. I don't know if it uses the last commit or last release tag. But it would be cool if it took Goa v2 by default, otherwise it would require manual go.mod edit and maybe VSCode would be confused, requiring it to close and reopen. Not a big deal, but it wouldn't be "automatic".

Separately, do you think it would be somehow possible to get "goagen" binary for the version used in a project in order to get reproducible builds? So far I've used "go get ..." on the global scope but I think it can get out of sync with the version used in the project eventually.

[raphael]

OK yeah I see your point. It seems though that creating a go.mod file for your project is going to become standard practice as Go standardizes on it. I would also expect that having to specify the version of your dependencies is going to be something that users get accustom to and that tools will help with going forward. In particular the import path will contain the version so I expect most tools to be able to pick that up automatically.

Today the way we make sure that the tool used to generate the code is always the same is by compiling the version under the vendor directory and using that to generate. We encapsulate all this within a Makefile.

[cubic3d]

@k3a as @raphael said, this is as intended in go modules. Having another repository for the same project vs. just adding /v2 to the actual projects repository does not make things different in go modules. In my opinion having another repository make things unclear in terms of what version is the latest (needs me to browse all repos first and find out) and upgrading requires changing of the include name.

If you import or add the /v2 repository for the first time in the project, you would get the latest release tagged. You can always change to any tag or commit by editing the go.mod or use any package manager / IDE that does that. VSCode work pretty fine with modules.

Goa v1 and v2 are not compatible designed, so "taking Goa v2 by default" have no advantage, because you have to decide which one you want to use in the first place.

[k3a]

I've looked at the problem again. Previously I wasn't aware about the most recent changes in Go module support and the original issue description didn't mention the current state and it all seemed too confusing.

I found this Go blog page the most helpful: https://blog.golang.org/using-go-modules
It includes clear examples of the workflow and it's easy to follow and my Go 1.12.4 fully supports it.

Surprisingly, they really prefer /v** sub-directory in a repo. I still find it strange because if you would specify goadesign/goa/v2, the cloned data would probably contain complete repo with v1 and v2 (and other v* subdirectories) kind of unnecessarily, right?

It seems to me that v* subdirectory semantic is just good to avoid naming confusion import "goadesign/goa/v2" makes it clear that it is just a "2" version of "goa". But I don't see anything bad about a separate repo import "goadesign/goa2".

Maybe I am missing something - please tell me if so - but I see another confusing thing about subdirectories and that is Github Issues. In every issue there now need to be a tag or prefix to mention a version number "generation" it refer to. Separate repos would be free from this confusion.

EDIT: There is an important gain if using subdirectories and that is they don't require explicit renaming when importing which is quite a big deal:

import "goadesign/goa/v2" // go now handles this!

func HelloMux() goa.ServeMux {
    return goa.NewMux()
}

With a separate repo, to avoid renaming there would have to be additional directory for top-level imports:

import "goadesign/goa2/goa" // looks a bit weird

func HelloMux() goa.ServeMux {
    return goa.NewMux()
}

or import renaming which would be annoying to use

import goa "goadesign/goa2" // ah... no!

func HelloMux() goa.ServeMux {
    return goa.NewMux()
}

So I think it is now pretty clear that merge to master to a subdirectory is surprisingly the best and officially preferred option...

[cubic3d]

@k3a

Surprisingly, they really prefer /v** sub-directory in a repo. I still find it weird because if you would specify goadesign/goa/v2, the cloned data would probably contain complete repo with v1 and v2 (and other v* subdirectories) kind of unnecessarily, is that correct?

Actually no, not correct. Go does not have a preference, but gives you a variety of possibilities. Module paths ending in /v* do not imply that the repository has a directory containing an actual version - this is just a module notation which can refer to a semantically tagged version. See above, where I linked a releasing guide for v2+. You really do not want to have a project like goa live in versioned directories inside a repository ;)

[k3a]

No, according to the example in the blog post it seems it indeed implies sub-directory. Please have a look here https://github.com/rsc/quote v3 is there. At this point there could be v4, v5, v6 as well. I don't mean that there is a new git repo in the subdirectory but that doing import "goadesign/goa/v2" would actually clone the full goadesign/goa git repo and then jump to some tag like 2.0.0 (latest stable of 2) and use just the code from the v2 sub-directory. But it seems to me a bit unnecessary to clone the full repo with all version sub-directories. Once there is Goa 13.0, it would clone complete history starting from v2 to later just use the code from the v2 subdirectory.

Hmm, but maybe the final tag which is used don't have to contain any other sub-directory than the one referred in the tag and it somehow clones the single tag only without complete git repo history. Or maybe it does and we shouldn't bother, we are not paying for Github bandwidth. Yes, this new module support is still a bit confusing to me... :( but imports look better than having goadesign/goa2 repo (see my previous post edits).

[cubic3d]

Sorry it's hard to write while you edit your comments every minute.
I see that it's using directories in the example, I just said that it does not imply that generally (and does not prefer, not does it).

[raphael]

Thanks for the discussion guys! As @cubic3d mentioned module versioning does not require subdirectories but merely that the module path has the version appended to it. I put together a POC for v3 (don't want to break v2). v3 is functionally exactly the same as v2 - the only difference is that it supports Go modules. Take a look at https://github.com/goadesign/goa/blob/v3/doc.go for an example of how to use it. We should test v3 with designs that are in GOPATH as well and if all works well I'm planning on making the v3 branch the default on GitHub which alleviates the issue of newcomers seeing the latest and greatest but also keeps the master branch intact for people that are currently using it.

[cubic3d]

@raphael the well known 'we make a v3' method since modules introduction, the echo framework guys from labstack did the same with v4 ;) I'll test as soon as I can and port an API project to goa 2.

[cubic3d]

Ok, in terms of using v3 as modules everything seems to be fine - no problems there.

The porting part however is a bit of pain at the moment regarding auth. We kinda 'misuse' BasicAuth in v1 by supplying scopes to it. This allows us to extract the scopes via goa.ContextRequiredScopes(ctx) in the middleware, authenticate the user against LDAP and match the scopes against LDAP groups. It even generates correct swagger, which contains the required scope, e.g.: BasicAuth (users:list). As far as I can see, v3 does not allow scopes to be defined on Security(BasicAuth) and also requires to have username and password payload defined on every method, instead of handling everything in middleware only.

That's a bit off-topic, is there a better place to discuss such things?

[raphael]

Thanks for the feedback @cubic3d - I suggest you open a separate issue to discuss your need.

[raphael]

v2 and v3 have been released, closing this issue - thank you for all the input!