Add an alternate import syntax with better support of Go modules

[debovema]

Abstract

This PR improves the way Flogo CLI manage imports of contributions:

• imports can be written with the classic syntax or a new alternate syntax (see details in Support classic and alternate import syntax in flogo.json core#44) in flogo.json file and in flogo install parameters
• all imports paths currently added in imports.go during flogo install will also be added in flogo.json file in imports array.
• Go dependencies are managed with go mod plumbering when it's possible, with go get when it's not
• contributions with a version suffix (@vX.Y.Z, @latest, @master, @feature-branch) can be used with both classic syntax and alternate syntax

Requirements

• For proper serialization of the JSON file (flogo.json), this PR requires Generate JSON schema of Flogo app descriptor core#20 to be merged.
• Application needs Support classic and alternate import syntax in flogo.json core#44 at runtime so this one is also required to be merged.

Tests

Step-by-step guide

To test this fork with fork from project-flogo/core#20 & project-flogo/core#44:

0. Optionally run following commands in a Go Docker container (successfully tested with Go 1.12.0 as well)

docker run --rm -it golang:1.11.5 bash

1. Install CLI from this fork with Generate JSON schema of Flogo app descriptor core#20

git clone https://github.com/square-it/cli.git /tmp/cli
cd /tmp/cli
git checkout install-contribs-with-version
echo "replace github.com/project-flogo/core => github.com/square-it/core generate-json-schema" >> go.mod
go install ./...

2. create an application (choose a. or b.)

a. use updated flogo.json with alternate import syntax from PR project-flogo/core#44 to create a new app (classic syntax is obviously still supported, the only difference is that alternate syntax will use go mod instead of go get plumbering)

cd /tmp
wget https://raw.githubusercontent.com/square-it/core/new-import-syntax/examples/alt/engine/flogo.json
flogo create app -f flogo.json

b. otherwise create directly a standard app with classic syntax

cd /tmp
flogo create app

3. build the application

a. if project was created using new syntax, override go.mod to use project-flogo/core#44

cd /tmp/app
echo "replace github.com/project-flogo/core => github.com/square-it/core new-import-syntax" >> ./src/go.mod
flogo build -e

b. otherwise

cd /tmp/app
flogo build -e

4. At last, to add a contribution with a version, use for instance:

cd /tmp/app
flogo install github.com/square-it/flogo-opentracing-listener@v0.0.2

One-liner test command

The test above can be run with the CLI from master then the CLI from this fork using this command:

docker run --rm -it golang:1.11.5 /bin/bash -c "$(wget -q https://gist.githubusercontent.com/debovema/78095d87578c092372037b1dbf05454d/raw/9dba3959add6911065b7e0baf2f20bc5a2b7adfe/test_run.sh -O -)"

Details of execution are in this gist.

[debovema]

go get can only fetch HEAD of a repository (master).
We need another solution (fully-)based on Go modules, except for legacy contribs.

[debovema]

I added a support for "legacy" imports which does not require project-flogo/core#44 new syntax
(but this new syntax still has advantages and should be used IMHO).

Legacy flavour

flogo.json

flogo.json imports with old syntax:

  "imports": [
    "github.com/project-flogo/contrib/activity/log",
    "github.com/project-flogo/contrib/trigger/rest",
    "github.com/project-flogo/flow"
  ],

Steps

flogo install will:

1. try go mod edit -require github.com/project-flogo/contrib/activity/log@master (see https://github.com/square-it/cli/blob/5b3cbe30cf14b316c384ddc85eff843bb15ef854/util/mod.go#L48)
2. but will (gracefully) fail, since there is no go.mod file @ https://github.com/project-flogo/contrib/tree/master/activity/log (see https://github.com/square-it/cli/blob/5b3cbe30cf14b316c384ddc85eff843bb15ef854/util/mod.go#L68)
3. and try go get -u github.com/project-flogo/contrib/activity/log instead (see https://github.com/square-it/cli/blob/5b3cbe30cf14b316c384ddc85eff843bb15ef854/util/mod.go#L74)
4. add github.com/project-flogo/contrib/activity/log in imports.go

Resulting go.mod file

module main

require (
        github.com/project-flogo/contrib v0.9.0-alpha.3 // indirect
        github.com/project-flogo/core v0.9.0-alpha.4.0.20190222151024-3eb86689b764
        github.com/project-flogo/flow v0.9.0-alpha.3.0.20190211150821-b5f5b5d71381
        github.com/stretchr/objx v0.1.1 // indirect
        github.com/stretchr/testify v1.3.0 // indirect
)

Some indirect dependencies exist (including github.com/project-flogo/contrib which should be direct).

Module flavour (preferred)

flogo.json

flogo.json imports with new syntax (project-flogo/core#44):

  "imports": [
    "github.com/project-flogo/contrib:/activity/log",
    "github.com/project-flogo/contrib:/trigger/rest",
    "github.com/project-flogo/flow"
  ],

Steps

flogo install will:

1. try go mod edit -require github.com/project-flogo/contrib@master (see https://github.com/square-it/cli/blob/5b3cbe30cf14b316c384ddc85eff843bb15ef854/util/mod.go#L48)
2. and will succeed :)
3. add github.com/project-flogo/contrib/activity/log in imports.go

Resulting go.mod file

module main

require (
        github.com/project-flogo/contrib v0.9.0-alpha.3.0.20190211153431-680ebf186e58
        github.com/project-flogo/core v0.9.0-alpha.4.0.20190222151024-3eb86689b764
        github.com/project-flogo/flow v0.9.0-alpha.3.0.20190211150821-b5f5b5d71381
)

Notice it is less verbose than the legacy one and each dependency is explicitly and directly imported.
The versions are guessed (from HEAD / master) but could also be fixed by adding @vX.Y.Z in the new syntax Flogo imports (for instance github.com/project-flogo/contrib@v0.9.0-alpha.3:/activity/log).

[fm-tibco]

So is the issue that

flogo install github.com/square-it/flogo-opentracing-listener@v0.0.2

uses go get under the covers? Is the only advantage mod require doesn't add the indirect imports? I think if you do go mod tidy it adds them back though. Also from reading this go issue golang/go#27643 , go get will continue to add imports to go.mod. (I'm not a fan of that, but I don't always understand where these go guys are coming from... a hobbyist point of view?)

Also when you create a project, it should create a go.mod by default, so not sure why anything would fail due a go.mod file not being there (since it should be there).

@skothari-tibco What are your thoughts on this?

I do like adding to imports on install, but we might have to add a "tidy" to remove imports to unused triggers and activities.

[fm-tibco]

I do like the code cleanup in general for handling the imports. I'm cool with adopting this if it works as well as before. @skothari-tibco thoughts? @mellistibco thoughts on adding all import to the flogo.json? I kind of like this, makes it clear as to what you are using, especially if you aren't a go coder. Maybe we can add a "tidy" like command that can prune any triggers or activities that aren't directly referenced.

One question I have is about the new contrib import format: Why this github.com/project-flogo/contrib@v0.9.0-alpha.4:/activity/log as opposed to this github.com/project-flogo/contrib/activity/logv0.9.0-alpha.4 ?

[skothari-tibco]

Hey @debovema , I tried the new cli with the steps you specified. It was super simple to follow the steps, thanks. So I tied flogo install github.com/skothari-tibco/flogo-components/activity/twillio.
It not only adds //indirect to the go.mod but also makes the imports section of flogo.json a little bit inconsistent. Like

"imports": [
    "github.com/project-flogo/contrib:/activity/log",
    "github.com/project-flogo/contrib:/trigger/rest",
    "github.com/project-flogo/flow",
    "github.com/skothari-tibco/flogo-components/activity/twillio",
    "github.com/square-it/flogo-opentracing-listener@v0.0.2"
  ],

Am I missing something? Is there a different way to install the contrib? Or is this the expected behavior?

[debovema]

Thank for your replies @fm-tibco.

I will try to answer to all your questions and explain everything as it has moved a lot since first commit on this PR.

First the issue is not only that it uses go get under the covers (even though I find it bad).
Using go get instead of go mod has following issues:

• some contributions which should be direct dependencies are written as indirect. Typical example is github.com/project-flogo/contrib. I agree that go mod tidy should add indirect but I am not sure it can change indirect dependencies to direct ones right ?
• some contributions fail to be installed with go get whereas they succeed with go mod like github.com/square-it/flogo-opentracing-listener@v0.0.2.
  I created a gist with a one-liner test:

docker run --rm -it golang:1.11.5 /bin/bash -c "$(wget -q https://gist.githubusercontent.com/debovema/78095d87578c092372037b1dbf05454d/raw/9dba3959add6911065b7e0baf2f20bc5a2b7adfe/test_run.sh -O -)"

Then, I agree that a project should always have a go.mod file. You might have misunderstood something in previous comments. I now agree that go get behaves differently in a module-aware directory. I was not really aware of that before, go modules are still brand new for everyone I think ;)

About your concerns around compatibility, I tried to evolve the new model to support both approach. There are two changes to take care of though, both living in project-flogo/core#44 for now:

1. first one is in github.com/project-flogo/core/app/app.go and is mandatory. Considering flogo.json imports array can have different formats we need to be able to parse any of these and render it back to a Go import path with optional alias such as: alias github.com/company/contrib/activity/test without : or @vX.Y.Z.
   This is used at engine startup to register the different contributions (activities, triggers and others).
   The regex used is the same as the one in this PR and is pretty failsafe. I think it's better to have a one line duplication code rather than creating new dependencies between modules.

2. the second one is optional and is in github.com/project-flogo/core/examples/engine/flogo.json. It adds the : separator for github.com/project-flogo/contrib:/activity/log for instance. It can be reverted to github.com/project-flogo/contrib/activity/log if you feel it's better for now.
   One other advantages (other than using go mod plumbering) in separating module from the whole import is that by improving tooling we can detect that two contributions belongs to the same module and act accordingly. For instance, skip the go mod download/verify of already handled modules, display contributions sorted by modules (and version as well), etc...
   I hope it answers your question about the difference between two formats.

The step-by-step guide can help you validate quickly the good behaviour of this PR in a Docker container. It is working for flogo create with both formats and it is also working to install a contribution requiring go mod (see gist above for detail).

[debovema]

@skothari-tibco : you're not missing anything ! :) This is the expected behaviour.

If you want to install your contribution with the new syntax, simply use flogo install github.com/skothari-tibco/flogo-components:/activity/twillio.

The runtime support of all kinds of imports in the imports array in flogo.json is brought by project-flogo/core#44 (see comment above).

However I detected a little bug if we try to install flogo install github.com/skothari-tibco/flogo-components:/activity/twillio then flogo install github.com/skothari-tibco/flogo-components/activity/twillio will result in a duplicate in the flogo.json.
I thought I handled this before but lost it after a rebase with master branch of project-flogo/core. I will fix this asap.

UPDATE: in fact, this cannot be avoided since there is no way to distinguish between github.com/skothari-tibco/flogo-components/activity/twillio and github.com/skothari-tibco/flogo-components:/activity/twillio since that would imply to know where the module lives when using github.com/skothari-tibco/flogo-components/activity/twillio and that's what I wanted to address in this PR.
Good news: having several times the same import is not an issue as the registering of contributions in core can handled this use case.
In conclusion, the contributions maintainers should tell the end users how they want their contributions to be installed.

[fm-tibco]

I think the reason the code is currently adding an indirect import is because it does the go get before creating the actual dependency in the import file. If go mod tidy is run it will convert any indirect imports that are used into direct imports. This comment is just meant to explain the current behavior and how it could be fixed by running tidy at the end.

before tidy:

module main

require (
	github.com/davecgh/go-spew v1.1.1 // indirect
	github.com/project-flogo/contrib v0.9.0-alpha.3 // indirect
	github.com/project-flogo/core v0.9.0-alpha.4
	github.com/stretchr/objx v0.1.1 // indirect
	github.com/stretchr/testify v1.3.0 // indirect
	go.uber.org/atomic v1.3.2 // indirect
	go.uber.org/multierr v1.1.0 // indirect
)

after tidy

module main

require (
	github.com/davecgh/go-spew v1.1.1 // indirect
	github.com/pkg/errors v0.8.1 // indirect
	github.com/project-flogo/contrib v0.9.0-alpha.3
	github.com/project-flogo/core v0.9.0-alpha.4
	github.com/stretchr/objx v0.1.1 // indirect
	github.com/stretchr/testify v1.3.0 // indirect
	go.uber.org/atomic v1.3.2 // indirect
	go.uber.org/multierr v1.1.0 // indirect
)

[debovema]

@skothari-tibco : updated previous comment

@fm-tibco : you are certainly right and I think the right place to do this go mod tidy is at https://github.com/square-it/cli/blob/78b3606b7f15f311425e18b0a819f952505d462d/api/project.go#L139
A commented function call is already there ;)

However that does not change the failing behaviour when using go get with complex contributions.

[fm-tibco]

As for go mod vs go get plumbing, I'm not 100% convinced one is better than the other at the moment. It would be nice if the Go guys would un-blur the lines between them. I agree with Nate Finch in that go development discussion I referred to above. Regardless I'm fine with using 'go mod' if things will be more clean and consistent.

As for the other advantages, I think we could sort contributions by name and version regardless of the ':' syntax.

I do agree that the ':' does have an advantage in avoiding a network call or any extra disk calls. The best we could do to optimize the current behavior is read-in the existing go.mod and see if a parent module exists, and then avoid the go mod or go get.

I'm fine if we support that syntax for the advanced user, but for lot of users they are probably just going just add the import for the specific thing they want and not try to determine where the mod file is located relative to the import they are doing.

So just to be clear the imports:

"github.com/project-flogo/contrib:/activity/log"
"github.com/project-flogo/contrib/activity/log"

are the same, they both get the latest tagged version

"github.com/project-flogo/contrib:@master/activity/log"
"github.com/project-flogo/contrib/activity/log@master"

are the same, they both get the master

"github.com/project-flogo/contrib:@1.0.0/activity/log"
"github.com/project-flogo/contrib/activity/log@1.0.0"

are the same, they both get the 1.0.0 version. Btw, I think we should support with and without the v, in my opinion the v is kind of redundant... but I know its in the tag and some people use it that way.

[debovema]

Yes, your three examples are correct except the version should be before the :

However I disagree with dropping the v in the versions as I think it is part of the Go modules "specification".

So fixed examples would look like:

"github.com/project-flogo/contrib:/activity/log"
"github.com/project-flogo/contrib/activity/log"

"github.com/project-flogo/contrib@master:/activity/log"
"github.com/project-flogo/contrib/activity/log@master"

"github.com/project-flogo/contrib@v1.0.0:/activity/log"
"github.com/project-flogo/contrib/activity/log@v1.0.0"

[fm-tibco]

I'm not saying drop support for it, but I do think we should allow
flogo install github.com/project-flogo/contrib/activity/log@1.0.0

Which is what other apps support, like npm for example... I normally don't think I need to add a 'v' when installing version 1.0.0 of something.. you just do npm install lodash@1.0.0 for example.

[debovema]

OK. I think we should delay this feature of having v as an option in a next PR, don't you think ? 😆

[skothari-tibco]

@debovema I do understand now. This In conclusion, the contributions maintainers should tell the end users how they want their contributions to be installed. helps. Now we have to decide if we want to proceed with the new syntax. Having different syntax in imports of json also needs to be discussed. I am with @fm-tibco , I'm still not sure if go mod edit is better. I see the issues many people have with go get and modules. But as someone who started out with module and not dep, go get makes perfect sense, it gets the contrib you want add, store it in$GOPATH/pkg/mod and add that to the go.mod .

[debovema]

As I said on Gitter, I think we should use the go mod approach to leverage all abilities provided by the Go modules system. The flogo-opentracing-listener contribution is an example where this approach is required.
The Go language is a key argument for our clients when it comes to choose Flogo and we want to be able to use the latest Go constructs.

[fm-tibco]

We need to confirm the following with the new approach:

flogo install github.com/project-flogo/contrib/activity/log  --> installs latest tagged release
flogo install github.com/project-flogo/contrib/activity/log@latest  --> installs latest tagged release
flogo install github.com/project-flogo/contrib/activity/log@master  --> installs latest code from master

go get supports getting latest, master and specific version and updates the go.mod. I haven't been able to get go mod edit to do the 'latest' version properly, I'm probably missing something. Hopefully we can sort that out

As for the open tracing listener, it works with go get in my environment, not sure why it doesn't work for you. According to the go documentation they expect users to use go get and mainly tools that understand the module graph should use go mod edit.

[debovema]

@fm-tibco : I will check this and update this PR accordingly

[debovema]

@fm-tibco : it was not such a big modification since the @latest is working the same way in go mod and in go get.

See especially this commit 4f64254 added today.

I did many tests mixing different syntaxes, different versions (with explicit versions, branches, latest...) and everything seems to be very fine.

This PR description was also reworked to speak of classic syntax and alternate syntax and emphasize the fact that only users who want to opt in will be concerned by these changes. If you still have questions, don't hesitate to ask me here or in Gitter.

If it's OK, merge order would be:

1. Generate JSON schema of Flogo app descriptor core#20
2. Support classic and alternate import syntax in flogo.json core#44
3. Add an alternate import syntax with better support of Go modules #32

Thanks again for your time everyone.

[skothari-tibco]

@debovema I tried flogo create -c v0.9.0-alpha.3 myApp . It updated the repo to v0.9.0-alpha.5, is it expected to do that?

[debovema]

Considering the module also depends on github.com/project-flogo/flow@v0.9.0-alpha4 and github.com/project-flogo/contrib@v0.9.0-alpha4 and that these modules both depend on github.com/project-flogo/core@v0.9.0-alpha4, the odd thing is that it's using v0.9.0-alpha.5 not v0.9.0-alpha.4
However on a semantic versioning point of view, they are both a v0.9.0 so it might use the latest available.
I will have a look on this.

[debovema]

I think the fact that it gets the latest available v0.9.0 is caused by the go get triggered by the contrib imports (log and rest trigger).

I did the following test flogo create -c v0.9.0-alpha.3 -f https://raw.githubusercontent.com/square-it/core/new-import-syntax/examples/alt/engine/flogo.json app which enforces the go mod only plumbering and the result is correct:

module main

require (                                                                                  
        github.com/project-flogo/contrib v0.9.0-alpha.4
        github.com/project-flogo/core v0.9.0-alpha.4
        github.com/project-flogo/flow v0.9.0-alpha.4
)

Correct because v0.9.0-alpha.3 would not satisfy the dependencies of flow and contrib modules in their v0.9.0-alpha.4 versions.

[skothari-tibco]

Wouldn't this create confusion? Also we may need flogo create -c v0.9.0-alpha.3 appName to have core v0.9.0-alpha.4 if we consider dependencies of flow /contrib, and not -alpha.5. Is there a way to achieve this?

[debovema]

@skothari-tibco: I created issue #35 to describe this and #36 to fix it.