engine: merge into the cli repository

[bassosimone]

We are going to use Go 1.16's new feature that enables embedding assets. To this end, it would greatly simplify our life to have a single Go repository containing all the sources. This issue is about merging the cli and the engine repositories.

We could merge the cli into the engine, or the engine into the cli. Since the cli is tagging publicly available version numbers we will merge the engine into the cli.

Because of Go strict semantic versioning rules, we cannot export anything from the engine into the new repository, otherwise, we will need to break the import path at basically every new release of ooniprobe.

Therefore, we will basically vendor the engine codebase inside internal/engine. We still to hammer out a bunch of details including what to do of the existing engine repository, etc.

[bassosimone]

The first problem I identified is the way in which probe-cli uses semantic versioning. Consider this:

$ pwd
/tmp
$ go version
go version go1.15.6 linux/amd64
$ go get -v github.com/ooni/probe-cli/cmd/ooniprobe
go: found github.com/ooni/probe-cli/cmd/ooniprobe in github.com/ooni/probe-cli v0.3.0-rc.2
[...]
# github.com/ooni/probe-engine/measurementkit
g++: error: /usr/lib/libmaxminddb.a: No such file or directory
[...]
g++: error: /lib/libz.a: No such file or directory
[...]
github.com/ooni/probe-cli/internal/onboard
$ echo $?
2

This was happening because the import path github.com/ooni/probe-cli implied v0 or v1 as documented by Golang's wiki. See how the code above was building v0.3.0-rc.2.

The first step to fix this issue has been to tag v0.4.0. This is what happens now:

$ go version
go version go1.15.6 linux/amd64
$ go build -v github.com/ooni/probe-cli/cmd/ooniprobe
go: finding module for package github.com/ooni/probe-cli/cmd/ooniprobe
go: found github.com/ooni/probe-cli/cmd/ooniprobe in github.com/ooni/probe-cli v0.4.0
$ ./ooniprobe 
Please, use github.com/ooni/probe-cli/v3

This is quite an improvement. The binary we build fails and tells us what import path we should be using. (I can further improve that by listing the correct command to build and the full import path, but you get the idea.)

A second improvement is to make sure we can use a recent version of Go to build the correct binary. I implemented this change in ooni/probe-cli#200. This is what we can do now:

$ gotip version
go version devel +32e789f Mon Feb 1 21:38:01 2021 +0000 linux/amd64
$ gotip install -v github.com/ooni/probe-cli/v3/cmd/ooniprobe@master
$ `gotip env GOPATH`/bin/ooniprobe version
3.5.0-alpha

Here I needed to force @master because we don't have a tagged version of probe-cli that includes ooni/probe-cli#200.

BTW, of the two options available for releasing v2-or-higher Go modules, I chose the easier one. Just change go.mod and all the import paths. The more radical one, which calls for creating a subdirectory seems to create too much management complexity for me at this stage. We may reconsider for the future. But, for now, I don't see a need to support several major versions concurrently, so we should be good.

[bassosimone]

Now, the second step (ooni/probe-cli#201) is that of putting the probe-engine codebase into the probe-cli codebase. At this stage, we only care about being able to produce a working ooniprobe binary. Once this piece of work is done, there is extra work to ensure we can do with the probe-cli's repository everything we can do with probe-engine.

[bassosimone]

Now, my task is to figure out which tests are failing and why. Exciting times 😬.

This work has now been done here: ooni/probe-cli#202. (For the records, the problem here is that some .gitignore files were too strict, so I didn't add everything I wanted to when I ran git add internal/engine.)

We're basically down to the same amount of failures we see in probe-engine.

[bassosimone]

In ooni/probe-cli#203 we're adding support for building miniooni from probe-cli.

[bassosimone]

In ooni/probe-cli#204 I'm migrating probe-engine workflows to probe-cli.

[bassosimone]

The next step is automatically publishing an Android package. We're working on this in ooni/probe-cli#205.

[bassosimone]

More refactoring underway in ooni/probe-cli#206. It's a set of minor changes and movements after we merged the two codebases. This is ~easy and quite useful to help me identify optional complexity to kill.

[bassosimone]

The merge is complete. We can now continue working on the engine inside of probe-cli. We have clearly deprecated the probe-engine repository. We will keep using it as a container for past issues. We will route future contributions to probe-cli/internal/engine.