Migrate to go modules: incompatible (>=v2) packages

[lanzafame]

Tracking issue for the migration of ipfs deps that are incompatible with go modules, i.e. they have a git tag that is semver and >=v2. This spans packages in both ipfs and libp2p.

Packages:

• github.com/ipfs/go-ipfs-cmds
• github.com/ipfs/go-datastore
• github.com/libp2p/go-libp2p
• github.com/libp2p/go-libp2p-circuit
• github.com/libp2p/go-libp2p-crypto
• github.com/libp2p/go-libp2p-host
• github.com/libp2p/go-libp2p-kad-dht
• github.com/libp2p/go-libp2p-kbucket
• github.com/libp2p/go-libp2p-metrics
• github.com/libp2p/go-libp2p-net
• github.com/libp2p/go-libp2p-peer
• github.com/libp2p/go-libp2p-peerstore
• github.com/libp2p/go-libp2p-pnet
• github.com/libp2p/go-libp2p-record
• github.com/libp2p/go-libp2p-routing
• github.com/libp2p/go-libp2p-secio
• github.com/libp2p/go-libp2p-swarm
• github.com/libp2p/go-stream-muxer

Along the way, I will attempt to migrate any incompatible packages that are dependencies of the above listed packages as well.

[Stebalien]

/cc

@ipfs/go-team, @libp2p/go-team

[anacrolix]

Can you define incompatible? It makes it sound like they can't work at all, when I think instead there may be a Go-mod specific meaning.

[lanzafame]

@anacrolix

i.e. they have a git tag that is semver and >=v2.

is the definition of incompatible.

[Kubuxu]

I think the consensus was to remove those tags and revert to v0 or v1.

[lanzafame]

@Kubuxu that is how we are going to progress forward at this point. I had wanted to maintain the existing git tags but it is more work than it is worth especially considering the inconsistency of gx releases with matching git tags.

[raulk]

@Kubuxu the latest idea was to prefix all existing tags with gx- and adapt gx release to produce gx- tags. That way the tag namespace is vacated for Go mod, but we preserve the gx lineage under a different namespace, as we’ll want to keep producing gx releases for now. This approach feels more like archiving history vs deleting history.

[Kubuxu]

I agree.

I'm all for not removing history (one of the reasons I created: https://github.com/ipfs/go-ipfs-archived/)

[bigs]

i'm into that @raulk @Kubuxu. could we also ship, with that update, a migration that checks to see if your local copy has been updated and gives you the option to do so if not? could be a git hook.

[anacrolix]

Not using the current major versions in libp2p was driven by incomplete third-party tooling support in libp2p/go-libp2p-kad-dht#236. As mentioned there, the core Go tooling works seamlessly with >= 2 major versions in import paths. Resetting versions will only aid support until version 2+, when again the issue with out-of-date tooling will pop up. On the downside, reverting to version 0 and 1 for projects may cause a lot of confusion, there is likely package managers other than gx in use that depend on the existing versioning, not to mention the human element of associating commits to version numbers. Many conversations around versioning in any project that has reset the version will begin with "version X? do you mean before/after the gx/version reset...". Many of these projects have been around for many years, I think pretending that they're still in v0 is vane and optimistic. Major versions are not a finite resource, and I think people are too easily encumbered by trying to make major versions a very serious thing.

[raulk]

@anacrolix gx doesn't use version numbers, it uses hashes; version numbers are purely vanity, that's why archiving them under a gx- tag namespace won't have implications for gx.

People using govendor and other third party tools should just upgrade to the official solution: gomod. We can't support every single ecosystem tool out there.

My point is that, prior to gomod, bumping up numbers in gx was somewhat done recklessly because it had no implications other than the text of the tag in Git.

However, with gomod, the implications of those version numbers are now visible. Not in metadata, but they actually surface in source files, yikes.

I strongly refuse making users have to go through this pain:

import (
	libp2p "github.com/libp2p/go-libp2p/v6"
	disc "github.com/libp2p/go-libp2p-discovery/v2"
	libp2pdht "github.com/libp2p/go-libp2p-kad-dht/v5"
	inet "github.com/libp2p/go-libp2p-net/v10"
	pstore "github.com/libp2p/go-libp2p-peerstore/v4"
	proto "github.com/libp2p/go-libp2p-protocol/v3"
	multiaddr "github.com/multiformats/go-multiaddr/v9"
	logging "github.com/whyrusleeping/go-logging/v800"
)

We are unlike most Go projects. We are an extremely modular project with independent version numbers that follow no predictable heuristic.

For the Kubernetes monorepo, using v4 (or whatever) all throughout is fine, because all imports will carry that number. We are not like that.

The above example is proof and it's horrendous DX. We'd be insane to accept inflicting that pain on our users, for no compelling gain.

To recap:

1. References to gx releases are hashes, not numbers. I don't expect ambiguity that cannot be resolved with a simple question.
2. gx vanity version numbers start with v1.0.0. Most modules are past v1. If we start gomod with v0.0.1, we have an entire [v0.0.1-v1.0.0) spectrum that's fully unambiguous.
3. To point 2, add the fact that most modules are beyond gx v1. For example, kad-dht is on gx v4.4.x, and if we trust our users to upgrade promptly, it's unlikely people are still running v3, so that makes the de facto unambiguous range [v0.0.1-v4.0.0).
4. Some devs on the team feel libp2p is pre-v1 in general, so this is a healthy reset. The alternative is straightjacketing us into a version lineage that had very different implications in the past (none), than it does now in the present with gomod (see above example).

[whyrusleeping]

@anacrolix's point about the confusion of resetting to <v1 versions is a good one. That sucks. But I think that what @raulk points out about the version numbers in the import paths sucks worse. This is being stuck between a rock and a hard place...

[anacrolix]

@anacrolix gx doesn't use version numbers, it uses hashes; version numbers are purely vanity, that's why archiving them under a gx- tag namespace won't have implications for gx.

It will for any other systems that rely on the tags.

People using govendor and other third party tools should just upgrade to the official solution: gomod. We can't support every single ecosystem tool out there.

Yes, although anybody that already has could have issues when the tags change under them.

However, with gomod, the implications of those version numbers are now visible. Not in metadata, but they actually surface in source files, yikes.

I strongly refuse making users have to go through this pain:

import (
	libp2p "github.com/libp2p/go-libp2p/v6"
	disc "github.com/libp2p/go-libp2p-discovery/v2"
	libp2pdht "github.com/libp2p/go-libp2p-kad-dht/v5"
	inet "github.com/libp2p/go-libp2p-net/v10"
	pstore "github.com/libp2p/go-libp2p-peerstore/v4"
	proto "github.com/libp2p/go-libp2p-protocol/v3"
	multiaddr "github.com/multiformats/go-multiaddr/v9"
	logging "github.com/whyrusleeping/go-logging/v800"
)

Note that this pain is only when you adopt go mod in "compatible" mode. Other packaging schemes are entirely unaffected with the right Go versions in use (for instance GOPATH users, and gx should be unaffected by major version use in paths, other systems may not). There are still replace directives and other tricks to make transitioning more gradual for go mod users. There's also the always ignored fact that you can still use go mod in the poorly-named "incompatible" mode, should you wish to avoid littering your import paths with major versions for a spell.

I think resetting the version is great if it eases the transition for consumers, but I'm not sure that it's necessary, or even correct to do it just because some linting tools break. Note that everything else in the downstream worked correctly without modification, even with major versioning throughout the import paths due to the core Go tool support.

I'm not hardline against the reset, but it smells of the aforementioned excitement that major version numbers always engender. If for example, we set go-libp2p-kad-dht to version 1 so the import paths don't change, as soon as some breaking changes are put in (for traversal stats and metrics for example), we go to version 2 and downstream linters break again. (And we definitely don't want our tools dictating changes.) If we went with version 0, we'd have downstream breakages fairly regularly due to the widespread dependence on the module and inability to communicate breaking changes.

• https://semver.org/#faq
• https://github.com/golang/go/wiki/Modules#faqs--semantic-import-versioning

[vyzo]

Let's not make a mess with the versions in the import paths.
Retagging the gx releases, as suggested by @raulk, is the correct path forward imho.

[hsanjuan]

I am not sure anyone would have got libp2p or ipfs to work by relying on the git tags. Things simply did not work without Gx. By default, Gx does not create git tags either when doing gx release. I would not be surprised if most projects either don't have tags, or have tags but not for all Gx releases, or even have wrong tags that do not correspond exactly to the Gx releases.

Renaming tags in top-level projects is a polite thing, but other than that I'm not sure it's really helpful. If I wanted to look at actual gx releases, rather than tags I would look at the history of lastpubver. If anyone relies on git tags at the moment in a way that things will break, renaming won't help so we might also just delete existing ones altogether. ¯_(ツ)_/¯

[Kubuxu]

Another suggestion from my side is starting from the bottom and going up the dependency chain. Meaning start with go-multihash for example (multiformats/go-multihash#94), do multiaddr, multiaddr-net and so on.

Going on it from the top will just create a mess of indirect dependencies. Ideally, there shouldn't be any/many in the top level modules.

[anacrolix]

Can we proceed then? I want to start using go-ipfs with local development versions of some libp2p packages via go mod ASAP. It's fine to switch in go mod immediately, and incrementally adopt modules from dependencies as they add it. The MVP here is go-ipfs with a go.mod file that works. #5849 was relatively easy to generate, and with some dependencies now already including go module support, can be updated in a few spots. #5435 also looks relevant.

[Kubuxu]

After renaming tags all of us will need to run git fetch --prune origin +refs/tags/*:refs/tags/* over all our local repos to remove old tags. Otherwise, if someone runs git push --tag old tags will be recreated on the remote.

[Kubuxu]

Ok, so the approach would be:

1. Rename all tags in ipfs, libp2p, multiformats (gxed??). * -> gx/*
2. Request all devs with write access to run git fetch --prune origin '+refs/tags/*:refs/tags/*'
3. Write script that checks for tag repushes and fixes them.
4. Start propagating go.mod bottom-up (while marking 0.0.1 releases).
5. Arrive at go-ipfs
6. ....
7. Profit?

How does it sound? Can I get some confirmations about the strategy (@Stebalien @whyrusleeping @raulk @ipfs/go-team)

[raulk]

@Kubuxu LGTM. I’d add making gx create the tag as part of gx release. By automating that, we reduce the chance of human error.

Also, integrating the “tag police” script with GitHub via a webhook, and having it open an issue in the affected repo rather than directly fixing it is probably better. Lesser magic and security exposure.

[raulk]

Should we push to get this over and done with this week across IPFS and libp2p? It’s high prio for us this quarter, and it’s bogging us down. I’d say let’s form a ninja team and just do it. @mvid, would you be having spare capacity to do the legwork on the libp2p side?

[Kubuxu]

@raulk there is whyrusleeping/gx#207 but it's been stuck for a while.

I also agree with webhook but what would be the action of the hook?
EDIT: Of course, I failed reading comprehension. (Opening issue)

[raulk]

@Kubuxu Let's find a way forward with that gx issue; one way or another we need to automate tag creation IMO. Proof lies in the status quo: gx release tagging is erratic across repos.

Re: webhook. I'm thinking we react on tag pushes and bark on tags >= v1.0.0 (start of gx range) that aren't prefixed with gx/. So if someone pushes v2.0.1, the script would open an issue with title "Possibly erroneous tag pushed: $TAG_NAME`, and assign it to the author.

WDYT?

[Kubuxu]

WDYT?

SGWM

One issue I've encontered is that go-multicodec is archived but it is still used by go-libp2p-pnet. We will have to rewrite that.

[Kubuxu]

@ipfs/go-team @libp2p/go-team

Renames in orgs:

• multiformats - go to github.com/multiformats in your GOPATH and run find . -maxdepth 1 -mindepth 1| while read dir; do; cd $dir; git fetch --prune origin '+refs/tags/*:refs/tags/*'; cd ..; done
• libp2p
• ipfs

[Kubuxu]

To all tagging devs: Run find libp2p multiformats -maxdepth 1 -mindepth 1 -type d | while read dir; do; cd $dir; git fetch --prune origin '+refs/tags/*:refs/tags/*'; cd ../..; done in $GOPATH/src/github.com/ to remove old tags to prevent repushing them.

It will refresh tags based on GitHub.

[anacrolix]

@Kubuxu thanks for the info, here and elsewhere. Should we have a document somewhere that contains things like how to fix the tags, what the travis configs should look like, and other common tasks that arise from adopting go mod? I keep needing to refer to this kind of stuff and get lost trying to remember where on GitHub or in an email I saw it mentioned.

[Kubuxu]

Sure. I will write this here and we can move it in future.

Travis files is https://github.com/ipfs/ci-helpers/blob/master/travis-ci/travis.example.yml
Fixing tags:

cd "$GOPATH/src/github.com/"
find libp2p multiformats ipfs -maxdepth 1 -mindepth 1 -type d | while read dir; do; cd $dir; git fetch --prune origin '+refs/tags/*:refs/tags/*'; cd ../..; done

Anything else?

[anacrolix]

I put it in a gist: https://gist.github.com/anacrolix/bc342e13e8052bf9ff737b49b735af8b.

[vyzo]

Can we get the magic incantations for the gomod workflow as well?
I am interested in how to update packages, creating new versions, and bubbling up the update to the dependents.

[Kubuxu]

I will include them in: https://github.com/ipfs/gomod

[raulk]

I'm finishing up a script to add/remove local replace directives in a module uber tree like https://github.com/raulk/uber-libp2p, to facilitate local development.

[raulk]

Ok, so the script is finished and reviewed by @Kubuxu. I'll be adding instructions to that repo tomorrow.

Since it's useful for the IPFS team to hack into libp2p regularly, we might need a version of this uber-repo that is cross-project.

[raulk]

This is... D _ O _ N _ E.