Unified CI Release - 2022-08-23

[galargh]

Resolves #362

Plan

• 2022-08-16: Send out release announcement to Slack (#ipdx + cross-reference in #go (?)) - https://www.notion.so/pl-strflt/Unified-CI-Release-2022-08-23-1390928a2b6645a5b11e81f6c9856386#de3795dbdf2049e5af613b545a081c48
• 2022-08-23+: Check how many unmerged PRs from web3-bot there are - https://github.com/search?q=is%3Aopen+author%3Aweb3-bot (make a note of the repos)
• 2022-08-23: Merge this PR on the 23rd
• 2022-08-23: Make sure the dispatch workflow and copy workflows succeeded
• 2022-08-23: Send out release announcement to shipped@ on the 23rd - https://www.notion.so/pl-strflt/Unified-CI-Release-2022-08-23-1390928a2b6645a5b11e81f6c9856386#972e45f2af0744f6b4b85d525778c215
• 2022-08-23+: Check how many unmerged PRs from web3-bot there are - https://github.com/search?q=is%3Aopen+author%3Aweb3-bot - and try to help resolve the issues that are blocking the PRs from being merged

How to opt-in to Go 1.19 before we perform the official Unified CI release?

I described it in the technical preview section of the README. The logic that enables it is already checked in to master.

Testing

• deploy next to protocol/.github-test-target - sync: update CI config files .github-test-target#37
• ...

[laurentsenta]

Announcements:
https://filecoinproject.slack.com/archives/C03KLC57LKB/p1660813953961239

https://discordapp.com/channels/899652135552167957/940620444564848720/1009756156136271922

[laurentsenta]

before release, web3 bot issues: 31

[laurentsenta]

After release, web3 bot issues: 94

A few new js errors
libp2p/js-libp2p-daemon#125
alanshaw/it-length-prefixed#48

I checked a few go repos, they were all failing because of (deprecated) ioutils errors.

[marten-seemann]

I checked a few go repos, they were all failing because of (deprecated) ioutils errors.

Do we know how many these are? Is there a list of those repos?

[galargh]

I checked a few go repos, they were all failing because of (deprecated) ioutils errors.

Do we know how many these are? Is there a list of those repos?

This would be a list of all PRs that were created since yesterday and were not merged automatically - https://github.com/search?q=is%3Aopen+author%3Aweb3-bot+created%3A%3E%3D2022-08-23

[marten-seemann]

That's 61 Go repositories out of ~120 that Unified CI is configured for. Do we have a plan to get all these PRs merged?

[galargh]

That's 61 Go repositories out of ~120 that Unified CI is configured for. Do we have a plan to get all these PRs merged?

Ideally, we'll see the PRs getting merged by maintainers (we included a link to an example of how to fix the ioutil deprecation in the communication we shared around the release to ease this process). Since we still don't have a great ownership story everywhere though, we'll definitely be helping with that ourselves too, just as we did in the past. Since the volume is quite hefty and there might be other errors lurking around in some repos, I'm afraid we don't have a specific timeline for this.

[marten-seemann]

I thought this wouldn't be necessary, but I clearly underestimate the magnitude of the problem: we could automate this.

The deprecation of ioutil is pretty simple, the functions were just moved to other packages. For example, we could run a replacement over all Go code doing:

• ioutil.ReadAll => io.ReadAll
• ioutil.WriteFile => os.WriteFile
• ...

The only tricky one, as far as I can see, is ioutil.ReadDir, but that one could be done manually.
After the replacement, one would have to run goimports -w to adjust import paths.

This could be done as part of the copy workflow, since it's a noop for repos that don't use ioutil. @galargh, wdyt?

[galargh]

Sure, if it's that straightforward I think it'd make sense to automate it and either add it to the copy workflow or just run the script locally. I'll check how many of those 60+ repos actually use ioutil today and I should be able to get on with the script tomorrow.

[marten-seemann]

Cool, thank you @galargh! I'd be happy to review, feel free to tag me.

[galargh]

I just got the stats:

repositories="$(gh api --paginate -X GET search/issues -F q='is:open author:web3-bot language:go created:2022-08-23' --jq '.items | map(.repository_url[29:])')"
echo ""
echo "REPOSITORY IOUTIL_SEARCH_HITS"
while read repo; do
  count="$(gh api -X GET search/code -F q="repo:${repo} ioutil" --jq '.total_count')";
  echo "${repo} ${count}"
  sleep 10
done <<< "$(jq -cr '.[]' <<< "${repositories}")"

REPOSITORY IOUTIL_SEARCH_HITS
multiformats/go-multiaddr 1
multiformats/ma-pipe 1
multiformats/go-multicodec 1
libp2p/go-yamux 4
libp2p/go-socket-activation 1
libp2p/go-openssl 0
libp2p/go-msgio 0
libp2p/go-mplex 2
libp2p/go-libp2p-xor 0
libp2p/go-libp2p-webrtc-direct 3
libp2p/go-libp2p-relay-daemon 1
libp2p/go-libp2p-raft 1
libp2p/go-libp2p-kad-dht 1
libp2p/go-libp2p-http 1
libp2p/go-libp2p-gostream 1
libp2p/go-libp2p-gorpc 0
libp2p/go-libp2p-circuit 2
libp2p/go-doh-resolver 2
libp2p/go-composable-routing 0
libp2p/go-cidranger 0
ipld/go-ipldtool 1
ipld/go-ipld-prime 6
ipld/go-ipld-btc 1
ipld/go-codec-dagpb 1
ipld/go-car 8
ipld/edelweiss 4
ipfs/tar-utils 2
ipfs/iptb-plugins 5
ipfs/iptb 4
ipfs/interface-go-ipfs-core 3
ipfs/go-unixfsnode 1
ipfs/go-unixfs 6
ipfs/go-pinning-service-http-client 1
ipfs/go-mfs 1
ipfs/go-merkledag 1
ipfs/go-log 1
ipfs/go-ipld-git 2
ipfs/go-ipld-cbor 1
ipfs/go-ipfs-pinner 0
ipfs/go-ipfs-keystore 2
ipfs/go-ipfs-files 10
ipfs/go-ipfs-example-plugin 0
ipfs/go-ipfs-api 4
ipfs/go-filestore 1
ipfs/go-ds-pebble 1
ipfs/go-ds-leveldb 1
ipfs/go-ds-flatfs 5
ipfs/go-ds-badger2 1
ipfs/go-delegated-routing 0
ipfs/go-datastore 6
ipfs/go-cid 0
ipfs/go-bs-sqlite3 1
ipfs/go-bitswap 1
ipfs-shipyard/git-remote-ipld 6
ipfs-shipyard/dnslink-dnsimple 1
filecoin-project/index-provider 7
filecoin-project/go-legs 1
filecoin-project/go-indexer-core 0
filecoin-project/go-bitfield 0
filecoin-project/go-address 1

I think this proves that it'd be worth to automate the ioutil upgrade path.

[galargh]

We forgot to clear source_refs when we were doing the release. We should include this a checkpoint in the future releases.