identify error: latest peer record does not exist

[yusefnapora]

This came up in the pubsub testground test after updating to go-libp2p v0.9.0 in libp2p/test-plans#7

cc @Stebalien @raulk @aarshkshah1992

Some of the test instances fail when connecting to other peers with the error message latest peer record does not exist. identify message incomplete!:

{"ts":1589905943561862041,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"connecting to peers after 0s"}}
{"ts":1589905943561888675,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"Wait for 30s warmup time before starting publisher"}}
{"ts":1589905943561991254,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-1-0\n"}}
{"ts":1589905943562009499,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-316-0\n"}}
{"ts":1589905943562019205,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-620-0\n"}}
{"ts":1589905943562027234,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-245-0\n"}}
{"ts":1589905943562035772,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-198-0\n"}}
{"ts":1589905943562042256,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-705-0\n"}}
{"ts":1589905943562049569,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-435-0\n"}}
{"ts":1589905943562060049,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-861-0\n"}}
{"ts":1589905943562068983,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-641-0\n"}}
{"ts":1589905943562079080,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-207-0\n"}}
{"ts":1589905943562087331,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-430-0\n"}}
{"ts":1589905943562096942,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-682-0\n"}}
{"ts":1589905943562102722,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-96-0\n"}}
{"ts":1589905943562124749,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-761-0\n"}}
{"ts":1589905943562139127,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-996-0\n"}}
{"ts":1589905943562148204,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-252-0\n"}}
{"ts":1589905943562155546,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-348-0\n"}}
{"ts":1589905943562165009,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-479-0\n"}}
{"ts":1589905943562177192,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-530-0\n"}}
{"ts":1589905943562186903,"msg":"","group_id":"publishers","run_id":"0c7b247a0490","event":{"type":"message","message":"honest-21-0 connecting to honest-267-0\n"}}
2020-05-19T16:32:23.759Z	ERROR	net/identify	identify/id.go:432	latest peer record does not exist. identify message incomplete!
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x6dfb97]

goroutine 234 [running]:
github.com/libp2p/go-libp2p-core/record.(*Envelope).Marshal(0x0, 0xc0005b68e0, 0x8, 0x8, 0x203000, 0x2)
	/go/pkg/mod/github.com/libp2p/go-libp2p-core@v0.5.6/record/envelope.go:196 +0x37
github.com/libp2p/go-libp2p/p2p/protocol/identify.(*IDService).populateMessage(0xc0001f1040, 0xc0007780b0, 0xf83440, 0xc00048a720, 0xc000354380)
	/go/pkg/mod/github.com/libp2p/go-libp2p@v0.9.0/p2p/protocol/identify/id.go:469 +0x4ca
github.com/libp2p/go-libp2p/p2p/protocol/identify.(*IDService).sendIdentifyResp(0xc0001f1040, 0xf83f40, 0xc0003129c0)
	/go/pkg/mod/github.com/libp2p/go-libp2p@v0.9.0/p2p/protocol/identify/id.go:402 +0x39a
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).SetStreamHandler.func1(0xc0001fe280, 0xe, 0x7f0c590e5198, 0xc0003129c0, 0x1, 0x0)
	/go/pkg/mod/github.com/libp2p/go-libp2p@v0.9.0/p2p/host/basic/basic_host.go:476 +0x9d
created by github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).newStreamHandler
	/go/pkg/mod/github.com/libp2p/go-libp2p@v0.9.0/p2p/host/basic/basic_host.go:327 +0x63c

The fact that it doesn't fail immediately suggests that the BasicHost is successfully creating a signed peer record, since it would fail in the constructor otherwise. But something is causing it to be missing when identify asks the peerstore for the record later.

Version Information

github.com/libp2p/test-plans/pubsub/test
cloud.google.com/go v0.26.0
github.com/AndreasBriese/bbloom v0.0.0-20190306092124-e2d15f34fcf9
github.com/BurntSushi/toml v0.3.1
github.com/Kubuxu/go-os-helper v0.0.1
github.com/OneOfOne/xxhash v1.2.2
github.com/aead/siphash v1.0.1
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6
github.com/avast/retry-go v2.6.0+incompatible
github.com/beorn7/perks v1.0.1
github.com/btcsuite/btcd v0.20.1-beta
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f
github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d
github.com/btcsuite/go-socks v0.0.0-20170105172521-4720035b7bfd
github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd
github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723
github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792
github.com/btcsuite/winsvc v1.0.0
github.com/cespare/xxhash v1.1.0
github.com/cespare/xxhash/v2 v2.1.1
github.com/client9/misspell v0.3.4
github.com/coreos/etcd v3.3.10+incompatible
github.com/coreos/go-etcd v2.0.0+incompatible
github.com/coreos/go-semver v0.3.0
github.com/cpuguy83/go-md2man v1.0.10
github.com/davecgh/go-spew v1.1.1
github.com/davidlazar/go-crypto v0.0.0-20190912175916-7055855a373f
github.com/dgraph-io/badger v1.6.1
github.com/dgraph-io/ristretto v0.0.2
github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2
github.com/dustin/go-humanize v1.0.0
github.com/fsnotify/fsnotify v1.4.7
github.com/go-check/check v0.0.0-20180628173108-788fd7840127
github.com/go-kit/kit v0.9.0
github.com/go-logfmt/logfmt v0.4.0
github.com/go-redis/redis/v7 v7.2.0
github.com/go-stack/stack v1.8.0
github.com/gogo/protobuf v1.3.1
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6
github.com/golang/mock v1.1.1
github.com/golang/protobuf v1.3.2
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
github.com/google/go-cmp v0.4.0
github.com/google/gofuzz v1.0.0
github.com/google/gopacket v1.1.17
github.com/google/renameio v0.1.0
github.com/google/uuid v1.1.1
github.com/gorilla/websocket v1.4.2
github.com/gxed/hashland/keccakpg v0.0.1
github.com/gxed/hashland/murmur3 v0.0.1
github.com/hashicorp/errwrap v1.0.0
github.com/hashicorp/go-multierror v1.1.0
github.com/hashicorp/golang-lru v0.5.4
github.com/hashicorp/hcl v1.0.0
github.com/hpcloud/tail v1.0.0
github.com/huin/goupnp v1.0.0
github.com/huin/goutil v0.0.0-20170803182201-1ca381bf3150
github.com/inconshreveable/mousetrap v1.0.0
github.com/influxdata/influxdb1-client v0.0.0-20191209144304-8bf82d3c094d
github.com/ipfs/go-cid v0.0.5
github.com/ipfs/go-datastore v0.4.4
github.com/ipfs/go-detect-race v0.0.1
github.com/ipfs/go-ds-badger v0.2.3
github.com/ipfs/go-ds-leveldb v0.4.2
github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8
github.com/ipfs/go-ipfs-util v0.0.1
github.com/ipfs/go-log v1.0.4
github.com/ipfs/go-log/v2 v2.0.5
github.com/jackpal/gateway v1.0.5
github.com/jackpal/go-nat-pmp v1.0.2
github.com/jbenet/go-cienv v0.1.0
github.com/jbenet/go-temp-err-catcher v0.1.0
github.com/jbenet/goprocess v0.1.4
github.com/jessevdk/go-flags v1.4.0
github.com/jrick/logrotate v1.0.0
github.com/json-iterator/go v1.1.9
github.com/julienschmidt/httprouter v1.2.0
github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d
github.com/kisielk/errcheck v1.2.0
github.com/kisielk/gotool v1.0.0
github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23
github.com/konsorten/go-windows-terminal-sequences v1.0.1
github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515
github.com/kr/pretty v0.2.0
github.com/kr/pty v1.1.1
github.com/kr/text v0.1.0
github.com/libp2p/go-addr-util v0.0.2
github.com/libp2p/go-buffer-pool v0.0.2
github.com/libp2p/go-conn-security-multistream v0.2.0
github.com/libp2p/go-eventbus v0.1.0
github.com/libp2p/go-flow-metrics v0.0.3
github.com/libp2p/go-libp2p v0.9.0
github.com/libp2p/go-libp2p-autonat v0.2.3
github.com/libp2p/go-libp2p-blankhost v0.1.6
github.com/libp2p/go-libp2p-circuit v0.2.2
github.com/libp2p/go-libp2p-core v0.5.6
github.com/libp2p/go-libp2p-crypto v0.1.0
github.com/libp2p/go-libp2p-discovery v0.4.0
github.com/libp2p/go-libp2p-loggables v0.1.0
github.com/libp2p/go-libp2p-mplex v0.2.3
github.com/libp2p/go-libp2p-nat v0.0.6
github.com/libp2p/go-libp2p-netutil v0.1.0
github.com/libp2p/go-libp2p-peer v0.2.0
github.com/libp2p/go-libp2p-peerstore v0.2.4
github.com/libp2p/go-libp2p-pnet v0.2.0
github.com/libp2p/go-libp2p-pubsub v0.2.6
github.com/libp2p/go-libp2p-secio v0.2.2
github.com/libp2p/go-libp2p-swarm v0.2.4
github.com/libp2p/go-libp2p-testing v0.1.1
github.com/libp2p/go-libp2p-tls v0.1.3
github.com/libp2p/go-libp2p-transport-upgrader v0.3.0
github.com/libp2p/go-libp2p-yamux v0.2.7
github.com/libp2p/go-maddr-filter v0.0.5
github.com/libp2p/go-mplex v0.1.2
github.com/libp2p/go-msgio v0.0.4
github.com/libp2p/go-nat v0.0.5
github.com/libp2p/go-netroute v0.1.2
github.com/libp2p/go-openssl v0.0.5
github.com/libp2p/go-reuseport v0.0.1
github.com/libp2p/go-reuseport-transport v0.0.3
github.com/libp2p/go-sockaddr v0.1.0
github.com/libp2p/go-stream-muxer v0.0.1
github.com/libp2p/go-stream-muxer-multistream v0.3.0
github.com/libp2p/go-tcp-transport v0.2.0
github.com/libp2p/go-ws-transport v0.3.1
github.com/libp2p/go-yamux v1.3.6
github.com/magiconair/properties v1.8.0
github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329
github.com/mattn/go-colorable v0.1.1
github.com/mattn/go-isatty v0.0.5
github.com/matttproud/golang_protobuf_extensions v1.0.1
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b
github.com/miekg/dns v1.1.28
github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1
github.com/minio/sha256-simd v0.1.1
github.com/mitchellh/go-homedir v1.1.0
github.com/mitchellh/mapstructure v1.1.2
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
github.com/modern-go/reflect2 v1.0.1
github.com/mr-tron/base58 v1.1.3
github.com/multiformats/go-base32 v0.0.3
github.com/multiformats/go-multiaddr v0.2.2
github.com/multiformats/go-multiaddr-dns v0.2.0
github.com/multiformats/go-multiaddr-fmt v0.1.0
github.com/multiformats/go-multiaddr-net v0.1.5
github.com/multiformats/go-multibase v0.0.2
github.com/multiformats/go-multihash v0.0.13
github.com/multiformats/go-multistream v0.1.1
github.com/multiformats/go-varint v0.0.5
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223
github.com/onsi/ginkgo v1.12.0
github.com/onsi/gomega v1.9.0
github.com/opentracing/opentracing-go v1.1.0
github.com/pelletier/go-toml v1.2.0
github.com/pkg/errors v0.9.1
github.com/pmezard/go-difflib v1.0.0
github.com/prometheus/client_golang v1.5.1
github.com/prometheus/client_model v0.2.0
github.com/prometheus/common v0.9.1
github.com/prometheus/procfs v0.0.8
github.com/rcrowley/go-metrics v0.0.0-20200313005456-10cdbea86bc0
github.com/rogpeppe/go-internal v1.3.0
github.com/russross/blackfriday v1.5.2
github.com/sirupsen/logrus v1.4.2
github.com/smola/gocompat v0.2.0
github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a
github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572
github.com/spaolacci/murmur3 v1.1.0
github.com/spf13/afero v1.1.2
github.com/spf13/cast v1.3.0
github.com/spf13/cobra v0.0.5
github.com/spf13/jwalterweatherman v1.0.0
github.com/spf13/pflag v1.0.3
github.com/spf13/viper v1.3.2
github.com/src-d/envconfig v1.0.0
github.com/stretchr/objx v0.1.1
github.com/stretchr/testify v1.5.1
github.com/syndtr/goleveldb v1.0.0
github.com/testground/sdk-go v0.2.1
github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8
github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1
github.com/whyrusleeping/go-logging v0.0.1
github.com/whyrusleeping/mafmt v1.2.8
github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9
github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7
github.com/whyrusleeping/timecache v0.0.0-20160911033111-cfcb2f1abfee
github.com/x-cray/logrus-prefixed-formatter v0.5.2
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77
go.opencensus.io v0.22.3
go.uber.org/atomic v1.6.0
go.uber.org/goleak v1.0.0
go.uber.org/multierr v1.5.0
go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee
go.uber.org/zap v1.15.0
golang.org/x/crypto v0.0.0-20200510223506-06a226fb4e37
golang.org/x/exp v0.0.0-20190121172915-509febef88a4
golang.org/x/lint v0.0.0-20190930215403-16217165b5de
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee
golang.org/x/net v0.0.0-20200519113804-d87ec0cfa476
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be
golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a
golang.org/x/sys v0.0.0-20200519105757-fe76b779f299
golang.org/x/text v0.3.2
golang.org/x/tools v0.0.0-20191216052735-49a3e744a425
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
google.golang.org/appengine v1.4.0
google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb
google.golang.org/grpc v1.20.1
gopkg.in/alecthomas/kingpin.v2 v2.2.6
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15
gopkg.in/errgo.v2 v2.1.0
gopkg.in/fsnotify.v1 v1.4.7
gopkg.in/src-d/go-cli.v0 v0.0.0-20181105080154-d492247bbc0d
gopkg.in/src-d/go-log.v1 v1.0.1
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
gopkg.in/yaml.v2 v2.2.5
honnef.co/go/tools v0.0.1-2019.2.3

[yusefnapora]

vyzo beat me to it: #939