invalid memory address or nil pointer dereference on variables on replication

[alvin-reyes]

Issue

The main estuary node runs into some runtime error (invalid or nil pointer reference) which causes the daemon to restart.

It seems that the deal address reference sets to nil or dereference upon access. We'll have to safely handle this to prevent this issue.

  * [/ip4/212.248.62.42/tcp/17160] dial backoff	{"app_version": "v0.1.1-6-gb34ddb9"}
2022-06-28T22:06:30.828Z	INFO	estuary	estuary/shuttle.go:132	handling shuttle message: TransferStatus	{"app_version": "v0.1.1-6-gb34ddb9"}
2022-06-28T22:06:30.828Z	INFO	estuary	estuary/shuttle.go:312	handling transfer status rpc update: 786697 false	{"app_version": "v0.1.1-6-gb34ddb9"}
2022-06-28T22:06:30.944Z	WARN	estuary	estuary/replication.go:1712	no status for deal: 784965	{"app_version": "v0.1.1-6-gb34ddb9"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x1ed88e5]

goroutine 5532059 [running]:
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End·dwrap·11()
	/home/why/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.2.0/trace/span.go:244 +0x2a
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0xc00ed68480, {0x0, 0x0, 0x0})
	/home/why/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.2.0/trace/span.go:283 +0x9a2
panic({0x3919c20, 0x99d3110})
	/usr/local/go/src/runtime/panic.go:1047 +0x266
main.(*ContentManager).checkDeal(0xc007d07680, {0x4217fd8, 0xc0ef20f0b0}, 0xc04d5d9e78)
	/home/why/code/estuary/replication.go:1617 +0x2e65
main.(*ContentManager).ensureStorage.func1(0x0)
	/home/why/code/estuary/replication.go:1320 +0x129
created by main.(*ContentManager).ensureStorage
	/home/why/code/estuary/replication.go:1317 +0x9ef
2022-06-28T22:06:33.985Z	INFO	estuary	estuary/main.go:512	estuary version: v0.1.1-6-gb34ddb9	{"app_version": "v0.1.1-6-gb34ddb9"}
2022-06-28T22:06:33.985Z	DEBUG	estuary	estuary/main.go:166	estuary cli flag database is set to postgres=host=127.0.0.1 user=estuary password=superpassword123 dbname=estuary	{"app_version": "v0.1.1-6-gb34ddb9"}
2022-06-28T22:06:33.985Z	DEBUG	estuary	estuary/main.go:166	estuary cli flag apilisten is set to 0.0.0.0:3004	{"app_version": "v0.1.1-6-gb34ddb9"}
2022-06-28T22:06:33.986Z	DEBUG	estuary	estuary/main.go:166	estuary cli flag datadir is set to /data/estuary	{"app_version": "v0.1.1-6-gb34ddb9"}
2022-06-28T22:06:33.986Z	DEBUG	estuary	estuary/main.go:166	estuary cli flag logging is set to false	{"app_version": "v0.1.1-6-gb34ddb9"}
2022-06-28T22:06:33.986Z	DEBUG	estuary	estuary/main.go:166	estuary cli flag hostname is set to https://api.estuary.tech	{"app_version": "v0.1.1-6-gb34ddb9"}
2022-06-28T22:06:33.986Z	DEBUG	estuary	estuary/main.go:166	estuary cli flag fail-deals-on-transfer-failure is set to true	{"app_version": "v0.1.1-6-gb34ddb9"}
2022-06-28T22:06:33.986Z	DEBUG	estuary	estuary/main.go:166	estuary cli flag disable-deal-making is set to false	{"app_version": "v0.1.1-6-gb34ddb9"}
2022-06-28T22:06:33.986Z	DEBUG	estuary	estuary/main.go:166	estuary cli flag disable-content-adding is set to false	{"app_version": "v0.1.1-6-gb34ddb9"}
2022-06-28T22:06:33.986Z	DEBUG	estuary	estuary/main.go:166	estuary cli flag disable-local-content-adding is set to true	{"app_version": "v0.1.1-6-gb34ddb9"}
2022-06-28T22:06:33.986Z	DEBUG	estuary	estuary/main.go:166	estuary cli flag jaeger-tracing is set to true	{"app_version": "v0.1.1-6-gb34ddb9"}
Wallet address is:  f3vnq2cmwig3qjisnx5hobxvsd4drn4f54xfxnv4tciw6vnjdsf5xipgafreprh5riwmgtcirpcdmi3urbg36a
2022-06-28T22:06:35.148Z	INFO	dt-impl	impl/impl.go:145	start data-transfer module
/ip4/139.178.68.217/tcp/6744/p2p/12D3KooWCVXs8P7iq6ao4XhfAmKWrEeuKFWCJgqe9jGDMTqHYBjw
/ip4/127.0.0.1/tcp/6744/p2p/12D3KooWCVXs8P7iq6ao4XhfAmKWrEeuKFWCJgqe9jGDMTqHYBjw
2022-06-28T22:06:35.149Z	INFO	estuary	estuary/replication.go:701	rebuilding staging zones.......	{"app_version": "v0.1.1-6-gb34ddb9"}

   ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.6.1
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\

[alvin-reyes]

@en0ma it'll be great if you can also check this.

I have started to include the fixes in my PR here: #298

[en0ma]

closing this has it has been addressed in #298