panic on boot when running gateway

[jeromegn]

Expected Behavior

Should boot up fine.

Current Behavior

I'm seeing a panic:

2021-05-06T20:18:32Z [info] panic: runtime error: invalid memory address or nil pointer dereference
2021-05-06T20:18:32Z [info] [signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x17519a9]
2021-05-06T20:18:32Z [info] goroutine 47 [running]:
2021-05-06T20:18:32Z [info] github.com/minio/minio/cmd.(*NotificationSys).ServerInfo(0x0, 0x0, 0x0, 0x0)
2021-05-06T20:18:32Z [info]     github.com/minio/minio/cmd/notification.go:1273 +0x29
2021-05-06T20:18:32Z [info] github.com/minio/minio/cmd.GetPeerOnlineCount(0x0, 0xc001090480)
2021-05-06T20:18:32Z [info]     github.com/minio/minio/cmd/notification.go:1359 +0x58
2021-05-06T20:18:32Z [info]     github.com/minio/minio/cmd/metrics-v2.go:1002 +0x37
2021-05-06T20:18:32Z [info] github.com/minio/minio/cmd.getNodeHealthMetrics.func1(0x281bd40, 0xc00003ab80, 0x6fc23ac00, 0xc001082d08, 0x254aa18)
2021-05-06T20:18:32Z [info] github.com/minio/minio/cmd.cachedRead.func1.1(0xc001082cc0, 0x0, 0x0, 0xc001082cd0)
2021-05-06T20:18:32Z [info]     github.com/minio/minio/cmd/metrics-v2.go:200 +0x3b
2021-05-06T20:18:32Z [info] github.com/minio/minio/cmd.(*timedValue).Get(0xc001082cc0, 0xc0004fcb10, 0x1f8d9e0, 0x11, 0xc000225e98)
2021-05-06T20:18:32Z [info]     github.com/minio/minio/cmd/utils.go:911 +0x53
2021-05-06T20:18:32Z [info] github.com/minio/minio/cmd.cachedRead(0x281bd40, 0xc00003ab80, 0xc0010c0120, 0x0, 0x0, 0x0)
2021-05-06T20:18:32Z [info]     github.com/minio/minio/cmd/metrics-v2.go:207 +0xf0
2021-05-06T20:18:32Z [info] github.com/minio/minio/cmd.populateAndPublish(0x2547960, 0xc0010c5f00)
2021-05-06T20:18:32Z [info]     github.com/minio/minio/cmd/metrics-v2.go:1537 +0x11a
2021-05-06T20:18:32Z [info] github.com/minio/minio/cmd.(*minioCollectorV2).Collect(0xc000bdbed0, 0xc001082ba0)
2021-05-06T20:18:32Z [info]     github.com/minio/minio/cmd/metrics-v2.go:1552 +0x105
2021-05-06T20:18:32Z [info] github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
2021-05-06T20:18:32Z [info]     github.com/prometheus/client_golang@v1.8.0/prometheus/registry.go:444 +0x12b
2021-05-06T20:18:32Z [info] created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
2021-05-06T20:18:32Z [info]     github.com/prometheus/client_golang@v1.8.0/prometheus/registry.go:536 +0xe4d

Possible Solution

Looks like *NotificationSys is nil here? I'm not sure. Seems odd!

Steps to Reproduce (for bugs)

I'm not sure how to reproduce exactly. I deployed it to fly.io, but it might be reproducible with Docker?

FROM minio/minio:RELEASE.2021-04-22T15-44-28Z

CMD ["minio", "gateway", "s3", "https://s3.dualstack.us-east-1.amazonaws.com/"]

Runtime environment:

MINIO_PROMETHEUS_AUTH_TYPE = "public"
MINIO_CACHE = "on"
MINIO_CACHE_DRIVES = "/data"
MINIO_CACHE_EXPIRY = 30
MINIO_CACHE_QUOTA = 90
MINIO_CACHE_AFTER = 0
MINIO_DOMAIN = "<redacted>"
MINIO_ROOT_USER = "<redacted>"

I'm using a 100G volume, with an ext4 filesystem, completely empty, mounted on /data.

Context

Trying to launch a small fleet of minio gateways to cache S3 puts and gets.

Regression

I'm not sure.

Your Environment

• Version used (minio --version): RELEASE.2021-04-22T15-44-28Z
• Server setup and configuration: 1 vCPU, 2GB RAM (running in a VM) w/ a 100G disk
• Operating System and version (uname -a): Linux 5.9.15

[harshavardhana]

@jeromegn you should turn-off prometheus metrics for now to avoid the crash. Let us investigate.

[jeromegn]

Thank you, that does allow my nodes to boot!