configuring the nats target to reconnect forever

[rayjanoka]

Description

We observed bucket notification events do not resume sending after the NATS server has been offline for more than 5 minutes. A shorter outage, of only a minute or so, does not experience this issue.

In the nats.go lib the client's MaxReconnect default setting is 60 attempts and ReconnectWait is 2 seconds, so it seems the client will only attempt reconnect for a short period before giving up.

I found that setting the nats.MaxReconnects(-1) on the client resolved the issue. I tested after a 20 minute outage, and the messages resumed sending on their own without a minio restart. 👍🏼

Motivation and Context

During testing we found that after an extended network outage was resolved, bucket events do not resume sending until minio is restarted.

How to test this PR?

1. turn up event notifications for bucket PUTs to nats
2. upload objects to the bucket and verify events are delivered to nats successfully
3. stop the nats cluster
4. wait 5 minutes
5. start the nats cluster
6. upload objects to the bucket
7. observe that messages are not delivered to the nats server
8. if a MINIO_NOTIFY_NATS_QUEUE_DIR is defined we can observe that messages are stuck queued in the minio folder, and are not delivered until a minio restart

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Optimization (provides speedup with no functional changes)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• [-] Fixes a regression (If yes, please add commit-id or PR # here)
• [-] Unit tests added/updated
• [-] Internal documentation updated
• [-] Create a documentation update request here

[minio-trusted]

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-compress-encrypt-dist-erasure.sh	✔️
mint-pools.sh	✔️
Deleting image on docker hub
Deleting image locally

[rayjanoka]

hmm, not sure why that one check failed. can we run that one again?

rebased, but it looks like it needs an approval to try again.