feat: configurable NATS connection retries

[TylerGillson]

Add configuration options & logic for NATS connection retries. Fixes #87.

In our use case, the NATS worker may boot before the NATs leader on occasion, hence these options would be very useful!

[maxpert]

Thanks for the build, but I anticipate that there should be a way to configure it in NATS configuration file and people can do it via that rather than adding additional bloat in configuration. Can you check if that is possible via nats configuration file?

[TylerGillson]

Thanks for the build, but I anticipate that there should be a way to configure it in NATS configuration file and people can do it via that rather than adding additional bloat in configuration. Can you check if that is possible via nats configuration file?

@maxpert My interpretation of the NATS client library is that the reconnection options are only applicable after an initial connection is established.

And I don't see anything promising in the server options either. I will nevertheless toy around with the reconnection options and let you know.

[TylerGillson]

@maxpert your anticipation was correct 😁

However, I still don't see a client configuration file or parser for same in the NATS library.

What do you think about the changes now (heavily influenced by this example)? Only one new configuration option.

Or WDYT about just enabling reconnections by default with some sane total wait time?

[maxpert]

You might be right. Let me explore myself too.

[TylerGillson]

You might be right. Let me explore myself too.

Ok - I did validate this code though, FWIW. My earlier comment was uninformed. The client retries work fine on the initial connection with these changes.

[maxpert]

Change time units, I will have to give it another pass once done. I think this PR makes sens.e

[maxpert]

Couple of more things to fine tune.

[TylerGillson]

@maxpert apologies for the mixup but I was testing with the wrong binary 🤦🏼‍♂️ Had to make a few more changes.

If the initial connection fails, nats.Connect() returns a *nats.Conn whose private conn field is nil, along with a nil error.

So I've reverted this PR back to the original approach where I handle the retry logic on the marmot side. I'd appreciate any insight or opinions you have from your own testing.

Seeing the following behaviour without the marmot-side retries:

[root@localhost marmot]# cat marmot.toml
# Path to target SQLite database
seq_map_path="/etc/kubernetes/marmot-sm.cbor"
db_path="state.db"

[nats]
# address of the nats leader
urls=[
  "nats://fakefakefakenotanats:4222"
]
# Number of retries if establishing a NATS connection
connect_retries=30

# Console STDOUT configurations
[logging]
# Configure console logging
verbose=true
# "console" | "json"
format="console"

[root@localhost marmot]# ./marmot -config marmot.toml
11:07AM DBG Opening database node_id=15460159781391886746 path=state.db
11:07AM DBG Forcing WAL checkpoint node_id=15460159781391886746
11:08AM PNC Unable to initialize snapshot storage error="dial tcp: lookup fakefakefakenotanats: Try again" node_id=15460159781391886746
panic: Unable to initialize snapshot storage

goroutine 1 [running]:
github.com/rs/zerolog/log.Panic.(*Logger).Panic.func1({0x107f170?, 0x0?})
        /root/.gvm/pkgsets/go1.21.3/global/pkg/mod/github.com/rs/zerolog@v1.29.1/log.go:376 +0x27
github.com/rs/zerolog.(*Event).msg(0xc0000a4480, {0x107f170, 0x25})
        /root/.gvm/pkgsets/go1.21.3/global/pkg/mod/github.com/rs/zerolog@v1.29.1/event.go:156 +0x2c2
github.com/rs/zerolog.(*Event).Msg(...)
        /root/.gvm/pkgsets/go1.21.3/global/pkg/mod/github.com/rs/zerolog@v1.29.1/event.go:108
main.main()
        /root/marmot/marmot.go:66 +0x70a
[root@localhost marmot]#

Since we aren't checking the status of the nats.Conn returned by nats.Connect().

[TylerGillson]

@maxpert - it is working as expected now:

[maxpert]

Perfect I will run couple of tests and try it in my sandbox this coming week.

[maxpert]

Tested and few minor nits I want to be ironed out

[TylerGillson]

@maxpert I added ReconnectWaitSeconds per your request, but haven't exposed timeout because I was trying to avoid configuration bloat. Currently the default timeout of 2s is set for the dialer for each connection attempt.

Would you like me to expose that option too?