EventBus container with persistence in CrashLoop - cluster log path is empty

[JonathanSerafini]

Describe the bug
Of 3 EventBus pods 1 was killed and fails to restart stating the following:

[1] 2020/08/12 16:20:17.935735 [FTL] STREAM: Failed to start: streaming state was recovered but cluster log path "eventbus-default/eventbus-default-stan-1" is empty

This is when configured with Persistence to an EBS volume.
According to nats-streaming-server, it would appear that we need to move the -cluster-log-path onto the persistent volume:

nats-io/nats-streaming-server#547

To Reproduce
Steps to reproduce the behavior:
not 100% on the exact initial condition, but this'll do it
note: this command was not used when the initial error occurred, i've just used it to reproduce.

1. Configure Eventbus with persistence
2. kubectl delete --force --grace-period 0 pod/eventbus-default-stan-0

Expected behavior
Nats should restore from persistent volumes.

Environment (please complete the following information):

• Kubernetes: 1.16.8-eks-e16311
• Argo Events: v0.17.0

Additional context
As a side note, it would be very convenient to be able to override the eventbus statefulset and container properties via the CRD to be able to override args or pass additional annotations.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[whynowy]

Thanks, will fix it!