NATS Streaming support

[pires]

nats-streaming is getting clustering support and so it's about time to support it.

[JeanMertz]

Has there been any progress on this in the recent months, or anything planned for the foreseeable future? It would definitely be great to see the operator support NATS Streaming.

I did find this repo via the official docs: https://github.com/canhnt/k8s-nats-streaming

But my only concern is that it requires a persistent volume with ReadWriteMany support, which is not something that's currently supported on GKE (on which we host our cluster). Wondering if there's a way around that.

correction: it uses NFS for that, and suggests using GlusterFS on Google Cloud, which would solve my above mentioned problem somewhat (even though it's not provided as a product on GCE, so it's self-managed, which is less than ideal for this crucial storage).

[wallyqs]

Hi @JeanMertz yes looking at NATS Streaming support would be next in the list along with prometheus support.

[wallyqs]

Alpha version of a NATS Streaming Operator built using the Operator SDK is also available here for now: https://github.com/nats-io/nats-streaming-operator
Will look at refactoring current version of NATS Operator so that a single operator is responsible of the multiple CRDs as well as that might be more convenient.

[pires]

As mentioned to @wallyqs privately, I think it's best for everybody if we keep both NATS and NATS-Streaming operators as one. After all, NATS needs to be in place for NATS-Streaming to work. Breaking them apart would mean more operations and more code to maintain.

[clanstyles]

I'm very interested in keeping the the two together as well.

[sirajmansour]

Are there any updates on this ? We're very keen to use it in our pipeline. Appreciate all the work you guys have been doing 👍

[wallyqs]

@sirajmansour I'll resume on this next week, can also try the nats streaming operator built with operator sdk: https://github.com/nats-io/nats-streaming-operator

[pires]

There has been a lot of work on the structure of the code, so we can have one operator to rule them all, instead of maintaining separate repos. I'll be working on this over the coming weeks.

[diogogmt]

@pires does that mean that the nats-streaming-operator will be deprecated in favor of this nats-opertor that will support both altogether?

[wallyqs]

@diogogmt we would continue to support the nats-streaming-operator too since some may like the separation of responsibilities and streaming operator fairly lightweight anyhow.

[pires]

OK so it seems we are misaligned and I'm stopping any work to support NATS Streaming in this repo.

[ColinSullivan1]

In the context of user experience, development, and contributions, what are the advantages and disadvantages of one operator for NATS versus having multiple operators?

For longer term planning, how would each approach unfold as we add new components (e.g. JetStream, Leaf Nodes), or deprecate parts of NATS?

[pires]

From the top of my head here's my pov on having two operators.

Pros

• Existing users of the nats-streaming-operator use the operator as is today.

Cons

• There are two codebases to maintain, including two different, non-integrated APIs.
• Their design is different, and quality is as well. For instance, nats-streaming-operator has some basic e2e tests when compared to nats-operator, which provides a more complete e2e test suite, covering functionality that is only available based on feature-gates.

[wallyqs]

OK from the pov of maintenance:

• The streaming operator is fairly straightforward as it is right now, with main controller logic is around 700 loc, technically does not block already from being able to import it and have it be embedded into this operator as a goroutine for example to have two controllers running if needed.

• nats-operator tests take 30 min to run so using separate repos could decouple the testing and improve in separate for the nats streaming operator to only test the stateful aspects required for nats streaming might be good too.

• The NATS Streaming Server also supports embedded mode which requires configuring the NATS transport itself that the nats-operator could be well suited for. Besides logistics around maintenance, I think this could be were nats-operator support could come in handy, since
  the other alternative is for streaming nodes to be able to set soft affinity to nats nodes.

For longer term planning, I think newer features related to NATS transport would be a good addition for the nats-operator.

[ColinSullivan1]

I think we should fold the NATS streaming operator functionality into this operator. test will run longer, but one stop shopping for users would be good imo. Flags could enable which components are installed - would it be feasible to have a streaming.enabled flag, disabled by default, to determine if streaming features are installed?

[pires]

@ColinSullivan1 I don't think we even need flags. Just a different API resource (maybe the one from the other operator impl) or changes to NatsCluster resource, and adapt the operator to understand those.