Js leaf deny

[matthiashanel]

From now on JS API traffic will be denied across leaf nodes, unless the system account is shared and the availability domain is identical.
The main reason we start to enforce this is that enabling jetstream in an availability domain would alter the api traffic flow.
For example, a leaf node without jetstream was able to communicate with a JS in the hub without using it's domain name.
If for an unrelated reason jetstream had to be enabled in a the leaf node, this default traffic would all of a sudden go some place else (remain local) where before it'd go to a hub.

To avoid this from the get go we suggest setting the domain in client applications as soon as you need to communicate with a JS other than the local one.
If domain names are already used in such a way, there is nothing to do.

We do also recommend setting domain names for each availability zone.
(Availability zones can only change when crossing leaf nodes. If the leaf node share the system account and specify the same domain name, they are considered within the same availability zone. When this is done, meta group leadership is pinned to server that accept leaf node connections. This is to avoid extra hops.)

If you are currently not using domain names and rely on the old behavior, the server contains a new option default_js_domain that contains a map from account name to default domain name.

default_js_domain:{
 ACC1:domain,
 ACC2:domain,
}

In case the JetStream you want to reach does not have a domain name set just specify "" as domain name.

Please be aware that if you are currently not using domain names, and have to specify the empty domain name, in order to connect accross leaf nodes both ends have to be configured in such a way. This is because with this change, each server is going to make it's individual decission about which traffic to deny. For example. If you have a hub without jetstream and a leaf node that offers JS.
Until you enabled a domain name in the leaf node and change your clients to provide this name, you have to set
default_js_domain:{your-account-in-hub:""} in the hub and default_js_domain:{your-account-in-leaf:""}
This will instruct both server to not deny default api traffic.

Alternatively you could also restart your leaf node and provide a domain name. Then, in the hub set default_js_domain:{your-account-in-hub:"leaf-domain-name"}

In mixed mode setups (rare use case) you will have to set the domain name in the non JS enabled server as well.
Basically telling them, to which availability domain they belong.

jetstream: {
	enabled: false
	domain: domain-this-server-belongs
}

In the rare case where a JS availability domain is extended via a non clustered leaf node, you need to explicitly instruct the server to be in clustered mode with this new JetStream config option extension_hint: will_extend.

There are two more situations where the extension_hint can be used, but the server will print when this option is applicable. Primarily they are there to assist eh server during initial startup until it was able to decide if it's leaf node connection was part of the same or a different availability domain.

[ripienaar]

This default_js_domain:{ ACC1:domain, ACC2:domain, } seems really weird, we have nowhere this kind of structure in the configuration, seems weird to introduce a new style of flag.

why can default_js_domain not be set on the account?

[matthiashanel]

This default_js_domain:{ ACC1:domain, ACC2:domain, } seems really weird, we have nowhere this kind of structure in the configuration, seems weird to introduce a new style of flag.

why can default_js_domain not be set on the account?

It covers jwt as well. (without a jwt change)
I suggested something like that early on and it was rejected.

When the same jwt based account is used on both ends of a leaf node, only having a single value might not be flexible enough. Meaning there might be different defaults depending on where you connect.

Lastly, is only to assist in the transition and is expected to be removed in a subsequent release.
Long term we want apps to provide a domain name if they are communicating with a js not in their (availability domain (in short cluster/supercluster))

[kozlovic]

Just some comments on the MQTT code/tests for now.

[kozlovic]

Granted it would probably be more work, but I thought we discussed that we would actually store a prefix here (either $JS.API or actual $JS.<domain>.API) and add this prefix to all API subjects. We would have to replace existing references to the constants such as JSApiStreamCreateT to something like mqttAPIStreamCreateT that would be defined as the same than JSApiStreamCreateT but without the prefix, in this case %s.STREAM.CREATE.%s. So in place where we currently have:

jsa.newRequest(mqttJSAStreamCreate, fmt.Sprintf(JSApiStreamCreateT, cfg.Name), 0, cfgb)

we would do:

jsa.newRequest(mqttJSAStreamCreate, fmt.Sprintf(mqttAPIStreamCreateT, jsa.apiPfx, cfg.Name), 0, cfgb)

or something similar.

I agree that the places impacted currently should be "low volume", but still, that makes use of more sprintf/memory allocs if you trim the prefix and create the actual prefix in each requests (in newRequestEx).

We could go with your approach first and improve later if you think it is best to minimize changes in this PR.

[matthiashanel]

I will submit a separate PR once this one is oked.

[kozlovic]

I have noticed a DATA RACE in Travis, which I can reproduce locally. Not saying that this is introduced by your changes but instead maybe the new tests show this.

I believe that where we have leafnode.go:1486:

siReply := acc.siReply

we should replace with:

siReply := copyBytes(acc.siReply)

Also, I noticed some missing defer nc.Close().

[kozlovic]

LGTM

[matthiashanel]

Early releases of the Jetstream feature attempted to "do the right thing" in a variety of circumstances, but the logic used proved less resilient to certain changes, allowing some actions-at-a-distance to break previously functioning deployments. We're updating the NATS server logic to provide a more resilient model.

Our advice for developers is that:

1. You should always set a Jetstream Domain in the server
2. If you only ever want to talk to Jetstream in the same server/cluster to which you connect, then the clients do not need to specify a Jetstream Domain
3. In all other cases, the client should set the Jetstream Domain explicitly

Note in particular that a deployment model with a backbone to which leafnodes or leafnode clusters connect needs particular care in enabling Jetstream in the backbone: if clients are not setting a Jetstream Domain and expecting to talk to the one leafnode they have in their account, such an upgrade can break working configurations. Administrators can use default_js_domain/mqtt.js_domain to provide mappings to assist in keeping such clients functioning across a transition. However, we recommend that this be considered a transient upgrade assistance mode, not a long term configuration.

We added js_domain to permanently support mqtt running on the non JetStream enabled end of a leaf node configuration.
As you upgrade these servers, we recommend specifying the domain for mqtt to use, absent a local one.
Regular nats clients need to specify the nats.Domain option.
For mqtt, js_domain is the way to configure that.

For example:

mqtt: {
    js_domain: target-domain
}

Updating clients with a new domain setting may not be as quick or easy as updating nats-server.
That is why we added the default_js_domain. It is a map from account name (or account nkey) to domain (or "" when none was set) to use for client's that don't set a domain.
To avoid outages we recommend first upgrading and re-configuring the server in which accounts and clusters do not have JetStream enabled.
In ngs, if a leaf node enabled JetStream for clients connected to ngs, the domain name is now required.
Existing ngs customers are not affected, and only have to care about upgrading their leaf nodes and connected clients.

For example:

default_js_domain:{
 ACC1:other-domain-name,
 ACC2:my-default-js-traffic-goes-to-this-domain,
 ACC3:""
}

You should only add to default_js_domain accounts that do not have Jetstream enabled.
Once you have changed all your clients to either use the local JetStream or provide the appropriate Domain name and mqtt has js_domain set where needed, you can remove default_js_domain from your configurations.

In the rare case where a JetStream availability domain is extended via a NON-clustered leaf node, you need to explicitly instruct the server to be in clustered mode with the new extension_hint JetStream configuration option.
All potential usages of extension_hint are related to NON-clustered server. Any other potential usages to speed up the startup process will be printed on server startup.
Possible values to set are: will_extend, no_extend

For example:

jetstream {
    extension_hint: will_extend
}

If your cluster consists of a mix of servers with and without JetStream enabled, servers without JetStream enabled need to set the domain name to the same value as servers with JetStream enabled.

jetstream: {
	enabled: false
	domain: name
}