missing or extra cluster name in server ls and --trace is not working

[matthiashanel]

config used:

server.conf

port: 4222
server_name: hub-server

leafnodes {
	port: 7422
}

include ./accounts.conf

leaf.conf

port: 4111
server_name: leaf-server

leafnodes {
	remotes = [
		{
			urls: ["nats-leaf://admin:admin@0.0.0.0:7422"]
			account: "SYS"
		},
		{
			urls: ["nats-leaf://acc:acc@0.0.0.0:7422"]
			account: "ACC"
		}
	]
}
include ./accounts.conf

accounts.conf

accounts {
	SYS: {
		users: [{user: admin, password: admin}]
	},
	ACC: {
		users: [{user: acc, password: acc}],
		jetstream: enabled
	}
}
system_account: SYS

# connections without credentials will operate under user acc and thus account ACC
no_auth_user: acc

nats cli command with either an extra cluster name or a missing one. (output is the same no matter which server nats is pointed at)
In addition --trace does not seem to work

nats --trace  --server nats://admin:admin@localhost:4111 server ls
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                           Server Overview                                                           │
├─────────────┬─────────────┬───────────┬───────────────┬────┬───────┬──────┬────────┬─────┬────────┬─────┬──────┬────────┬───────────┤
│ Name        │ Cluster     │ IP        │ Version       │ JS │ Conns │ Subs │ Routes │ GWs │ Mem    │ CPU │ Slow │ Uptime │ RTT       │
├─────────────┼─────────────┼───────────┼───────────────┼────┼───────┼──────┼────────┼─────┼────────┼─────┼──────┼────────┼───────────┤
│ hub-server  │             │ 0.0.0.0   │ 2.3.0-beta.10 │ no │ 0     │ 76   │ 0      │ 0   │ 29 MiB │ 0.1 │ 0    │ 12.99s │ 900.956µs │
│ leaf-server │ leaf-server │ 0.0.0.0   │ 2.3.0-beta.10 │ no │ 1     │ 77   │ 0      │ 0   │ 30 MiB │ 0.0 │ 0    │ 9.35s  │ 887.021µs │
├─────────────┼─────────────┼───────────┼───────────────┼────┼───────┼──────┼────────┼─────┼────────┼─────┼──────┼────────┼───────────┤
│             │ 1 Clusters  │ 2 Servers │               │ 0  │ 1     │ 153  │        │     │ 59 MiB │     │ 0    │        │           │
╰─────────────┴─────────────┴───────────┴───────────────┴────┴───────┴──────┴────────┴─────┴────────┴─────┴──────┴────────┴───────────╯

╭────────────────────────────────────────────────────────────────────────────────╮
│                                Cluster Overview                                │
├─────────────┬────────────┬───────────────────┬───────────────────┬─────────────┤
│ Cluster     │ Node Count │ Outgoing Gateways │ Incoming Gateways │ Connections │
├─────────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│ leaf-server │ 1          │ 0                 │ 0                 │ 1           │
├─────────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│             │ 1          │ 0                 │ 0                 │ 1           │
╰─────────────┴────────────┴───────────────────┴───────────────────┴─────────────╯

[matthiashanel]

When I enable jetstream with different domains in each server I get this output (notice the Missing line)

nats --trace  --server nats://admin:admin@localhost:4222 server ls
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                           Server Overview                                                            │
├─────────────┬─────────────┬───────────┬───────────────┬─────┬───────┬──────┬────────┬─────┬────────┬─────┬──────┬────────┬───────────┤
│ Name        │ Cluster     │ IP        │ Version       │ JS  │ Conns │ Subs │ Routes │ GWs │ Mem    │ CPU │ Slow │ Uptime │ RTT       │
├─────────────┼─────────────┼───────────┼───────────────┼─────┼───────┼──────┼────────┼─────┼────────┼─────┼──────┼────────┼───────────┤
│ leaf-server │ leaf-server │ 0.0.0.0   │ 2.3.0-beta.10 │ yes │ 0     │ 83   │ 0      │ 0   │ 30 MiB │ 0.0 │ 0    │ 16.31s │ 842.252µs │
│ hub-server  │             │ 0.0.0.0   │ 2.3.0-beta.10 │ yes │ 1     │ 80   │ 0      │ 0   │ 32 MiB │ 0.2 │ 0    │ 12.94s │ 814.467µs │
├─────────────┼─────────────┼───────────┼───────────────┼─────┼───────┼──────┼────────┼─────┼────────┼─────┼──────┼────────┼───────────┤
│             │ 1 Clusters  │ 2 Servers │               │ 2   │ 1     │ 163  │        │     │ 62 MiB │     │ 0    │        │           │
╰─────────────┴─────────────┴───────────┴───────────────┴─────┴───────┴──────┴────────┴─────┴────────┴─────┴──────┴────────┴───────────╯

Missing 4294967295 server(s)

╭────────────────────────────────────────────────────────────────────────────────╮
│                                Cluster Overview                                │
├─────────────┬────────────┬───────────────────┬───────────────────┬─────────────┤
│ Cluster     │ Node Count │ Outgoing Gateways │ Incoming Gateways │ Connections │
├─────────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│ leaf-server │ 1          │ 0                 │ 0                 │ 0           │
├─────────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│             │ 1          │ 0                 │ 0                 │ 0           │
╰─────────────┴────────────┴───────────────────┴───────────────────┴─────────────╯

[ripienaar]

hm, so this is must be the result of deteting the amount of connected servers via the new statz addition, can you show statsz output on these machines?

[matthiashanel]

Being able to specify a receive count would be nice.
Here is statsz from both boxes.

> nats --server nats://admin:admin@localhost:4222 req '$SYS.REQ.SERVER.PING.STATSZ' ""
19:34:40 Sending request on "$SYS.REQ.SERVER.PING.STATSZ"
19:34:40 Received on "_INBOX.i83Y5ceMqgUJFLzdq5jGwp.tRMNmpQ1" rtt 599.115µs
{"server":{"name":"hub-server","host":"0.0.0.0","id":"NDIH47GRL7D6G3ZFGLJYJI6QGOU75JFOJBOYCBBQPKD7PQY3DBYVK6OH","ver":"2.3.0-beta.10","seq":48,"jetstream":false,"time":"2021-06-28T23:34:40.616254Z"},"statsz":{"start":"2021-06-28T23:32:20.529777Z","mem":37031936,"cores":16,"cpu":0.3,"connections":1,"total_connections":3,"active_accounts":2,"subscriptions":76,"sent":{"msgs":23,"bytes":9006},"received":{"msgs":12,"bytes":4799},"slow_consumers":0,"active_servers":2}}

>nats --server nats://admin:admin@localhost:4111 req '$SYS.REQ.SERVER.PING.STATSZ' ""
19:34:53 Sending request on "$SYS.REQ.SERVER.PING.STATSZ"
19:34:53 Received on "_INBOX.2KOPU3jzVDCwbYzurdtvdk.iAgntKdr" rtt 692.601µs
{"server":{"name":"leaf-server","host":"0.0.0.0","id":"NA23HJVAWYGXN57WIR743DOCTRDOYEWQYP7K25XH73YRVFMWWWBOKQ76","cluster":"leaf-server","ver":"2.3.0-beta.10","seq":30,"jetstream":false,"time":"2021-06-28T23:34:53.646743Z"},"statsz":{"start":"2021-06-28T23:32:16.771149Z","mem":33652736,"cores":16,"cpu":0,"connections":1,"total_connections":1,"active_accounts":2,"subscriptions":77,"sent":{"msgs":13,"bytes":6095},"received":{"msgs":22,"bytes":7947},"slow_consumers":0,"active_servers":2}}

>

btw --trace does not work on ping either

nats --server nats://admin:admin@localhost:4222 server ping  --trace
hub-server                                                   rtt=693.965µs
leaf-server                                                  rtt=727.602µs

---- ping statistics ----
2 replies max: 0.00 min: 0.00 avg: 0.00

[ripienaar]

So is active servers wrong here? If it is we need to know why and fix that. Else I can’t rely on that number at all.

trace tend to be mainly added where APIs are accessed so many server req ones won’t have it.

You can specify the expected servers generally as last, optional, argument.

[ripienaar]

Ah so this must be because cluster name is not set in the hub? That should not be possible because jetstream requires it. Though your req results shows JS off while list shows JS on, are these the same servers and same settings producing both?

[matthiashanel]

It's not set in the hub. But it's also not set in the leaf.
Yep same servers with config from above.

[ripienaar]

then this is working as expected, your non cluster named environment is not a cluster, why would it be reported as a cluster?

[matthiashanel]

why does the non clustered leaf report as clustered then?
One non clustered server shows as clustered and the other one doesen't.

[ripienaar]

Statz for that leaf has "cluster":"leaf-server" so its clearly configured with a cluster name, which the other doesnt have.

[matthiashanel]

I literally did not configure that. see above

[ripienaar]

Servers are full of magical magic things doing magical setting changes for you. Somethings doing it :) I know in some cases we auto generate cluster names but I would imagine its not going to be auto generated as leaf-server.

Regardless, this is the cause and I can only do what statz tells me alas

[matthiashanel]

oh man magic is fine, inconsistency is not. but yeah, wrong place for this