New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connection closes and server shuts down on launch, especially on devices with low-performance #676

Open
meinside opened this Issue Oct 19, 2018 · 2 comments

Comments

Projects
None yet
2 participants
@meinside

meinside commented Oct 19, 2018

Hi, I'm not sure if it's a known issue, but I've been pulling hairs for over three days due to this issue, so I had to ask here.

I was building a NATS/STAN cluster built up with three Raspberry Pis:

  • Pi 3B+ (quad core + 1GB RAM),
  • Pi 3B (quad core + 1GB RAM),
  • and Pi Zero W (single core + 512MB RAM).

There's no problem on 3B+ and 3B, but the one with least computing power - Zero W, constantly closes connection on launch and shuts down immediately.

Strange thing is, when I run gnatsd locally and make nats-streaming-server connect to it, it just works without any problem.

And also, when tls options are removed from the config, nats-streaming-server(with embedded NATS server) works OK.

So now I'm wondering if there would be any case of connection failure on devices with low-performance, especially when tls option is enabled and embedded NATS server is running.


Here is my stan.conf (with embedded NATS) that doesn't work:

################
# NATS specific configurations

listen: 0.0.0.0:60200 # host/port to listen for client connections

https: 0.0.0.0:60202 # HTTPS monitoring port

tls: {
	cert_file: "./certs/server-cert.pem"
	key_file: "./certs/server-key.pem"
	ca_file: "./certs/ca.pem"
	verify: true
	timeout: 10
}

# Authorization for client connections
authorization {
	users = [
		{
			user: USER
			password: PASSWORD
		}
	]
}

# Cluster definition
cluster {
	listen: 0.0.0.0:60201 # host/port for inbound route connections

	tls {
		cert_file: "./certs/server-cert.pem"
		key_file: "./certs/server-key.pem"
		ca_file: "./certs/ca.pem"
		verify: true
	    timeout: 10
	}

	# Authorization for route connections
	authorization {
		user: CLUSTER_USER
		password: PASSWORD
	}

	# Routes are actively solicited and connected to from this server.
	# Other servers can connect to us if they supply the correct credentials
	# in their routes definitions from above.
	routes = [
		"nats://CLUSTER_USER:PASSWORD@192.168.0.29:60101"
	]
}

# logging options
debug: false
trace: false
logtime: true
log_file: "/tmp/nats.log"

# pid file
pid_file: "/tmp/nats.pid"

# Some system overides

# max_connections
max_connections: 100

# max_subscriptions (per connection)
max_subscriptions: 100

# maximum protocol control line
max_control_line: 512

# maximum payload
max_payload: 65536

# Duration the server can block on a socket write to a client.
# Exceeding the deadline will designate a client as a slow consumer.
write_deadline: "10s"

################
# STAN specific configurations

streaming: {
	id: "stan-rpi"

	store: "file"
	dir: "/data/stan/datastore"

	hb_interval: "10s"
	hb_timeout: "10s"
	hb_fail_count: 5

	#ft_group: "ft"	# for fault-tolerance setting

	sd: false	# debug logging
	sv: false	# trace logging

	tls: {
		client_cert: "./certs/cert.pem"
		client_key: "./certs/key.pem"
		client_ca: "./certs/ca.pem"
		timeout: 10
	}

	store_limits: {
		max_channels: 100
		max_msgs: 1000
		max_bytes: 10MB
		max_age: "6h"
		max_subs: 100
		max_inactivity: "6h"
	}

	file: {
		compact: true
		compact_fragmentation: 50
		compact_interval: 300
		compact_min_size: 100MB
		buffer_size: 2MB
		crc: true
		sync_on_flush: true
		file_descriptors_limit: 100
		parallel_recovery: 1
	}

	cluster: {
		node_id: "stan-node-rpi0w"
		bootstrap: false
		#peers: ["stan-node-rpi3bplus", "stan-node-rpi3b"]

		log_path: "/data/stan/log"
		log_cache_size: 1024
		log_snapshots: 1
		trailing_logs: 256
		sync: true
		raft_logging: false
	}
}

I started with nats-streaming-server -sc stan.conf -c stan.conf -user USER -pass PASSWORD

and it shut down with following logs:

[31616] 2018/10/19 11:19:44.120633 [INF] STREAM: Starting nats-streaming-server[stan-rpi] version 0.11.2                           [22/1981][31616] 2018/10/19 11:19:44.121198 [INF] STREAM: ServerID: THbPvQLhAw6bRZUQgASQ5E
[31616] 2018/10/19 11:19:44.121312 [INF] STREAM: Go version: go1.11.1                                                                       [31616] 2018/10/19 11:19:44.124801 [INF] Starting nats-server version 1.3.0
[31616] 2018/10/19 11:19:44.125068 [DBG] Go build version go1.11.1                                                                          [31616] 2018/10/19 11:19:44.125192 [INF] Git commit [not set]
[31616] 2018/10/19 11:19:44.127373 [INF] Starting https monitor on 0.0.0.0:60202                                                            [31616] 2018/10/19 11:19:44.127933 [INF] Listening for client connections on 0.0.0.0:60200
[31616] 2018/10/19 11:19:44.128084 [INF] TLS required for client connections
[31616] 2018/10/19 11:19:44.128182 [DBG] Server id is f9Qm3FOXrGpXjrO0PhAFyO                                                                [31616] 2018/10/19 11:19:44.128267 [INF] Server is ready
[31616] 2018/10/19 11:19:44.130426 [INF] Listening for route connections on 0.0.0.0:60201                                                   [31616] 2018/10/19 11:19:44.131857 [DBG] Trying to connect to route on 192.168.0.29:60101
[31616] 2018/10/19 11:19:44.135166 [DBG] 192.168.0.29:60101 - rid:1 - Starting TLS route client handshake
[31616] 2018/10/19 11:19:44.161735 [TRC] STREAM:  NATS conn opts: { [nats://USER:PASSWORD@127.0.0.1:60200] false false _NSS-stan-rpi-send fa
lse false true 0x118a9a0 true -1 250ms 2s 30s 0s 2m0s 2 0x47fb14 0x47fb48 0x47fae0 <nil> 0x47fa94 -1 8192    <nil> <nil> false}             [31616] 2018/10/19 11:19:44.163514 [DBG] 127.0.0.1:58446 - cid:2 - Client connection created
[31616] 2018/10/19 11:19:44.164281 [DBG] 127.0.0.1:58446 - cid:2 - Starting TLS client connection handshake
[31616] 2018/10/19 11:19:46.156095 [DBG] 192.168.0.29:60101 - rid:1 - TLS handshake complete
[31616] 2018/10/19 11:19:46.156344 [DBG] 192.168.0.29:60101 - rid:1 - TLS version 1.2, cipher suite TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
[31616] 2018/10/19 11:19:46.156469 [DBG] 192.168.0.29:60101 - rid:1 - Route connect msg sent
[31616] 2018/10/19 11:19:46.158204 [INF] 192.168.0.29:60101 - rid:1 - Route connection created
[31616] 2018/10/19 11:19:46.176384 [DBG] 192.168.0.29:60101 - rid:1 - Registering remote route "MTid6xV2O2hR1AbarTqg1T"
[31616] 2018/10/19 11:19:46.176699 [DBG] 192.168.0.29:60101 - rid:1 - Sent local subscriptions to route
[31616] 2018/10/19 11:19:46.177124 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.close.stan-rpi  RSID:3:6]
[31616] 2018/10/19 11:19:46.177399 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.raft.stan-rpi.join  RSID:5:2]
[31616] 2018/10/19 11:19:46.177635 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.raft.stan-rpi.join  RSID:26:2]
[31616] 2018/10/19 11:19:46.177841 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.raft.stan-rpi.bootstrap  RSID:5:3]
[31616] 2018/10/19 11:19:46.178057 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.discover.stan-rpi  RSID:3:1]
[31616] 2018/10/19 11:19:46.178278 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.discover.stan-rpi.pings  RSID:3:7]
[31616] 2018/10/19 11:19:46.178492 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.pub.stan-rpi.>  RSID:3:2]
[31616] 2018/10/19 11:19:46.178720 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.sub.stan-rpi  RSID:3:3]
[31616] 2018/10/19 11:19:46.178944 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.unsub.stan-rpi  RSID:3:4]
[31616] 2018/10/19 11:19:46.179190 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.subclose.stan-rpi  RSID:3:5]
[31616] 2018/10/19 11:19:46.179418 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.1cUxi585L53adi0YBWfICy.*  RSID:5:10]
[31616] 2018/10/19 11:19:46.179652 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.1cUxi585L53adi0YBWfI5i  RSID:5:4]
[31616] 2018/10/19 11:19:46.180423 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.accept  RSID:5:1]
[31616] 2018/10/19 11:19:46.180748 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.request._INBOX.1cU
xi585L53adi0YBWfIjd  RSID:5:26]
[31616] 2018/10/19 11:19:46.181046 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.request._INBOX.1cU
xi585L53adi0YBWfIm3  RSID:5:27]
[31616] 2018/10/19 11:19:46.181373 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.request._INBOX.1cU
xi585L53adi0YBWfIHo  RSID:5:12]
[31616] 2018/10/19 11:19:46.181611 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.request._INBOX.1cU
xi585L53adi0YBWfILR  RSID:5:14]
[31616] 2018/10/19 11:19:46.181839 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3b.stan-rpi.accept  RSID:26:1]
[31616] 2018/10/19 11:19:46.182080 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3b.stan-rpi.request._INBOX.1Z76kFB
NoCi6etQi6J3B8P  RSID:26:6]
[31616] 2018/10/19 11:19:46.182331 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3b.stan-rpi.request._INBOX.1Z76kFB
NoCi6etQi6J3B3o  RSID:26:5]
[31616] 2018/10/19 11:19:46.297953 [INF] STREAM: Shutting down.
[31616] 2018/10/19 11:19:46.298804 [DBG] 127.0.0.1:58446 - cid:2 - Client connection closed
[31616] 2018/10/19 11:19:46.299469 [ERR] 127.0.0.1:58446 - cid:2 - TLS handshake error: read tcp 127.0.0.1:60200->127.0.0.1:58446: use of cl
osed network connection
[31616] 2018/10/19 11:19:46.299781 [DBG] 192.168.0.29:60101 - rid:1 - Router connection closed
[31616] 2018/10/19 11:19:46.301241 [INF] Server Exiting..
[31616] 2018/10/19 11:19:46.303031 [DBG] Router accept loop exiting..
[31616] 2018/10/19 11:19:46.303282 [FTL] STREAM: Failed to start: write tcp 127.0.0.1:58446->127.0.0.1:60200: i/o timeout

And here are splitted configs for both NATS server and STAN that work:

This is nats.conf:

################
# NATS specific configurations

listen: 0.0.0.0:60200 # host/port to listen for client connections

https: 0.0.0.0:60202 # HTTPS monitoring port

tls: {
	cert_file: "./certs/server-cert.pem"
	key_file: "./certs/server-key.pem"
	ca_file: "./certs/ca.pem"
	verify: true
	timeout: 10
}

# Authorization for client connections
authorization {
	users = [
		{
			user: USER
			password: PASSWORD
		}
	]
}

# Cluster definition
cluster {
	listen: 0.0.0.0:60201 # host/port for inbound route connections

	tls {
		cert_file: "./certs/server-cert.pem"
		key_file: "./certs/server-key.pem"
		ca_file: "./certs/ca.pem"
		verify: true
	    timeout: 10
	}

	# Authorization for route connections
	authorization {
		user: CLUSTER_USER
		password: PASSWORD
	}

	# Routes are actively solicited and connected to from this server.
	# Other servers can connect to us if they supply the correct credentials
	# in their routes definitions from above.
	routes = [
		"nats://CLUSTER_USER:PASSWORD@192.168.0.29:60101"
	]
}

# logging options
debug: false
trace: false
logtime: true
log_file: "/tmp/nats.log"

# pid file
pid_file: "/tmp/nats.pid"

# Some system overides

# max_connections
max_connections: 100

# max_subscriptions (per connection)
max_subscriptions: 100

# maximum protocol control line
max_control_line: 512

# maximum payload
max_payload: 65536

# Duration the server can block on a socket write to a client.
# Exceeding the deadline will designate a client as a slow consumer.
write_deadline: "10s"

I started NATS with gnatsd -c nats.conf,

and this is stan2.conf that connects to NATS server launched above:

# logging options
debug: false
trace: false
logtime: true
log_file: "/tmp/nats.log"

# pid file
pid_file: "/tmp/nats.pid"

# Some system overides

# max_connections
max_connections: 100

# max_subscriptions (per connection)
max_subscriptions: 100

# maximum protocol control line
max_control_line: 512

# maximum payload
max_payload: 65536

# Duration the server can block on a socket write to a client.
# Exceeding the deadline will designate a client as a slow consumer.
write_deadline: "10s"

################
# STAN specific configurations

streaming: {
	id: "stan-rpi"

	store: "file"
	dir: "/data/stan/datastore"

	nats_server_url: "nats://USER:PASSWORD@127.0.0.1:60200"

	hb_interval: "10s"
	hb_timeout: "10s"
	hb_fail_count: 5

	#ft_group: "ft"	# for fault-tolerance setting

	sd: false	# debug logging
	sv: false	# trace logging

	tls: {
		client_cert: "./certs/cert.pem"
		client_key: "./certs/key.pem"
		client_ca: "./certs/ca.pem"
		timeout: 10
	}

	store_limits: {
		max_channels: 100
		max_msgs: 1000
		max_bytes: 10MB
		max_age: "6h"
		max_subs: 100
		max_inactivity: "6h"
	}

	file: {
		compact: true
		compact_fragmentation: 50
		compact_interval: 300
		compact_min_size: 100MB
		buffer_size: 2MB
		crc: true
		sync_on_flush: true
		file_descriptors_limit: 100
		parallel_recovery: 1
	}

	cluster: {
		node_id: "stan-node-rpi0w"
		bootstrap: false
		#peers: ["stan-node-rpi3bplus", "stan-node-rpi3b"]

		log_path: "/data/stan/log"
		log_cache_size: 1024
		log_snapshots: 1
		trailing_logs: 256
		sync: true
		raft_logging: false
	}
}

I ran it with nats-streaming-server -sc stan2.conf.

And these work perfectly.


For your information, there's a seed STAN server with embedded NATS running on 192.168.0.29.

If you need any more information about this, please let me know.

Thanks in advance.

@kozlovic

This comment has been minimized.

Member

kozlovic commented Oct 19, 2018

Can you show the output of the working setup (when using 2 config files)?

@meinside

This comment has been minimized.

meinside commented Oct 21, 2018

I'm sorry I kept you waiting.

This is the output (with all possible debug options on) of the working setup on the low-perf machine:

[24277] 2018/10/21 17:08:45.930891 [INF] Starting nats-server version 1.3.1
[24277] 2018/10/21 17:08:45.935160 [DBG] Go build version go1.11.1
[24277] 2018/10/21 17:08:45.936967 [INF] Git commit [not set]
[24277] 2018/10/21 17:08:45.937348 [WRN] Plaintext passwords detected, use nkeys or bcrypt.
[24277] 2018/10/21 17:08:45.940776 [INF] Starting https monitor on 0.0.0.0:60202
[24277] 2018/10/21 17:08:45.942546 [INF] Listening for client connections on 0.0.0.0:60200
[24277] 2018/10/21 17:08:45.944078 [INF] TLS required for client connections
[24277] 2018/10/21 17:08:45.945281 [DBG] Server id is ZMeiQrEyi58C4lFmA0OEeO
[24277] 2018/10/21 17:08:45.945661 [INF] Server is ready
[24277] 2018/10/21 17:08:45.953105 [INF] Listening for route connections on 0.0.0.0:60201
[24277] 2018/10/21 17:08:45.956023 [DBG] Trying to connect to route on 192.168.0.29:60101
[24277] 2018/10/21 17:08:45.963507 [DBG] 192.168.0.29:60101 - rid:1 - Starting TLS route client handshake
[24277] 2018/10/21 17:08:47.334763 [DBG] 192.168.0.29:60101 - rid:1 - TLS handshake complete
[24277] 2018/10/21 17:08:47.335024 [DBG] 192.168.0.29:60101 - rid:1 - TLS version 1.2, cipher suite TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
[24277] 2018/10/21 17:08:47.335168 [DBG] 192.168.0.29:60101 - rid:1 - Route connect msg sent
[24277] 2018/10/21 17:08:47.336438 [INF] 192.168.0.29:60101 - rid:1 - Route connection created
[24277] 2018/10/21 17:08:47.342604 [DBG] 192.168.0.29:60101 - rid:1 - Registering remote route "397B43bYp5TkCq0VOO9w0g"
[24277] 2018/10/21 17:08:47.346235 [DBG] 192.168.0.29:60101 - rid:1 - Sent local subscriptions to route
[24277] 2018/10/21 17:08:47.349332 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.UiuE0OrOG6TSdWwu5UJ5bl  RSID:13:1]
[24277] 2018/10/21 17:08:47.351292 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.izXIYT7IPQAz3HaDWVwdhW.*  RSID:9:7]
[24277] 2018/10/21 17:08:47.353503 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.UiuE0OrOG6TSdWwu5UJ5pQ  RSID:13:6]
[24277] 2018/10/21 17:08:47.355522 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.izXIYT7IPQAz3HaDWVwf3O  RSID:4:58]
[24277] 2018/10/21 17:08:47.357525 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.izXIYT7IPQAz3HaDWVwf7a  RSID:4:59]
[24277] 2018/10/21 17:08:47.359612 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.hRWNeoO963tW1kCEHpBh8o  RSID:5:4]
[24277] 2018/10/21 17:08:47.361542 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.izXIYT7IPQAz3HaDWVwemc  RSID:4:54]
[24277] 2018/10/21 17:08:47.363641 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.izXIYT7IPQAz3HaDWVweqo  RSID:4:55]
[24277] 2018/10/21 17:08:47.365663 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.izXIYT7IPQAz3HaDWVwev0  RSID:4:56]
[24277] 2018/10/21 17:08:47.367836 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.hRWNeoO963tW1kCEHpBhXK.*  RSID:5:9]
[24277] 2018/10/21 17:08:47.370008 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.izXIYT7IPQAz3HaDWVweAs.*  RSID:12:8]
[24277] 2018/10/21 17:08:47.372761 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.UiuE0OrOG6TSdWwu5UJ68w  RSID:13:11]
[24277] 2018/10/21 17:08:47.375068 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.izXIYT7IPQAz3HaDWVwfBm  RSID:4:60]
[24277] 2018/10/21 17:08:47.377020 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.UiuE0OrOG6TSdWwu5UJ5ff.*  RSID:13:3]
[24277] 2018/10/21 17:08:47.379366 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.UiuE0OrOG6TSdWwu5UJ5xE  RSID:13:8]
[24277] 2018/10/21 17:08:47.381318 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.UiuE0OrOG6TSdWwu5UJ618  RSID:13:9]
[24277] 2018/10/21 17:08:47.383567 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.UiuE0OrOG6TSdWwu5UJ652  RSID:13:10]
[24277] 2018/10/21 17:08:47.385742 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.UiuE0OrOG6TSdWwu5UJ5di  RSID:13:2]
[24277] 2018/10/21 17:08:47.387731 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.UiuE0OrOG6TSdWwu5UJ5lW  RSID:13:5]
[24277] 2018/10/21 17:08:47.389729 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.hRWNeoO963tW1kCEHpBi5e.*  RSID:2:8]
[24277] 2018/10/21 17:08:47.391951 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.UiuE0OrOG6TSdWwu5UJ5tK  RSID:13:7]
[24277] 2018/10/21 17:08:47.394687 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _INBOX.izXIYT7IPQAz3HaDWVwezC  RSID:4:57]
[24277] 2018/10/21 17:08:47.396647 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.accept  RSID:5:1]
[24277] 2018/10/21 17:08:47.398926 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.request._INBOX.hRWNeoO963tW1kCEHpsvSw  RSID:5:12931]
[24277] 2018/10/21 17:08:47.401257 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.request._INBOX.hRWNeoO963tW1kCEHpVtbC  RSID:5:6040]
[24277] 2018/10/21 17:08:47.403709 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.request._INBOX.hRWNeoO963tW1kCEHq6nfK  RSID:5:17084]
[24277] 2018/10/21 17:08:47.405952 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3b.stan-rpi.accept  RSID:9:1]
[24277] 2018/10/21 17:08:47.408784 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3b.stan-rpi.request._INBOX.izXIYT7IPQAz3HaDWVx7KI  RSID:9:20]
[24277] 2018/10/21 17:08:47.411163 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3b.stan-rpi.request._INBOX.izXIYT7IPQAz3HaDWVx7Sg  RSID:9:22]
[24277] 2018/10/21 17:08:47.413682 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3b.stan-rpi.request._INBOX.izXIYT7IPQAz3HaDWWUGuQ  RSID:9:11652]
[24277] 2018/10/21 17:08:47.416060 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3b.stan-rpi.request._INBOX.izXIYT7IPQAz3HaDWVx7OU  RSID:9:21]
[24277] 2018/10/21 17:08:47.418305 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3b.stan-rpi.request._INBOX.izXIYT7IPQAz3HaDWWUGyc  RSID:9:11653]
[24277] 2018/10/21 17:08:47.420614 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3b.stan-rpi.request._INBOX.izXIYT7IPQAz3HaDWWT9v0  RSID:9:11258]
[24277] 2018/10/21 17:08:47.422985 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.close.stan-rpi  RSID:2:56]
[24277] 2018/10/21 17:08:47.425178 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.discover.stan-rpi  RSID:2:51]
[24277] 2018/10/21 17:08:47.427268 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.discover.stan-rpi.pings  RSID:2:57]
[24277] 2018/10/21 17:08:47.428224 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.pub.stan-rpi.>  RSID:2:52]
[24277] 2018/10/21 17:08:47.430531 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.sub.stan-rpi  RSID:2:53]
[24277] 2018/10/21 17:08:47.432792 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.subclose.stan-rpi  RSID:2:55]
[24277] 2018/10/21 17:08:47.434774 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.snap.stan-rpi.pong  RSID:7:19]
[24277] 2018/10/21 17:08:47.436851 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.snap.stan-rpi.messages.message  RSID:6:57]
[24277] 2018/10/21 17:08:47.440456 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.snap.stan-rpi.ping  RSID:7:23]
[24277] 2018/10/21 17:08:47.448186 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.snap.stan-rpi.requests.camera.rpi  RSID:6:62]
[24277] 2018/10/21 17:08:47.449257 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.snap.stan-rpi.requests.webhook.incoming  RSID:6:56]
[24277] 2018/10/21 17:08:47.450173 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.snap.stan-rpi.requests.weather  RSID:6:58]
[24277] 2018/10/21 17:08:47.454954 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.snap.stan-rpi.requests.translation  RSID:6:59]
[24277] 2018/10/21 17:08:47.456008 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.snap.stan-rpi.requests.peterbank  RSID:6:60]
[24277] 2018/10/21 17:08:47.457005 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.snap.stan-rpi.requests.status.bot  RSID:6:61]
[24277] 2018/10/21 17:08:47.457876 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.raft.stan-rpi.join  RSID:5:2]
[24277] 2018/10/21 17:08:47.458804 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.raft.stan-rpi.join  RSID:9:2]
[24277] 2018/10/21 17:08:47.462726 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.raft.stan-rpi.bootstrap  RSID:5:3]
[24277] 2018/10/21 17:08:47.463884 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.acks.UiuE0OrOG6TSdWwu5UJ5jZ  RSID:13:4]
[24277] 2018/10/21 17:08:47.464789 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB _STAN.unsub.stan-rpi  RSID:2:54]
[24277] 2018/10/21 17:08:48.664550 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.request._INBOX.hRWNeoO963tW1kCEHqAdOQ  RSID:5:18239]
[24285] 2018/10/21 17:08:49.202347 [INF] STREAM: Starting nats-streaming-server[stan-rpi] version 0.11.2
[24285] 2018/10/21 17:08:49.204100 [INF] STREAM: ServerID: 5kgGsaKFPIUp91OuQ3IUwH
[24285] 2018/10/21 17:08:49.205311 [INF] STREAM: Go version: go1.11.1
[24285] 2018/10/21 17:08:49.221085 [TRC] STREAM:  NATS conn opts: { [nats://USER:PASSWORD@127.0.0.1:60200] false false _NSS-stan-rpi-send false false true 0x2801c00 true -1 250ms 2s 30s 0s 2m0s 2 0x47fb14 0x47fb48 0x47fae0 <nil> 0x47fa94 -1 8192    <nil> <nil> false}
[24277] 2018/10/21 17:08:49.225531 [DBG] 127.0.0.1:33650 - cid:2 - Client connection created
[24277] 2018/10/21 17:08:49.226388 [DBG] 127.0.0.1:33650 - cid:2 - Starting TLS client connection handshake
[24277] 2018/10/21 17:08:49.318430 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.request._INBOX.hRWNeoO963tW1kCEHqAdYE  RSID:5:18240]
[24277] 2018/10/21 17:08:49.550689 [DBG] 127.0.0.1:33650 - cid:2 - TLS handshake complete
[24277] 2018/10/21 17:08:49.552125 [DBG] 127.0.0.1:33650 - cid:2 - TLS version 1.2, cipher suite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[24277] 2018/10/21 17:08:49.554719 [TRC] 127.0.0.1:33650 - cid:2 - ->> [CONNECT {"verbose":false,"pedantic":false,"user":"USER","pass":"[REDACTED]","tls_required":true,"name":"_NSS-stan-rpi-send","lang":"go","version":"1.6.0","protocol":1,"echo":true}]
[24277] 2018/10/21 17:08:49.557387 [TRC] 127.0.0.1:33650 - cid:2 - ->> [PING]
[24277] 2018/10/21 17:08:49.558695 [TRC] 127.0.0.1:33650 - cid:2 - <<- [PONG]
[24285] 2018/10/21 17:08:49.571820 [TRC] STREAM:  NATS conn opts: { [nats://USER:PASSWORD@127.0.0.1:60200] false false _NSS-stan-rpi-general false false true 0x29a6000 true -1 250ms 2s 30s 0s 2m0s 2 0x47fb14 0x47fb48 0x47fae0 <nil> 0x47fa94 -1 8192    <nil> <nil> false}
[24277] 2018/10/21 17:08:49.575872 [DBG] 127.0.0.1:33652 - cid:3 - Client connection created
[24277] 2018/10/21 17:08:49.578806 [DBG] 127.0.0.1:33652 - cid:3 - Starting TLS client connection handshake
[24277] 2018/10/21 17:08:49.895962 [DBG] 127.0.0.1:33652 - cid:3 - TLS handshake complete
[24277] 2018/10/21 17:08:49.897352 [DBG] 127.0.0.1:33652 - cid:3 - TLS version 1.2, cipher suite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[24277] 2018/10/21 17:08:49.899811 [TRC] 127.0.0.1:33652 - cid:3 - ->> [CONNECT {"verbose":false,"pedantic":false,"user":"USER","pass":"[REDACTED]","tls_required":true,"name":"_NSS-stan-rpi-general","lang":"go","version":"1.6.0","protocol":1,"echo":true}]
[24277] 2018/10/21 17:08:49.902253 [TRC] 127.0.0.1:33652 - cid:3 - ->> [PING]
[24277] 2018/10/21 17:08:49.903585 [TRC] 127.0.0.1:33652 - cid:3 - <<- [PONG]
[24285] 2018/10/21 17:08:49.924074 [TRC] STREAM:  NATS conn opts: { [nats://USER:PASSWORD@127.0.0.1:60200] false false _NSS-stan-rpi-acks false false true 0x29a6460 true -1 250ms 2s 30s 0s 2m0s 2 0x47fb14 0x47fb48 0x47fae0 <nil> 0x47fa94 -1 8192    <nil> <nil> false}
[24277] 2018/10/21 17:08:49.928076 [DBG] 127.0.0.1:33654 - cid:4 - Client connection created
[24277] 2018/10/21 17:08:49.930858 [DBG] 127.0.0.1:33654 - cid:4 - Starting TLS client connection handshake
[24277] 2018/10/21 17:08:50.263903 [DBG] 127.0.0.1:33654 - cid:4 - TLS handshake complete
[24277] 2018/10/21 17:08:50.265367 [DBG] 127.0.0.1:33654 - cid:4 - TLS version 1.2, cipher suite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[24277] 2018/10/21 17:08:50.267762 [TRC] 127.0.0.1:33654 - cid:4 - ->> [CONNECT {"verbose":false,"pedantic":false,"user":"USER","pass":"[REDACTED]","tls_required":true,"name":"_NSS-stan-rpi-acks","lang":"go","version":"1.6.0","protocol":1,"echo":true}]
[24277] 2018/10/21 17:08:50.270303 [TRC] 127.0.0.1:33654 - cid:4 - ->> [PING]
[24277] 2018/10/21 17:08:50.271691 [TRC] 127.0.0.1:33654 - cid:4 - <<- [PONG]
[24285] 2018/10/21 17:08:50.298926 [TRC] STREAM:  NATS conn opts: { [nats://USER:PASSWORD@127.0.0.1:60200] false false _NSS-stan-rpi-raft false false true 0x29a6a80 true -1 250ms 2s 30s 0s 2m0s 2 0x47fb14 0x47fb48 0x47fae0 <nil> 0x47fa94 -1 8192    <nil> <nil> false}
[24277] 2018/10/21 17:08:50.305918 [DBG] 127.0.0.1:33658 - cid:5 - Client connection created
[24277] 2018/10/21 17:08:50.307898 [DBG] 127.0.0.1:33658 - cid:5 - Starting TLS client connection handshake
[24277] 2018/10/21 17:08:50.637940 [DBG] 127.0.0.1:33658 - cid:5 - TLS handshake complete
[24277] 2018/10/21 17:08:50.640521 [DBG] 127.0.0.1:33658 - cid:5 - TLS version 1.2, cipher suite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[24277] 2018/10/21 17:08:50.645111 [TRC] 127.0.0.1:33658 - cid:5 - ->> [CONNECT {"verbose":false,"pedantic":false,"user":"USER","pass":"[REDACTED]","tls_required":true,"name":"_NSS-stan-rpi-raft","lang":"go","version":"1.6.0","protocol":1,"echo":true}]
[24277] 2018/10/21 17:08:50.649105 [TRC] 127.0.0.1:33658 - cid:5 - ->> [PING]
[24277] 2018/10/21 17:08:50.649535 [TRC] 127.0.0.1:33658 - cid:5 - <<- [PONG]
[24277] 2018/10/21 17:08:50.662209 [TRC] 192.168.0.29:60101 - rid:1 - ->> [UNSUB RSID:5:18239]
[24277] 2018/10/21 17:08:50.662496 [TRC] 192.168.0.29:60101 - rid:1 - <-> [DELSUB RSID:5:18239]
[24285] 2018/10/21 17:08:50.670850 [TRC] STREAM:  NATS conn opts: { [nats://USER:PASSWORD@127.0.0.1:60200] false false _NSS-stan-rpi-raft_snap false false true 0x29a6ee0 true -1 250ms 2s 30s 0s 2m0s 2 0x47fb14 0x47fb48 0x47fae0 <nil> 0x47fa94 -1 8192    <nil> <nil> false}
[24277] 2018/10/21 17:08:50.677250 [DBG] 127.0.0.1:33660 - cid:6 - Client connection created
[24277] 2018/10/21 17:08:50.681141 [DBG] 127.0.0.1:33660 - cid:6 - Starting TLS client connection handshake
[24277] 2018/10/21 17:08:51.070054 [DBG] 127.0.0.1:33660 - cid:6 - TLS handshake complete
[24277] 2018/10/21 17:08:51.077950 [DBG] 127.0.0.1:33660 - cid:6 - TLS version 1.2, cipher suite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[24277] 2018/10/21 17:08:51.083267 [TRC] 127.0.0.1:33660 - cid:6 - ->> [CONNECT {"verbose":false,"pedantic":false,"user":"USER","pass":"[REDACTED]","tls_required":true,"name":"_NSS-stan-rpi-raft_snap","lang":"go","version":"1.6.0","protocol":1,"echo":true}]
[24277] 2018/10/21 17:08:51.087309 [TRC] 127.0.0.1:33660 - cid:6 - ->> [PING]
[24277] 2018/10/21 17:08:51.087740 [TRC] 127.0.0.1:33660 - cid:6 - <<- [PONG]
[24285] 2018/10/21 17:08:51.088807 [INF] STREAM: Recovering the state...
[24285] 2018/10/21 17:08:51.172518 [INF] STREAM: Recovered 7 channel(s)
[24285] 2018/10/21 17:08:51.175226 [INF] STREAM: Cluster Node ID : stan-node-rpi0w
[24285] 2018/10/21 17:08:51.177360 [INF] STREAM: Cluster Log Path: /home/meinside/files/stan/log
[24277] 2018/10/21 17:08:51.193723 [TRC] 127.0.0.1:33658 - cid:5 - ->> [SUB raft.stan-rpi.stan-node-rpi0w.stan-rpi.accept  1]
[24277] 2018/10/21 17:08:51.199354 [TRC] 192.168.0.29:60101 - rid:1 - <<- [SUB raft.stan-rpi.stan-node-rpi0w.stan-rpi.accept  RSID:5:1]
[24277] 2018/10/21 17:08:51.204195 [TRC] 127.0.0.1:33658 - cid:5 - ->> [PING]
[24277] 2018/10/21 17:08:51.206837 [TRC] 127.0.0.1:33658 - cid:5 - <<- [PONG]
[24285] 2018/10/21 17:08:51.247571 [INF] STREAM: raft: Initial configuration (index=11): [{Suffrage:Voter ID:stan-node-rpi3bplus Address:stan-rpi.stan-node-rpi3bplus.stan-rpi} {Suffrage:Voter ID:stan-node-rpi3b Address:stan-rpi.stan-node-rpi3b.stan-rpi} {Suffrage:Voter ID:stan-node-rpi0w Address:stan-rpi.stan-node-rpi0w.stan-rpi}]
[24285] 2018/10/21 17:08:51.252233 [DBG] STREAM: Loaded existing state for Raft group stan-rpi
[24285] 2018/10/21 17:08:51.253855 [DBG] STREAM: Discover subject:           _STAN.discover.stan-rpi
[24285] 2018/10/21 17:08:51.256755 [DBG] STREAM: Publish subject:            _STAN.pub.stan-rpi.>
[24285] 2018/10/21 17:08:51.257124 [DBG] STREAM: Subscribe subject:          _STAN.sub.stan-rpi
[24285] 2018/10/21 17:08:51.257406 [DBG] STREAM: Subscription Close subject: _STAN.subclose.stan-rpi
[24285] 2018/10/21 17:08:51.257775 [DBG] STREAM: Unsubscribe subject:        _STAN.unsub.stan-rpi
[24285] 2018/10/21 17:08:51.258057 [DBG] STREAM: Close subject:              _STAN.close.stan-rpi
[24277] 2018/10/21 17:08:51.258946 [TRC] 127.0.0.1:33652 - cid:3 - ->> [PING]
[24277] 2018/10/21 17:08:51.259175 [TRC] 127.0.0.1:33652 - cid:3 - <<- [PONG]
[24285] 2018/10/21 17:08:51.260788 [INF] STREAM: raft: Node at stan-rpi.stan-node-rpi0w.stan-rpi [Follower] entering Follower state (Leader: "")
[24277] 2018/10/21 17:08:51.265062 [TRC] 127.0.0.1:33658 - cid:5 - ->> [SUB _STAN.raft.stan-rpi.join  2]
[24277] 2018/10/21 17:08:51.266907 [TRC] 192.168.0.29:60101 - rid:1 - <<- [SUB _STAN.raft.stan-rpi.join  RSID:5:2]
[24285] 2018/10/21 17:08:51.268440 [INF] STREAM: Message store is RAFT_FILE
[24285] 2018/10/21 17:08:51.268837 [INF] STREAM: Store location: /home/meinside/files/stan/datastore
[24285] 2018/10/21 17:08:51.272620 [INF] STREAM: ---------- Store Limits ----------
[24285] 2018/10/21 17:08:51.273007 [INF] STREAM: Channels:                  100 *
[24285] 2018/10/21 17:08:51.273273 [INF] STREAM: --------- Channels Limits --------
[24285] 2018/10/21 17:08:51.273532 [INF] STREAM:   Subscriptions:           100
[24285] 2018/10/21 17:08:51.273784 [INF] STREAM:   Messages     :          1000
[24285] 2018/10/21 17:08:51.274157 [INF] STREAM:   Bytes        :      10.00 MB
[24285] 2018/10/21 17:08:51.274415 [INF] STREAM:   Age          :        6h0m0s
[24285] 2018/10/21 17:08:51.274668 [INF] STREAM:   Inactivity   :        6h0m0s
[24285] 2018/10/21 17:08:51.274920 [INF] STREAM: ----------------------------------
[24277] 2018/10/21 17:08:51.313203 [TRC] 192.168.0.29:60101 - rid:1 - ->> [UNSUB RSID:5:18240]
[24277] 2018/10/21 17:08:51.313549 [TRC] 192.168.0.29:60101 - rid:1 - <-> [DELSUB RSID:5:18240]
[24285] 2018/10/21 17:08:54.556152 [INF] STREAM: raft: Heartbeat timeout from "" reached, starting election
[24285] 2018/10/21 17:08:54.556650 [INF] STREAM: raft: Node at stan-rpi.stan-node-rpi0w.stan-rpi [Candidate] entering Candidate state in term 62
[24277] 2018/10/21 17:08:54.575017 [TRC] 127.0.0.1:33658 - cid:5 - ->> [SUB raft.stan-rpi.stan-node-rpi0w.stan-rpi.request._INBOX.5kgGsaKFPIUp91OuQ3IUyJ  3]
[24277] 2018/10/21 17:08:54.576844 [TRC] 192.168.0.29:60101 - rid:1 - <<- [SUB raft.stan-rpi.stan-node-rpi0w.stan-rpi.request._INBOX.5kgGsaKFPIUp91OuQ3IUyJ  RSID:5:3]
[24277] 2018/10/21 17:08:54.579417 [TRC] 127.0.0.1:33658 - cid:5 - ->> [PING]
[24277] 2018/10/21 17:08:54.591320 [TRC] 127.0.0.1:33658 - cid:5 - <<- [PONG]
[24277] 2018/10/21 17:08:54.598227 [TRC] 127.0.0.1:33658 - cid:5 - ->> [SUB raft.stan-rpi.stan-node-rpi0w.stan-rpi.request._INBOX.5kgGsaKFPIUp91OuQ3IV0L  4]
[24277] 2018/10/21 17:08:54.606819 [TRC] 192.168.0.29:60101 - rid:1 - <<- [SUB raft.stan-rpi.stan-node-rpi0w.stan-rpi.request._INBOX.5kgGsaKFPIUp91OuQ3IV0L  RSID:5:4]
[24277] 2018/10/21 17:08:54.610038 [TRC] 127.0.0.1:33658 - cid:5 - ->> [PING]
[24277] 2018/10/21 17:08:54.611283 [TRC] 127.0.0.1:33658 - cid:5 - <<- [PONG]
[24277] 2018/10/21 17:08:54.619459 [TRC] 127.0.0.1:33658 - cid:5 - ->> [SUB _INBOX.5kgGsaKFPIUp91OuQ3IV2N.*  5]
[24277] 2018/10/21 17:08:54.623308 [TRC] 192.168.0.29:60101 - rid:1 - <<- [SUB _INBOX.5kgGsaKFPIUp91OuQ3IV2N.*  RSID:5:5]
[24277] 2018/10/21 17:08:54.624193 [TRC] 127.0.0.1:33658 - cid:5 - ->> [PUB raft.stan-rpi.stan-node-rpi3b.stan-rpi.accept _INBOX.5kgGsaKFPIUp91OuQ3IV2N.5kgGsaKFPIUp91OuQ3IV4P 129]
[24277] 2018/10/21 17:08:54.625968 [TRC] 127.0.0.1:33658 - cid:5 - ->> MSG_PAYLOAD: [{"id":"stan-rpi.stan-node-rpi0w.stan-rpi","inbox":"raft.stan-rpi.stan-node-rpi0w.stan-rpi.request._INBOX.5kgGsaKFPIUp91OuQ3IUyJ"}]
[24277] 2018/10/21 17:08:54.627062 [TRC] 192.168.0.29:60101 - rid:1 - <<- [MSG raft.stan-rpi.stan-node-rpi3b.stan-rpi.accept RSID:9:1 _INBOX.5kgGsaKFPIUp91OuQ3IV2N.5kgGsaKFPIUp91OuQ3IV4P 129]
[24277] 2018/10/21 17:08:54.629448 [TRC] 127.0.0.1:33658 - cid:5 - ->> [PUB raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.accept _INBOX.5kgGsaKFPIUp91OuQ3IV2N.5kgGsaKFPIUp91OuQ3IV6R 129]
[24277] 2018/10/21 17:08:54.632016 [TRC] 127.0.0.1:33658 - cid:5 - ->> MSG_PAYLOAD: [{"id":"stan-rpi.stan-node-rpi0w.stan-rpi","inbox":"raft.stan-rpi.stan-node-rpi0w.stan-rpi.request._INBOX.5kgGsaKFPIUp91OuQ3IV0L"}]
[24285] 2018/10/21 17:08:54.633780 [TRC] STREAM: raft: Votes needed: 2
[24285] 2018/10/21 17:08:54.634541 [TRC] STREAM: raft: Vote granted from stan-node-rpi0w in term 62. Tally: 1
[24277] 2018/10/21 17:08:54.635582 [TRC] 192.168.0.29:60101 - rid:1 - <<- [MSG raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.accept RSID:5:1 _INBOX.5kgGsaKFPIUp91OuQ3IV2N.5kgGsaKFPIUp91OuQ3IV6R 129]
[24277] 2018/10/21 17:08:54.640196 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.request._INBOX.hRWNeoO963tW1kCEHqAdi2  RSID:5:18241]
[24277] 2018/10/21 17:08:54.644480 [TRC] 192.168.0.29:60101 - rid:1 - ->> [MSG _INBOX.5kgGsaKFPIUp91OuQ3IV2N.5kgGsaKFPIUp91OuQ3IV6R RSID:5:5 92]
[24277] 2018/10/21 17:08:54.645869 [TRC] 192.168.0.29:60101 - rid:1 - ->> MSG_PAYLOAD: [{"inbox":"raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.request._INBOX.hRWNeoO963tW1kCEHqAdi2"}]
[24277] 2018/10/21 17:08:54.646936 [TRC] 127.0.0.1:33658 - cid:5 - <<- [MSG _INBOX.5kgGsaKFPIUp91OuQ3IV2N.5kgGsaKFPIUp91OuQ3IV6R 5 92]
[24277] 2018/10/21 17:08:54.648152 [TRC] 192.168.0.29:60101 - rid:1 - ->> [SUB raft.stan-rpi.stan-node-rpi3b.stan-rpi.request._INBOX.izXIYT7IPQAz3HaDWWUH70  RSID:9:11655]
[24277] 2018/10/21 17:08:54.652480 [TRC] 192.168.0.29:60101 - rid:1 - ->> [MSG _INBOX.5kgGsaKFPIUp91OuQ3IV2N.5kgGsaKFPIUp91OuQ3IV4P RSID:5:5 88]
[24277] 2018/10/21 17:08:54.651626 [TRC] 127.0.0.1:33658 - cid:5 - ->> [PUB raft.stan-rpi.stan-node-rpi3bplus.stan-rpi.request._INBOX.hRWNeoO963tW1kCEHqAdi2 99]
[24277] 2018/10/21 17:08:54.653346 [TRC] 192.168.0.29:60101 - rid:1 - ->> MSG_PAYLOAD: [{"inbox":"raft.stan-rpi.stan-node-rpi3b.stan-rpi.request._INBOX.izXIYT7IPQAz3HaDWWUH70"}]
[24277] 2018/10/21 17:08:54.655060 [TRC] 127.0.0.1:33658 - cid:5 - <<- [MSG _INBOX.5kgGsaKFPIUp91OuQ3IV2N.5kgGsaKFPIUp91OuQ3IV4P 5 88]
...
...

This is the file:

nats.log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment