Could not send status to leader on Raspi

[CableMate]

Hi,
I'm working on a Raspi 4 with 2.0.0-alpha.6.
The node service is running, I can login and created a leader secret for the tap.
But the tap-service is not able to send its data to the leader: journalctl -xe reports

nzyme-tap[84942]: [ERROR][2024-01-10 12:47:36][link::leaderlink] Could not send status to leader. error sending request for url (https://localhost:22900/api/taps/status): connection closed before message completed

Trying to test this URI in Chrome supplies a "500", which means, the URI does react.

My nzyme-tap.conf:

[general]
leader_secret = "mysecret"
leader_uri = "https://127.0.0.1:22900/"
accept_insecure_certs = true

[ethernet_interfaces.enp6s0]
active = false

[wifi_interfaces.wlan1]
active = true
channels_2g = [1,2, 3, 4, 5, 7, 8, 9, 10, 12, 13]
channels_5g = []
channels_6g = []
.
.
.

What could be the reason?

Regards,

CableMate

[lennartkoopmann]

Hi,

can you post your nzyme-node logs as well? Can you also check if you have a very high system load? Something is aborting the connection between your tap and node. One reason could be a hard timeout kicking in.

Thank you!

[CableMate]

Hi,

I attached nzyme.log after a new start of nzyme.log.

The htop shows a load avareage of 2.44, which means no overload, I think.

I can access the login page in the local browser, which is the same port.

Regards.

[lennartkoopmann]

and in that state you cannot login from another machine, but only localhost?

[CableMate]

I cannot login from another machine, only from localhost.
But this is clear, because the port 29600 is only listening on IP 127.0.0.1.
I cannot understand why nzyme-tap is not able to connect to this port, although it runs on the same device and the local browser can connect. nzyme-tap is in the same scope as the browser:

[lennartkoopmann]

Can you post the nzyme-tap log as well?

[CableMate]

I'am able to send you the complete output of "jourjournalctl -xe" tomorrow.
BTW: Is there a file like /var/log/nzyme.log for nzyme-tap?

[lennartkoopmann]

There is no log file currently, but it may be added in the future. The node process will start to write into the journal soon and have an optional log file as well.

[CableMate]

I attached the output of "journalctl -xef" after starting nzyme-tap:
nzyme-tap-start.log
and
nzyme.log

[lennartkoopmann]

This clearly looks like the tap is not able to send its recorded information to the node. You can see the timeout in the tap log and the reset connection in the node log.

A few reasons I can think of:

1. There is some sort of firewall, even though it's on localhost
2. System is overloaded and not able to transmit before the timeout hits
3. Your tap is somehow recording enormous amounts of data that can't be sent before the timeout hits
4. TLS handshake/negotiation takes too long

The timeout is 10 seconds.

You say that you can connect to tcp/12900 on localhost so it's unlikely to be 1). The screenshot shows healthy system usage and it's unlikely to be 2).

To debug 3), can you try to start the tap without any captures and see if it shows up in the list of taps in the web interface? Remove all wifi_interfaces.* and ethernet_interfaces.* to disable all captures.

Do you do ethernet capturing? What type of WiFi adapters are you using?

For 4), do you have any custom TLS configurations anywhere? Did you configure your own certs in nzyme?

[CableMate]

No, I don't ethernet capturing.
I use no custom TLS-configuration (accept_insecure_certs = true).

see if it shows up in the list of taps in the web interface.

New situation:

Inspite of many messages "Could not send status to leader" I see

and

Inspite of I modified nzyme-tap.conf that no interface is used (active=false):

[general]
leader_secret = "qjSdCdMfLQtTOYCJdun1zAUWewFnJ1nhwwlvbpCx7scG8yJqhFR5iXCihQ5o69q7"
leader_uri = "https://127.0.0.1:22900/"
accept_insecure_certs = true

[ethernet_interfaces.enp6s0]
active = false

[wifi_interfaces.wlan1]
active = false
channels_2g = [1,2, 3, 4, 5, 7, 8, 9, 10, 12, 13]
channels_5g = []
channels_6g = []

[wifi_interfaces.wlan2]
active = false
#channels_2g = [2, 3, 4, 5, 7, 8, 9, 10, 12, 13]
channels_2g = []
channels_5g = [100, 104, 108, 112, 136, 140]
#channels_5g = [36, 40, 44, 48, 52, 56, 60, 64, 173]
#channels_6g = [5, 9, 13, 17, 21, 25, 29, 109, 113, 153, 157, 161, 165, 169, 173, 177, 181, 185, 189, 229, 233]
channels_6g = []

[performance]
ethernet_brokers = 1
wifi_brokers = 1
wifi_broker_buffer_capacity = 65535
ethernet_broker_buffer_capacity = 65535

[misc]
training_period_minutes = 5

and started new.

The connected WLAN adapter are 2 AWUS036ACM

journalctl -xef stll shows "Could not send status to leader. error sending request for url (https://127.0.0.1:22900/api/taps/status): operation timed out":
nzyme-tap.log

Activcating wifi 1 again All Taps shows 19% CPU.

htop shows avagarage load 2.46:

I assume, that the timeout messages have not a really meaning except having an irritating effect, nzyme-tap is really running.

[lennartkoopmann]

The timeout message is coming from the operating system and means that the socket was closed because the operation took too long.

The htop screenshot also shows that you have a very high memory usage and are almost entirely out of swap space. That would explain the timeouts you are seeing. Some requests are probably make it through, leading to an active tap status.

Looks like you are running the tap as well as the node on a system that also has other things running on it and you are overloaded.

[CableMate]

It should be possible to increase the swapspace. I'll have look how to do this. But a 8GB version of the Raspi is a better ideas of course.
I observed also, that the node sever is also sometimes not responding. Espacially when Teamviewer is running. Teamviewer is using realative much resources in comparison with VNC.

I planned to try to install it next on a Intel NUC under Mint, but I cannot update to Mint 21.3 due to dependecies with old Hardware and their not more maintained drivers. Under Mint 20 libc6 should not updated to 2.34.

Thank you for your support so far!