Detailed report for two nodes stuck on "Monitoring failed" for several days, despite extensive troubleshooting. Posting full findings in case it helps diagnose a real backend/protocol issue.
Affected nodes (both Oracle Cloud, Amsterdam, hosting-type IP):
- 0x56ea62be9eb70aab5f5b4adec661b1c35cd7ab8e — 141.148.244.209 (Oracle Corporation, ASN 31898)
- 0x9b4b7b364560b6ce1da06f0e9279c54d3df2a0f1 — 158.178.150.244 (Oracle Svenska AB, ASN 31898)
Control node (same account, same Oracle Cloud tenancy, working normally, "Online", earning):
- 0xd5bfe74f0425ff41eaae34227e8b217dc31c8df3 — 141.148.235.39 (Oracle Corporation, ASN 31898)
All three instances are in the SAME Oracle Cloud tenancy and VCN family, with identical Default Security Lists (all protocols, all ports, both ingress and egress, fully open — verified directly in the OCI console). So this is not an Oracle Cloud network/firewall issue.
What we ruled out, in order:
-
Stale iptables MYST chain from an earlier unclean restart ("Chain already exists") — found and fixed on node 1. Did not resolve the monitoring status.
-
Host-level iptables misconfiguration/duplicates — audited and cleaned on node 1 (removed duplicate rules, reordered, persisted via netfilter-persistent). Did not resolve the monitoring status.
-
Quality oracle rate-limiting — confirmed directly via curl:
curl -v -X POST https://quality.mysterium.network/api/v3/providers/monitoring-status
< HTTP/2 429
< x-rate-limit-limit: 100
< x-rate-limit-remaining: 0
Too many requests...
This explained the "invalid character 'T' looking for beginning of value" JSON parse errors in the node logs at the time (the node's JSON parser chokes on the plain-text 429 body). This was self-inflicted by repeated troubleshooting restarts and has since cleared (confirmed via repeat curl, now returns 400 instead of 429, and the errors have stopped appearing in journalctl). However, "Monitoring failed" persisted for days afterward, so this was not the (or not the only) root cause.
-
WireGuard tunnel creation — verified via journalctl on both affected nodes: ip link add dev myst0 type wireguard, ip link set dev myst0 up, ip address replace dev myst0 10.182.0.1/24 all succeed with no errors. The wireguard kernel module is loaded correctly (lsmod | grep wireguard shows the module and its dependencies present).
-
SNAT/MASQUERADE for the tunnel subnet — verified via journalctl: the node correctly adds and removes a SNAT rule for each session (iptables -A POSTROUTING --source 10.182.0.2/24 ... --jump SNAT --to <host-ip>), matching the working node's behavior exactly.
-
Oracle Cloud Security Lists — verified directly in the OCI console for all three instances. Identical, fully open (all protocols/ports, ingress and egress).
What we observed:
On both affected nodes, WireGuard sessions consistently last 1–3 seconds and transfer ~944 bytes (matching a handshake, no real payload), then get cleaned up:
Accepting new WireGuard connection
[ip link add / set up / address replace — all succeed]
[~1-2 seconds later]
Cleaning up session <id>
ip link del dev myst0
No errors are logged anywhere in this sequence on either affected node. This happens roughly every 45 minutes to 1 hour (i.e., relatively infrequent connection attempts from consumers), and every single attempt fails the same way.
We don't have a way to inspect what happens on the wire during that 1–2 second window (would need tcpdump timed exactly to a connection attempt, which — given attempts are ~hourly — isn't practical for us to keep monitoring for it in this session).
Given identical Security Lists, identical host OS/network stack behavior in the logs, and only the node identity/IP differing, we suspect either:
- something specific to how these two identities are scored/routed by the P2P broker or quality oracle, or
- a subtle WireGuard endpoint negotiation issue that only manifests for these providers (MTU, keepalive timing, or similar), or
- packet loss on the path between these specific consumers and these specific nodes that we can't diagnose from our side.
Happy to run further diagnostics (tcpdump during a live session, packet captures, etc.) if someone from the team can point us at what's most useful to check next.
Detailed report for two nodes stuck on "Monitoring failed" for several days, despite extensive troubleshooting. Posting full findings in case it helps diagnose a real backend/protocol issue.
Affected nodes (both Oracle Cloud, Amsterdam, hosting-type IP):
Control node (same account, same Oracle Cloud tenancy, working normally, "Online", earning):
All three instances are in the SAME Oracle Cloud tenancy and VCN family, with identical Default Security Lists (all protocols, all ports, both ingress and egress, fully open — verified directly in the OCI console). So this is not an Oracle Cloud network/firewall issue.
What we ruled out, in order:
Stale iptables MYST chain from an earlier unclean restart ("Chain already exists") — found and fixed on node 1. Did not resolve the monitoring status.
Host-level iptables misconfiguration/duplicates — audited and cleaned on node 1 (removed duplicate rules, reordered, persisted via netfilter-persistent). Did not resolve the monitoring status.
Quality oracle rate-limiting — confirmed directly via curl:
This explained the "invalid character 'T' looking for beginning of value" JSON parse errors in the node logs at the time (the node's JSON parser chokes on the plain-text 429 body). This was self-inflicted by repeated troubleshooting restarts and has since cleared (confirmed via repeat curl, now returns 400 instead of 429, and the errors have stopped appearing in journalctl). However, "Monitoring failed" persisted for days afterward, so this was not the (or not the only) root cause.
WireGuard tunnel creation — verified via journalctl on both affected nodes:
ip link add dev myst0 type wireguard,ip link set dev myst0 up,ip address replace dev myst0 10.182.0.1/24all succeed with no errors. The wireguard kernel module is loaded correctly (lsmod | grep wireguardshows the module and its dependencies present).SNAT/MASQUERADE for the tunnel subnet — verified via journalctl: the node correctly adds and removes a SNAT rule for each session (
iptables -A POSTROUTING --source 10.182.0.2/24 ... --jump SNAT --to <host-ip>), matching the working node's behavior exactly.Oracle Cloud Security Lists — verified directly in the OCI console for all three instances. Identical, fully open (all protocols/ports, ingress and egress).
What we observed:
On both affected nodes, WireGuard sessions consistently last 1–3 seconds and transfer ~944 bytes (matching a handshake, no real payload), then get cleaned up:
No errors are logged anywhere in this sequence on either affected node. This happens roughly every 45 minutes to 1 hour (i.e., relatively infrequent connection attempts from consumers), and every single attempt fails the same way.
We don't have a way to inspect what happens on the wire during that 1–2 second window (would need tcpdump timed exactly to a connection attempt, which — given attempts are ~hourly — isn't practical for us to keep monitoring for it in this session).
Given identical Security Lists, identical host OS/network stack behavior in the logs, and only the node identity/IP differing, we suspect either:
Happy to run further diagnostics (tcpdump during a live session, packet captures, etc.) if someone from the team can point us at what's most useful to check next.