err: sendto: destination address required #5
Comments
|
So I'm seeing shutdown logs in there too, not sure if that was you or from an error. Those multiple failed heartbeats are a little more concerning - unless you had shut down the other side of the connection. |
|
Gotcha. Here's an asciinema video (top panel is my desktop (server), the 2 other panels are my laptop (client)) https://asciinema.org/a/uuKQNDtPIdaguQ1WSAdTMmse3 Note that I'm already using both tailscale and wireguard on both computers. |
|
Can take a look in a little bit and see whats going on. One other thing to check - are there overlapping address assignments between Tailscale, your devices, and webmesh? Webmesh side you can configure the internal IPv4 prefix at bootstrap - the IPv6 one is randomly generated. One way to rule that out would be to try running it with ipv4 disabled. |
|
Here's another recording with https://asciinema.org/a/Q26LZhOZCCHVIxUb0CgJoWvjT Note that I'm using webmesh with tailscale (100.85.215.110 is a tailscale ip). |
|
Is there something like |
|
It works if I connect using lan ips instead of tailscale. EDIT: but it doesn't if I try to connect to my vps from my desktop. Only my vps's ports are open. |
|
To the At least one node needs to accessible currently for peerings to work. But I am currently working on other methods of discovery. |
In my tests, at least one node was accessible. I'll do more test, though. |
|
I hope to be able to look more closely tonight. One more thing that you can try (and is a really easy way to shoot yourself in the foot that I need to document better) - is that given this is a sort of "zero-trust" solution, the default behavior only lets people peer up if there is a Network ACL allowing it. Those are managed via the admin API, but you can set a default allow-all rule at bootstrap with |
|
Here's a nixpkgs test. It works only with the I'll test again with my vps. Don't hesitate if you want to know how to run this test with nixpkgs. import ./make-test-python.nix ({ pkgs, ... }: {
name = "webmesh";
meta.maintainers = with pkgs.lib.maintainers; [ bbigras ];
nodes = {
server = {
networking = {
interfaces.eth1 = {
ipv4.addresses = [
{ address = "192.168.2.101"; prefixLength = 24; }
];
};
firewall = {
trustedInterfaces = [ "webmesh0" ];
allowedTCPPorts = [
8443
9443
];
allowedUDPPorts = [
51820
];
};
};
systemd.services.webmesh = {
after = ["network-online.target"];
wantedBy = [ "multi-user.target" ];
script = ''
${pkgs.webmesh}/bin/webmesh-node \
--global.insecure \
--global.no-ipv6 \
--global.detect-endpoints \
--global.detect-private-endpoints \
--bootstrap.enabled \
--bootstrap.default-network-policy=accept \
--global.primary-endpoint 192.168.2.101
'';
};
};
client = {
networking = {
interfaces.eth1 = {
ipv4.addresses = [
{ address = "192.168.2.102"; prefixLength = 24; }
];
};
firewall = {
trustedInterfaces = [ "webmesh0" ];
};
};
systemd.services.wmctl = {
after = ["network-online.target"];
wantedBy = [ "multi-user.target" ];
script = ''
${pkgs.webmesh}/bin/wmctl \
connect --insecure --no-ipv6 --join-server=192.168.2.101:8443
'';
};
};
};
testScript =
''
server.start()
server.wait_for_unit("webmesh.service")
server.wait_for_open_port(8443)
client.start()
client.wait_for_unit("wmctl.service")
client.wait_for_open_port(9443)
client.succeed("ping -c1 172.16.0.1")
client.sleep(120)
client.succeed("ping -c1 172.16.0.1")
'';
}) |
|
I'm gonna try to replicate it locally - and if I fail may reach out for more infoz |
|
Also note that I don't have to use |
|
I think I've found an issue - not sure if it is related to yours. But in a similar setup the second client keeps dropping the peer. I'll let you know what I figure out. |
|
Just to give you a quick update. I'm trying to hammer out one last bug and then I'll tag a new release. I think most related to your issue was in the storage update trigger on each node. It was doing some hacky logic to make sure it was fully up to date - and a recent fix had made that no longer necessary. I think that old hack was causing peer refreshes not to happen at the right times. I'm having another issue where peer refreshes are sometimes returning the wrong internal IPs to be set to the allowed IPs. I'm not sure if this is something you are experiencing - but I hope to figure it out before I push this other fix out. |
|
I'm still working on the second issue - but if you are able to build from |
|
I still need |
|
That is likely unrelated. Detection is non-deterministic. So if you want a specific one to be tagged as the primary - you have to specify. That being said - I can try to look into it more. |
|
Worth noting there are also the |
|
I'm testing with my desktop to my vps and it seems to stay connected now. 🎉 I don't mind using |
|
Sweet - I fixed my other bug so about to tag a new release. Will close this, but feel free to open a new issue if anything arises. |
I'm following the https://webmeshproj.github.io/guides/personal-vpn/ guide. I'm able to connect and some ping works, but I get this on the server side:
{"time":"2023-08-07T02:02:13.95968374-04:00","level":"INFO","msg":"starting webmesh node","version":"unknown","commit":"unknown","buildDate":"unknown"} {"time":"2023-08-07T02:02:13.976051984-04:00","level":"INFO","msg":"using CN as node ID","component":"mesh","node-id":"server"} {"time":"2023-08-07T02:02:13.976094557-04:00","level":"INFO","msg":"loading plugin","name":"mtls"} {"time":"2023-08-07T02:02:14.010115941-04:00","level":"INFO","msg":"All 0 tables opened in 0s","component":"raft","storage":"/var/lib/webmesh/store","component":"raftbadger"} {"time":"2023-08-07T02:02:14.017333494-04:00","level":"INFO","msg":"Discard stats nextEmptySlot: 0","component":"raft","storage":"/var/lib/webmesh/store","component":"raftbadger"} {"time":"2023-08-07T02:02:14.017356434-04:00","level":"INFO","msg":"Set nextTxnTs to 0","component":"raft","storage":"/var/lib/webmesh/store","component":"raftbadger"} {"time":"2023-08-07T02:02:14.076359285-04:00","level":"INFO","msg":"All 0 tables opened in 0s","component":"badger"} {"time":"2023-08-07T02:02:14.090821071-04:00","level":"INFO","msg":"Discard stats nextEmptySlot: 0","component":"badger"} {"time":"2023-08-07T02:02:14.090842809-04:00","level":"INFO","msg":"Set nextTxnTs to 0","component":"badger"} {"time":"2023-08-07T02:02:14.091883921-04:00","level":"INFO","msg":"starting raft instance","component":"raft","storage":"/var/lib/webmesh/store","listen-addr":"[::]:9443"} {"time":"2023-08-07T02:02:14.091997004-04:00","level":"INFO","msg":"initial configuration","component":"raft","index":0,"servers":["%+v",null]} {"time":"2023-08-07T02:02:14.092069274-04:00","level":"INFO","msg":"bootstrapping cluster","component":"mesh","node-id":"server"} {"time":"2023-08-07T02:02:14.092073906-04:00","level":"INFO","msg":"entering follower state","component":"raft","follower":{},"leader-address":"","leader-id":""} {"time":"2023-08-07T02:02:18.538521544-04:00","level":"WARN","msg":"heartbeat timeout reached, starting election","component":"raft","last-leader-addr":"","last-leader-id":""} {"time":"2023-08-07T02:02:18.53858546-04:00","level":"INFO","msg":"entering candidate state","component":"raft","node":{},"term":2} {"time":"2023-08-07T02:02:18.538669733-04:00","level":"INFO","msg":"election won","component":"raft","term":2,"tally":1} {"time":"2023-08-07T02:02:18.538681113-04:00","level":"INFO","msg":"entering leader state","component":"raft","leader":{}} {"time":"2023-08-07T02:02:18.597710471-04:00","level":"INFO","msg":"newly bootstrapped cluster, setting IPv4/IPv6 networks","component":"mesh","node-id":"server","ipv4-network":"172.16.0.0/12","ipv6-network":"fda1:420:b42e::/48"} {"time":"2023-08-07T02:02:18.598660537-04:00","level":"INFO","msg":"registering ourselves as a node in the cluster","component":"mesh","node-id":"server","server-id":"server"} {"time":"2023-08-07T02:02:18.598683097-04:00","level":"INFO","msg":"generating wireguard key for ourselves","component":"mesh","node-id":"server"} {"time":"2023-08-07T02:02:18.598689159-04:00","level":"INFO","msg":"generating new wireguard key","component":"mesh","node-id":"server"} {"time":"2023-08-07T02:02:18.598998931-04:00","level":"INFO","msg":"starting network manager","component":"mesh","node-id":"server"} {"time":"2023-08-07T02:02:18.599013156-04:00","level":"INFO","msg":"Configuring firewall","component":"net-manager","opts":{"ID":"server","DefaultPolicy":"accept","WireguardPort":51821,"RaftPort":9443,"GRPCPort":8443}} {"time":"2023-08-07T02:02:18.643151554-04:00","level":"INFO","msg":"Configuring wireguard","component":"net-manager","opts":{"NodeID":"server","ListenPort":51821,"Name":"","ForceName":false,"ForceTUN":false,"PersistentKeepAlive":0,"MTU":1350,"AddressV4":"172.16.0.1/32","AddressV6":"fda1:420:b42e:442::/64","Metrics":false,"MetricsInterval":15000000000,"DisableIPv4":false,"DisableIPv6":false}} {"time":"2023-08-07T02:02:18.643335776-04:00","level":"INFO","msg":"creating wireguard interface","component":"wireguard","name":"webmesh0"} {"time":"2023-08-07T02:02:18.657321514-04:00","level":"INFO","msg":"re-adding ourselves to the cluster with the acquired wireguard address","component":"mesh","node-id":"server"} {"time":"2023-08-07T02:02:18.657365768-04:00","level":"INFO","msg":"updating configuration","component":"raft","command":0,"server-id":"server","server-addr":"172.16.0.1:9443","servers":["%+v",[{"Suffrage":0,"ID":"server","Address":"172.16.0.1:9443"}]]} {"time":"2023-08-07T02:02:18.657447005-04:00","level":"INFO","msg":"initial bootstrap complete","component":"mesh","node-id":"server"} {"time":"2023-08-07T02:02:18.657459953-04:00","level":"INFO","msg":"mesh connection is ready, starting services"} {"time":"2023-08-07T02:02:18.657884285-04:00","level":"INFO","msg":"Starting gRPC server on [::]:8443","component":"server"} {"time":"2023-08-07T02:02:38.015101241-04:00","level":"INFO","msg":"started call","component":"server","protocol":"grpc","grpc.component":"server","grpc.service":"v1.Node","grpc.method":"Join","grpc.method_type":"unary","peer.address":"100.127.110.11:36454","grpc.start_time":"2023-08-07T02:02:38-04:00","grpc.time_ms":"0.01"} {"time":"2023-08-07T02:02:38.015353545-04:00","level":"INFO","msg":"join request received","component":"node-server","op":"join","id":"admin","request":{"id":"admin","public_key":"sDNLid5iqiuIrFJ4sCYoUgaE4Xlbdy2taZazqYdZBF0=","raft_port":9443,"grpc_port":8443,"assign_ipv4":true}} {"time":"2023-08-07T02:02:38.016100312-04:00","level":"INFO","msg":"adding non-voter to cluster","component":"node-server","op":"join","id":"admin","raft_address":"172.16.0.2:9443"} {"time":"2023-08-07T02:02:38.01612437-04:00","level":"INFO","msg":"updating configuration","component":"raft","command":1,"server-id":"admin","server-addr":"172.16.0.2:9443","servers":["%+v",[{"Suffrage":0,"ID":"server","Address":"172.16.0.1:9443"},{"Suffrage":1,"ID":"admin","Address":"172.16.0.2:9443"}]]} {"time":"2023-08-07T02:02:38.016176729-04:00","level":"INFO","msg":"added peer, starting replication","component":"raft","peer":"admin"} {"time":"2023-08-07T02:02:38.016396001-04:00","level":"ERROR","msg":"failed to appendEntries to","component":"raft","peer":{"Suffrage":1,"ID":"admin","Address":"172.16.0.2:9443"},"error":"dial tcp 172.16.0.2:9443: connect: no route to host"} {"time":"2023-08-07T02:02:38.016547253-04:00","level":"INFO","msg":"sending join response","component":"node-server","op":"join","id":"admin","response":{"address_ipv4":"172.16.0.2/32","address_ipv6":"fda1:420:b42e:ceea::/64","network_ipv4":"172.16.0.0/12","network_ipv6":"fda1:420:b42e::/48","peers":[{"id":"server","public_key":"5sdQXSMQGUXWzj7Ri8ZyYPoom2++4ISdWk+9sWSZDjg=","primary_endpoint":"100.85.215.110:51821","wireguard_endpoints":["100.85.215.110:51821"],"address_ipv4":"172.16.0.1/32","address_ipv6":"fda1:420:b42e:442::/64","allowed_ips":["172.16.0.1/32","fda1:420:b42e:442::/64"]}],"mesh_domain":"webmesh.internal."}} {"time":"2023-08-07T02:02:38.016636105-04:00","level":"INFO","msg":"finished call","component":"server","protocol":"grpc","grpc.component":"server","grpc.service":"v1.Node","grpc.method":"Join","grpc.method_type":"unary","peer.address":"100.127.110.11:36454","grpc.start_time":"2023-08-07T02:02:38-04:00","grpc.code":"OK","grpc.time_ms":"1.548"} {"time":"2023-08-07T02:02:38.017910703-04:00","level":"WARN","msg":"could not ping descendant","component":"net-manager","descendant":"admin","error":"run pinger: write ip4 0.0.0.0->172.16.0.2: sendto: destination address required"} {"time":"2023-08-07T02:02:38.018289245-04:00","level":"WARN","msg":"could not ping descendant","component":"net-manager","descendant":"admin","error":"run pinger: write ip4 0.0.0.0->172.16.0.2: sendto: destination address required"} {"time":"2023-08-07T02:02:41.339334625-04:00","level":"ERROR","msg":"failed to heartbeat to","component":"raft","peer":"172.16.0.2:9443","backoff time":10000000,"error":"dial tcp 172.16.0.2:9443: i/o timeout"} {"time":"2023-08-07T02:02:44.949781283-04:00","level":"ERROR","msg":"failed to heartbeat to","component":"raft","peer":"172.16.0.2:9443","backoff time":10000000,"error":"dial tcp 172.16.0.2:9443: i/o timeout"} {"time":"2023-08-07T02:02:48.366660528-04:00","level":"ERROR","msg":"failed to heartbeat to","component":"raft","peer":"172.16.0.2:9443","backoff time":10000000,"error":"dial tcp 172.16.0.2:9443: i/o timeout"} {"time":"2023-08-07T02:02:51.944757019-04:00","level":"ERROR","msg":"failed to heartbeat to","component":"raft","peer":"172.16.0.2:9443","backoff time":20000000,"error":"dial tcp 172.16.0.2:9443: i/o timeout"} {"time":"2023-08-07T02:02:55.425564585-04:00","level":"ERROR","msg":"failed to heartbeat to","component":"raft","peer":"172.16.0.2:9443","backoff time":40000000,"error":"dial tcp 172.16.0.2:9443: i/o timeout"} {"time":"2023-08-07T02:02:55.741667558-04:00","level":"INFO","msg":"shutting down gRPC server"} {"time":"2023-08-07T02:02:55.741742112-04:00","level":"INFO","msg":"Shutting down gRPC server","component":"server"} {"time":"2023-08-07T02:02:55.741822634-04:00","level":"INFO","msg":"shutting down mesh connection"} {"time":"2023-08-07T02:02:55.741864178-04:00","level":"INFO","msg":"creating new db snapshot","component":"snapshots"} {"time":"2023-08-07T02:02:55.741992909-04:00","level":"INFO","msg":"Number of ranges found: 2","component":"badger"} {"time":"2023-08-07T02:02:55.742081389-04:00","level":"INFO","msg":"DB.Backup Streaming about 0 B of uncompressed data (0 B on disk)","component":"badger"} {"time":"2023-08-07T02:02:55.742121461-04:00","level":"INFO","msg":"Sent range 0 for iteration: [, 2f72656769737472792f65646765732f61646d696e2f736572766572fffffffffffffff1) of size: 0 B","component":"badger"} {"time":"2023-08-07T02:02:55.742132076-04:00","level":"INFO","msg":"Sent range 1 for iteration: [2f72656769737472792f65646765732f61646d696e2f736572766572fffffffffffffff1, ) of size: 0 B","component":"badger"} {"time":"2023-08-07T02:02:55.754551205-04:00","level":"INFO","msg":"DB.Backup Sent data of size 2.2 KiB","component":"badger"} {"time":"2023-08-07T02:02:55.754931487-04:00","level":"INFO","msg":"db snapshot complete","component":"snapshots","duration":"13.017689ms","size":"982 B"} {"time":"2023-08-07T02:02:55.754985262-04:00","level":"INFO","msg":"starting snapshot up to","component":"raft","index":20} {"time":"2023-08-07T02:02:55.755028353-04:00","level":"INFO","msg":"creating new snapshot","component":"raft","storage":"/var/lib/webmesh/store","component":"snapshotstore","path":"/var/lib/webmesh/store/snapshots/2-20-1691388175755.tmp"} {"time":"2023-08-07T02:02:55.781345175-04:00","level":"INFO","msg":"snapshot complete up to","component":"raft","index":20} {"time":"2023-08-07T02:02:55.781571056-04:00","level":"ERROR","msg":"failed to transfer leadership","component":"raft","storage":"/var/lib/webmesh/store","error":"cannot find peer"} {"time":"2023-08-07T02:02:56.721167203-04:00","level":"ERROR","msg":"failed to appendEntries to","component":"raft","peer":{"Suffrage":1,"ID":"admin","Address":"172.16.0.2:9443"},"error":"dial tcp 172.16.0.2:9443: i/o timeout"} {"time":"2023-08-07T02:02:58.988560091-04:00","level":"ERROR","msg":"failed to heartbeat to","component":"raft","peer":"172.16.0.2:9443","backoff time":80000000,"error":"dial tcp 172.16.0.2:9443: i/o timeout"} {"time":"2023-08-07T02:02:58.988724645-04:00","level":"INFO","msg":"Lifetime L0 stalled for: 0s","component":"badger"} {"time":"2023-08-07T02:02:59.016826222-04:00","level":"INFO","msg":"Level 0 [ ]: NumTables: 01. Size: 1.3 KiB of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB\nLevel 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 3 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 4 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 5 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 6 [B]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel Done","component":"badger"} {"time":"2023-08-07T02:02:59.031477119-04:00","level":"INFO","msg":"Lifetime L0 stalled for: 0s","component":"raft","storage":"/var/lib/webmesh/store","component":"raftbadger"} {"time":"2023-08-07T02:02:59.099681848-04:00","level":"INFO","msg":"Level 0 [ ]: NumTables: 01. Size: 2.3 KiB of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB\nLevel 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 3 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 4 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 5 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 6 [B]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel Done","component":"raft","storage":"/var/lib/webmesh/store","component":"raftbadger"}I think this cause disconnections.
0.1.2
The text was updated successfully, but these errors were encountered: