-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] - container consistently fails to verify its connection (even though it is still connected), and eventually restarts its process because of this, cutting off other containers in the docker compose #258
Comments
I've tried pinging various things from inside |
Same here. Usually have to go in and 'docker compose protonwire restart' to reset the entire stack but definitely looking for a fix here. |
Curl exit code 6 is DNS error. From the logs I can see that metadata endpoints are indeed reachable, So I guess the problem is with Can you try to resolve
If they do not work, try setting ping may be a different issue as it typically requires additional caps ( |
@tprasadtp excellent, thank you, I'll give these commands a shot and see what I get! I don't know if changing the API endpoint will work, since I already attempted to do that as noted in my bug report, but it's worth a shot too. |
Alright I ran the suggested commands inside the protonwire container and here's what I got:
It also appeared that I can ping both IP addresses and URLs, and even curl icanhazip.com, from inside the protonwire client, but the docker container behind it still couldn't access the internet (perhaps because protonwire had restarted so many times it finally became stable, but the other docker couldn't talk to the newly started processes) . This was presenting differently than it had been when I made the bug report, so I restarted the docker compose in order to see if I could do your commands while it was having the problems I initially described, for better diagnostics. Instead, now the problem is gone! The connection can be verified the first time, so protonwire doesn't need to restart, which means the container behind it can access the internet. I'm worried this will happen again though. |
It might be that server you selected was offline for a while, which obviously results in connection errors. Because DNS is kinda circular dependency it may result in DNS errors when wireguard client is up. (DNS is 10.2.0.1 which is only accessible if VPN is connected). Can you provide logs with timestamps and exact server name? How are you running containers behind the VPN? There also might be a race condition which can lead to networking being broken inside them. |
I restarted my computer and I'm running into the exact same problems again, with basically the exact same logs, despite choosing a completely different server so I highly doubt that it's the server randomly being offline. Also as you can see in the docker composed that I put in the original issue post I have the docker container that's behind proton wire set up to depend on proton wire so there is no race condition there. |
Here is the current output of the three commands you listed above, now that protonwire is failing again:
|
10.2.0.1 is the protonvpn server which also handles DNS, so connection errors to server breaks everything. Whats the output of If it does not work, can you try to run with |
output of
output of
|
So it looks like it can connect fine to the internet. Try this command and check if it returns your public IP. If yes then something is wrong with routing/wireguard tunnel if not its only the DNS which is borked and need to be fixed.
If you get protonVPN public IP from the above command, use |
I ran that command, and I got the protonVPN endpoint IP address back, so it looks like it is just a DNS issue. I'm not using systemd-resolved though, and I'd rather not even slightly risk any data leaks (I have clinical anxiety and just don't want the mental space taken up by that lol) so I guess I'll just wait till a proper fix for the DNS is found. |
Next time you encounter the issue can you try the following?
I cannot reproduce the issue locally. Did the issue occur after upgrade to 7.2 or later ? |
Sure, no problem! I'll run them right now
I'm not sure, but I think so — I'd had the container up for like two or more straight months, then I finally rebooted my server and restarted the container, and that's when the problems started, which implies it started when the image was rebuilt, I think, |
@tprasadtp oh shit, I just realized, even though I thought I updated the container, it's still on version 7.0.3 according to |
Interesting. Now that I've updated, it works fine... for now, at least. Edit (later): nope, it's broken again |
I think change to updating the resolv.conf logic might be the issue. But openresolv(used in 7.0.x) is also broken as it does not behave well with bind mounted /etc/resolv.conf.
Can you please try these and give their outputs? |
Alright. It started working again this morning when I restarted it (connection was able to be verified), but I'll wait until it fails again and run those commands. In the meantime, check out these logs, it's doing something new — verifying the connection is still failing, but then it's reconnecting to the server, which it wasn't doing before, and reconnnecting successfully too. It also says the server is online:
|
Here's the command output:
here's the inspect output: [
{
"Id": "...",
"Created": "2023-10-01T22:52:30.692743725Z",
"Path": "/usr/bin/protonwire",
"Args": [
"connect",
"--container"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 237751,
"ExitCode": 0,
"Error": "",
"StartedAt": "2023-10-02T13:51:31.637724376Z",
"FinishedAt": "2023-10-02T13:51:31.279691757Z"
},
"Image": "sha256:952d93f4bd6ae8144ff6692c9e3e864a0aa93d2cd0fbc5be6593d152faaf7887",
"ResolvConfPath": "/var/lib/docker/containers/952e79f5dfaa2e757e1c7ee199869ad538d3e1389459395b552ddd1ec32ddeed/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/952e79f5dfaa2e757e1c7ee199869ad538d3e1389459395b552ddd1ec32ddeed/hostname",
"HostsPath": "/var/lib/docker/containers/952e79f5dfaa2e757e1c7ee199869ad538d3e1389459395b552ddd1ec32ddeed/hosts",
"LogPath": "/var/lib/docker/containers/952e79f5dfaa2e757e1c7ee199869ad538d3e1389459395b552ddd1ec32ddeed/952e79f5dfaa2e757e1c7ee199869ad538d3e1389459395b552ddd1ec32ddeed-json.log",
"Name": "/protonwire",
"RestartCount": 85,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "projectname_default",
"PortBindings": {
"6881/tcp": [
{
"HostIp": "",
"HostPort": "6881"
}
],
"6881/udp": [
{
"HostIp": "",
"HostPort": "6881"
}
],
"6969/tcp": [
{
"HostIp": "",
"HostPort": "6969"
}
]
},
"RestartPolicy": {
"Name": "unless-stopped",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": [
"NET_ADMIN"
],
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": null,
"DnsOptions": null,
"DnsSearch": null,
"ExtraHosts": [],
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Sysctls": {
"net.ipv4.conf.all.rp_filter": "2",
"net.ipv6.conf.all.disable_ipv6": "1"
},
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"Mounts": [
{
"Type": "tmpfs",
"Target": "/tmp"
},
{
"Type": "bind",
"Source": "/opt/docker/projectname/wg2-private.key",
"Target": "/etc/protonwire/private-key",
"ReadOnly": true
}
],
"MaskedPaths": [
"/proc/asound",
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware"
],
"ReadonlyPaths": [
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
],
"Init": true
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/577524c738e4e67596b7c164e4351a831e437fec127c110856ba3770c51ca24e-init/diff:/var/lib/docker/overlay2/0d5bbbeed3e21b388eb547b5a36712277dbb681c2970a2ebde7e01d477d183b7/diff:/var/lib/docker/overlay2/9478a60b4dc66f264edb60b6975c158f9777ee78c7e92e9a7692d716e87da10e/diff:/var/lib/docker/overlay2/bab38cec549e0a06f842239d6642393e8aef4c23982ad6da53a38892ae1d0296/diff:/var/lib/docker/overlay2/a4a97968382108a583457cd6b5af0de13ed4ae96ed52292357fa6657d06699fe/diff",
"MergedDir": "/var/lib/docker/overlay2/577524c738e4e67596b7c164e4351a831e437fec127c110856ba3770c51ca24e/merged",
"UpperDir": "/var/lib/docker/overlay2/577524c738e4e67596b7c164e4351a831e437fec127c110856ba3770c51ca24e/diff",
"WorkDir": "/var/lib/docker/overlay2/577524c738e4e67596b7c164e4351a831e437fec127c110856ba3770c51ca24e/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "tmpfs",
"Source": "",
"Destination": "/tmp",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Type": "bind",
"Source": "/opt/docker/projectname/wg2-private.key",
"Destination": "/etc/protonwire/private-key",
"Mode": "",
"RW": false,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "952e79f5dfaa",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": true,
"AttachStderr": true,
"ExposedPorts": {
"6881/tcp": {},
"6881/udp": {},
"6969/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PROTONVPN_SERVER=...",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/usr/bin/protonwire",
"connect",
"--container"
],
"Image": "ghcr.io/tprasadtp/protonwire:latest",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {
"com.docker.compose.config-hash": "cfdc52264b86d28cbca5a42920dea92335e01d29f4840e732559510780f9c09b",
"com.docker.compose.container-number": "1",
"com.docker.compose.depends_on": "",
"com.docker.compose.image": "sha256:952d93f4bd6ae8144ff6692c9e3e864a0aa93d2cd0fbc5be6593d152faaf7887",
"com.docker.compose.oneoff": "False",
"com.docker.compose.project": "projectname",
"com.docker.compose.project.config_files": "/opt/docker/projectname/docker-compose.yml",
"com.docker.compose.project.working_dir": "/opt/docker/projectname",
"com.docker.compose.service": "protonwire",
"com.docker.compose.version": "2.21.0",
"io.github.tprasadtp.metadata.git.branch": "HEAD",
"io.github.tprasadtp.metadata.git.commit": "3adea60a05cd076ee612f4f818dd886d21e8a430",
"io.github.tprasadtp.metadata.git.shortCommit": "3adea60",
"io.github.tprasadtp.metadata.git.tag": "7.2.3",
"io.github.tprasadtp.metadata.version.major": "7",
"io.github.tprasadtp.metadata.version.minor": "2",
"io.github.tprasadtp.metadata.version.patch": "3",
"io.github.tprasadtp.metadata.version.prerelease": "",
"io.github.tprasadtp.metadata.version.snapshot": "false",
"org.opencontainers.image.created": "2023-09-24T20:38:01Z",
"org.opencontainers.image.description": "\"ProtonVPN Wireguard Client for Linux\"",
"org.opencontainers.image.documentation": "https://tprasadtp.github.io/protonwire",
"org.opencontainers.image.licenses": "GPLv3",
"org.opencontainers.image.revision": "3adea60a05cd076ee612f4f818dd886d21e8a430",
"org.opencontainers.image.source": "\"https://github.com/tprasadtp/protonwire\"",
"org.opencontainers.image.title": "protonwire",
"org.opencontainers.image.url": "https://ghcr.io/tprasadtp/protonwire",
"org.opencontainers.image.vendor": "\"Prasad Tengse <tprasadtp@users.noreply.github.com>\"",
"org.opencontainers.image.version": "7.2.3"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "0ee1735cd5c84e0ea1201d9247dd6bf5fc8039d96539781fd7e8690f14dfe361",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"6881/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "6881"
},
{
"HostIp": "::",
"HostPort": "6881"
}
],
"6881/udp": [
{
"HostIp": "0.0.0.0",
"HostPort": "6881"
},
{
"HostIp": "::",
"HostPort": "6881"
}
],
"6969/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "6969"
},
{
"HostIp": "::",
"HostPort": "6969"
}
]
},
"SandboxKey": "/var/run/docker/netns/0ee1735cd5c8",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"projectname_default": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"protonwire",
"protonwire",
"952e79f5dfaa"
],
"NetworkID": "f2e23327b187848d908ce53d5a1b76ca8ab8ac101a8f15d2dbdf6b9e36dca5f9",
"EndpointID": "48b152225ed5514aff6ea1d2df8d8461ef27a3a8a5204229926982be3cc2439c",
"Gateway": "172.21.0.1",
"IPAddress": "172.21.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:15:00:02",
"DriverOpts": null
}
}
}
}
] |
Just to be clear this is output when connection fails, correct? Everything looks fine from routing table and routing priorities. Are you still able to resolve some random DNS from inside the container? |
Yeah
I'll check in a bit |
Its really weird. You can ping to DNS server it routes via correct interface, but only DNS resolution fails?
Take your time. I re-built the container image with debian as the base image Debian bookworm has podman can you try to run it with podman? Also rootless uses user-mode networking. I don't think its an issue but can you try to run container it in podman rootful mode? Have you disabled Netshield in your protonvpn configuration? |
Yes |
Hmmm, This is really strange. Try it with debian image |
I can somewhat reproduce the issue, but it recovers on its own quickly.
But it happens sporadically but it recovers quickly. Only changes I have from your configuration is I have pushed Any output from
when the issue occurs might shed light on whats happening. If that too does not help, try to run with |
When using this container image, ping isn't provided for some reason. I'll try the stuff in your most recent comment instead. |
Okay, trying with the 7.3.0-alpha1 container image, and the different environment variables you mentioned, and I'm still getting the same failure to verify the connection because it can't resolve the DNS of the domain as before. Here's the output from the three commands you gave:
|
I really appreciate your patience, you are being awesome! 🎉
Ahh my bad I missed iputils-ping (which was provided by busybox on alpine) 7.3.0-alpha2 should have it. Lets narrow it down.
grab
|
Cool I'll go through this tomorrow if that's okay. We'll get to the bottom of this! 😄 |
I'm also having the same issue. Was working fine earlier today but no joy the past hour. This fails from within the protonwire container but runs fine on the host machine. Same result when using SKIP_DNS_CONFIG 1 too, but I get a different error from protonwire.
UPDATE Seems to have stopped working again shortly after. |
figured out the issue on my end...i had multiple containers using the same wireguard private key...i created new keys for each instance and the issue went away. |
For anyone else facing this issue.. make sure you are utilizing the PRIVATE KEY NOT the public key. IDK why it can present itself as a CURL issue, but hey, knowledge is power. |
Version
7.2.3
Credential and Server Validation
nl-free-127.protonvpn.net
or server name likeNL#1
) as mentioned in the docs.System Architecture
x86_64
Kernel Version
6.1.0-12-amd64
Running on a NAS?
No
Runtime
Systemd (>244) Unit, docker-rootless
Version of Runtime
My configuration
Whitelisting API endpoints
I am not using ad-blocking DNS server or gateway
Troubleshooting & Runtime
Container/Pod/systemd log output with DEBUG=1 or
--debug
flagAny additional info
I've also tried changing the
IPCHECK_URL
to https://api.ipify.org/, but it didn't help. It still continues to fail to verify.Code of Conduct & PII Redaction
The text was updated successfully, but these errors were encountered: