WSL 2.0: networkingMode=mirrored makes Docker unable to forward ports

[halimsamy]

Windows Version

Microsoft Windows [Version 10.0.22631.2338]

WSL Version

2.0.0.0

Are you using WSL 1 or WSL 2?

• WSL 2

Kernel Version

5.15.123.1-1

Distro Version

Ubuntu 20.04.6

Other Software

Docker version 24.0.6, build ed223bc

Repro Steps

• Change networkingMode to mirrored
• docker run -d -p 8080:80 nginx:alpine (example)
• Go to localhost:8080
• Cannot connect
• Try from WSL curl http//localhost:8080 same issue

Expected Behavior

To forward the port and be able to connect to my containers

Actual Behavior

Doesn't forward the port, so I cannot connect to my containers.

Diagnostic Logs

No response

[driver1998]

Interestingly, the port forwarding does work from another machine on the same network as host. Just not on the host machine itself.

Machine A:

• Change networkingMode to mirrored
• docker run -d -p 8080:80 nginx:alpine
• Go to localhost:8080
• Cannot connect
• Set Windows firewall rules to allow 8080 inbound
• Go to localhost:8080
• Cannot connect

Machine B:

• Go to A:8080
• works

[greenhat616]

Same issue

---

Maybe related issues:
docker/for-win#13686

[giovannicandido]

Same problem here

[lengthofrope]

Yup, it's unfortunate but I have the same issue. I am on the release channel with windows version 10.0.22621.2359

[over-star]

Same problem here

[speller]

In my case, Docker ports are not exposed to the host Windows machine at all after the upgrade to 2.0 even if I set networkingMode=NAT which is the default value. I can't see opened ports in the tcpview.exe utility. Previously, I was able to see them opened by the wslhost.exe process (or similar name).

Switching to NAT helped me.

[Yukari316]

same here

[tuxiaobei-scu]

+1

[bkfino]

+1

[ptr1120]

+1

[halimsamy]

@benhillis Any ideas/updates here?

[DukeCrimson]

same here

[tumugin]

+1

[Aerglonus]

Same issue here, the only way I found to make it "work" was adding ignoredPorts=8080 to the wslconfig, but if container has something like 4800:8080 need to add both ports to ignoredPorts=8080,4800 still doing this some containers might not work at all, also if cloudflare tunnel is running on WSL this will break to when mirrored and theres no way to know which ports is using to add it to the ignoredPorts

[halimsamy]

Same issue here, the only way I found to make it "work" was adding ignoredPorts=8080 to the wslconfig, but if container has something like 4800:8080 need to add both ports to ignoredPorts=8080,4800 still doing this some containers might not work at all, also if cloudflare tunnel is running on WSL this will break to when mirrored and theres no way to know which ports is using to add it to the ignoredPorts

I think that's a workaround that would work... but I am waiting for someone from the WSL team to answer us here, did they get to know that is the issue, and if they have any plans to fix this?

[lengthofrope]

Same issue here, the only way I found to make it "work" was adding ignoredPorts=8080 to the wslconfig, but if container has something like 4800:8080 need to add both ports to ignoredPorts=8080,4800 still doing this some containers might not work at all, also if cloudflare tunnel is running on WSL this will break to when mirrored and theres no way to know which ports is using to add it to the ignoredPorts

That seems to work. Nice find.

[zcobol]

ignorePorts is misleading. What it does is making in this case port 8080 available to request inside the container only, but is not forwarded to Windows side. Look for more details at https://learn.microsoft.com/en-us/windows/wsl/wsl-config

If you run get-nettcpConnection -LocalPort 8080 there's nothing listening on that port.

After starting the web server with docker run -d -p 8080:80 nginx:alpine requests inside the WSL container works:

zcobol@debian:~$ curl -I localhost:8080
HTTP/1.1 200 OK
Server: nginx/1.25.2
Date: Mon, 25 Sep 2023 16:41:36 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 15 Aug 2023 19:24:07 GMT
Connection: keep-alive
ETag: "64dbd0d7-267"
Accept-Ranges: bytes

Requests from Windows side timeout with:

curl: (28) Failed to connect to localhost port 8080 after 21276 ms: Couldn't connect to server

From another server it work, as mentioned by @driver1998 :

tux@raspi:~$ curl -I 192.168.1.105:8080
HTTP/1.1 200 OK
Server: nginx/1.25.2
Date: Mon, 25 Sep 2023 16:43:45 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 15 Aug 2023 19:24:07 GMT
Connection: keep-alive
ETag: "64dbd0d7-267"
Accept-Ranges: bytes

[mew1033]

Just ran into this as well. AWS sam build --use-container doesn't work. Gives the same error

 Ports are not available: exposing port TCP 127.0.0.1:5232 -> 0.0.0.0:0: listen tcp 127.0.0.1:5232: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.

(The port number changes randomly every time)

Based on this: https://github.com/aws/aws-sam-cli/blob/c5b9b1e399a1e5c938ef72934a14ede934e17bac/samcli/local/docker/container.py#L124-L125
I added every port from 5000-9000 to the ignoredPorts list. It's ugly, but for now it works.

[halimsamy]

Just ran into this as well. AWS sam build --use-container doesn't work. Gives the same error

 Ports are not available: exposing port TCP 127.0.0.1:5232 -> 0.0.0.0:0: listen tcp 127.0.0.1:5232: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.

(The port number changes randomly every time)

Based on this: https://github.com/aws/aws-sam-cli/blob/c5b9b1e399a1e5c938ef72934a14ede934e17bac/samcli/local/docker/container.py#L124-L125 I added every port from 5000-9000 to the ignoredPorts list. It's ugly, but for now it works.

It seems like it's generally a problem with WSL itself. I am looking for a fix soon, since the new network mode is so much useful but it's not useable (usefully) in the current state.

[imacarthur741]

I ma having the same issue with apache. No changes other than added --experimental but now nothing works.

(98)Address already in use: AH00072: make_sock: could not bind to address [::]:80
(98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
[no listening sockets available, shutting down](apache2.service: Control process exited, code=exited, status=1/FAILURE)

root@ACER-Nitro:/usr/sbin# lsof -nP -iTCP -sTCP:LISTEN
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-r 308 systemd-resolve 14u IPv4 22507 0t0 TCP 127.0.0.53:53 (LISTEN)
mysqld 385 mysql 21u IPv4 32886 0t0 TCP 127.0.0.1:33060 (LISTEN)
mysqld 385 mysql 23u IPv4 27396 0t0 TCP 127.0.0.1:3306 (LISTEN)
miniserv. 1867 root 5u IPv4 52863 0t0 TCP *:10000 (LISTEN)

[zohaibhassan156]

Same problem

[zohaibhassan156]

ignoredPorts=8025,1025 work for me too. I had to use mailpit in docker

docker run -d \
--name=mailpit \
--restart unless-stopped \
-p 8025:8025 \
-p 1025:1025 \
axllent/mailpit

[shigenobuokamoto]

there seem to be two issues why Docker containers cannot connect from Windows.

1. ignorePorts works
   Docker Desktop, probably

2. the port forwarding does work from another machine on the same network as host. Just not on the host machine itself.
   docker-ce that installed on Linux

temporary measures for 2 ....
disable iptables and use docker-proxy

/etc/docker/daemon.json

{
    "iptables": false
}

when using mirrored, the behavior seems to be different from the previous localhostforwarding.

use docker-proxy(listen on Linux)

windows  --->  linux(WSL)
  localhostforwarding (until)
ok    127.0.0.1   --->     127.0.0.1(lo)
ok    127.0.0.1   <---     127.0.0.1(lo)

  netowrkingMode=mirrored
ok    127.0.0.1   --->     127.0.0.1(loopback0)
ok    127.0.0.1   <---     127.0.0.1(loopback0)

interface is different, but the behavior remains the same.

use iptables(listeon on container)

windows  --->  linux(WSL) ---> Docker container
  localhostforwarding (until)
ok    127.0.0.1   ---> 127.0.0.1(lo)    /   172.x.x.1(br-xxx)  --->  172.x.x.x(eth0)
ok    127.0.0.1   <--- 127.0.0.1(lo)    /   172.x.x.1(br-xxx)  <---  172.x.x.x(eth0)

  netowrkingMode=mirrored
ok    127.0.0.1   ---> 127.0.0.1(loopback0)/127.0.0.1(br-xxxx) --->  172.x.x.x(eth0)
                                          ~~~~~~~~~
ng                                                           <- - -  172.x.x.x(eth0)

via localhostforwarding(until), source address(Windows) was the docker network gateway (=pointing to linux).

via mirrored, source address is 127.0.0.1.
for this reason that packet is returned to 127.0.0.1(localhost on container) and does not reach Windows.
in the case of access from another node(PC), there is no problem because source is his address.

[shigenobuokamoto]

@felipecrs

I wonder why iptables: false would not be enough for you.

most issues can be resolved by modifying daemon.json. however, the usability may change by not using iptables.
bigger issues is that cannot use docker unless setting daemon.json.
my script is just a sample implementation workaround, but if it has a reasonable effect without any negative effects on everyone, i hope that i will be able to propose its inclusion in /init.

https://gist.github.com/shigenobuokamoto/b565d468541fc8be7d7d76a0434496a0

my script is very simple, just apply the following rules to nftables.

table ip nat {
    chain WSLPREROUTING {
        type nat hook prerouting priority dstnat - 1; policy accept;
        iif "loopback0" ip daddr 127.0.0.1 counter dnat to 127.0.0.1;
    }
}

this chain is processed immediately before PREROUTING, and will DNAT destination of the packet that arrives from loopback0 interface to 127.0.0.1 to 127.0.0.1. it may seem like a wasteful process, but adding this action breaks the prerouting hook and disables any Docker rules set in the PREROUTING chain.

using userland-proxy only for access from localhost is the same as Docker's default behavior.

to: who have already used it
i have made some changes from the previous one,

before
  add chain   ip nat PREROUTING { type nat hook prerouting priority dstnat; policy accept; };
  insert rule ip nat PREROUTING iif loopback0 counter accept comment mirrored;

after
  add chain   ip nat WSLPREROUTING { type nat hook prerouting priority dstnat - 1; policy accept; };
  insert rule ip nat WSLPREROUTING iif loopback0  ip daddr 127.0.0.1 counter dnat to 127.0.0.1 comment mirrored;

both are methods to prevent the PREROUTING NAT rule.
there is no change in behavior when using with Docker, but i think this is safer because it does not affect the chain managed by iptables.

if you like, please try this as well.

[andreaprilucm]

@felipecrs When you disable iptables manipulations by docker, you can't have isolated networks for your docker compose stacks.
It has the same effect as adding --network host to each container. This is acceptable for simple use cases where you start a few unrelated containers.

Imagine that I have an application composed of two containers: a web server (Tomcat port 8080) and a database (oracle port 1521). I create a compose.yaml file containing two services : tomcat and oracle. And I start the pair of containers with docker compose up.
In my compose.yaml, I only expose the web server port to the outside world. The application running in the Tomcat container refers to the database as "oracle port 1521" (oracle is the name of the service in compose.yaml).

When iptables manipulation is disabled, the Oracle database binds to 127.0.0.1:1521. If you try to start a second compose stack, Oracle will complain that the port is already in use.

If iptables manipulation is enabled (default), each stack will have its own network ; e.g. "172.19.0.0/16" and "172.20.0.0/16" and Oracle in the first stack will bind to 172.19.0.3:1521 wile the one in the second stack will bind to 172.20.0.3:1521. In the tomcat container of the first stack, the oracle host name refers to 172.19.0.3 while in the Tomcat container the oracle host name refers to 172.20.0.3.

The fix proposed by Shigenobu NAT the packets arriving on the loopback0 interface (the one the Windows packets come from) before forwarding them on the docker0 interface (the one Docker bridges its networks). See his comment of Sep, 30 2023 for the explanation of the problem.

If you want more details on the networking of Docker, see https://docs.docker.com/network/network-tutorial-standalone/ and https://docs.docker.com/network/network-tutorial-host/.

So, disabling iptables manipulation is fine for some simple use cases but you will quickly get stuk.

[felipecrs]

Thanks both, very useful information.

[felipecrs]

Although networkingMode=mirrored is great, I hope it does not get promoted to default without having this issue resolved first. 😅

[felipecrs]

Just to renew this issue, I tested again with the just released WSL 2.2.1, but same problem still occurs.

[NiklasBr]

Still an issue with v4.28.0

[andreaprilucm]

@jsayer101 I don't understand your configuration.
Where is your portainer server deployed, your portainers agents, your firewal/VPN server, the browser you use to access your portainer server (outside your corporate LAN?)... ? Could you draw a picture of it?

If you want to access ports on your WSL from other hosts, does your windows firewall authorize it?

Have you tried the startup script propose above?

[foureight84]

Hello,

We're trying to have a setup of WSL working behind a corporate VPN (Cisco Any Connect).

On Win11/WSL2, so far we have to use networkingMode=mirrored to get WSL to be able to access the Internet. (works fine)

But we're also trying to use docker inside WSL. Now, with this setup we can't access our containers that exposes specific ports from our Windows machine. For example portainer on port 9000.

This issue cause us to not be able to use Docker on WSL behind a corporate VPN.

@jsayer101 I don't understand your configuration. Where is your portainer server deployed, your portainers agents, your firewal/VPN server, the browser you use to access your portainer server (outside your corporate LAN?)... ? Could you draw a picture of it?

If you want to access ports on your WSL from other hosts, does your windows firewall authorize it?

Have you tried the startup script propose above?

Yea something is out of order. Corporate firewall wouldn't prevent you from accessing WSL on your own computer.

Anyway, add this to your WSL OS (#10494 (comment)) and add the following settings to your .wslconfig. It's similar to what you have but with additional flags. This is working fine across all of my Win11 PCs

[wsl2]
networkingMode=mirrored
firewall=false
dnsTunneling=true

[experimental]
useWindowsDnsCache=true
hostAddressLoopback=true

[dharapvj]

@shigenobuokamoto

could you please try this?
https://gist.github.com/shigenobuokamoto/b565d468541fc8be7d7d76a0434496a0

Your hack works fine for me. Thank you so much!.. Just one small issue.. it breaks other IPv6 access going to internet for me. My docker is not configured for IPv6. So.. could you help me understand what parts of the gist should be removed so that it does not mess with ipv6 routing stuff?

❯ ssh -6 ubuntu@xxx.com -vv
OpenSSH_8.9p1 Ubuntu-3ubuntu0.6, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /home/vj/.ssh/config
debug1: /home/vj/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "xxx.com" port 22
debug1: Connecting to xxx.com [2406:0000:000:0000:000:0000:0000:c711] port 22.
debug1: connect to address 2406:0000:000:0000:000:0000:0000:c711 port 22: Connection timed out
ssh: connect to host xxx.com port 22: Connection timed out

With normal mirrored mode, this connectivity used to work. I can still connect fine via same command from windows host. just not from WSL.

[shigenobuokamoto]

@dharapvj

Your hack works fine for me. Thank you so much!.. Just one small issue.. it breaks other IPv6 access going to internet for me. My docker is not configured for IPv6. So.. could you help me understand what parts of the gist should be
removed so that it does not mess with ipv6 routing stuff?

if you exclude the line containing "ip6", IPv6 support will be disabled.

$ grep -v ip6 network-mirrored.service > mirrored-noip6.service

however, since this support controls the source port for sources other than WSL container (= the traffic routed by WSL container), it is strange that access via SSH is no longer possible.

[dharapvj]

Thank you @shigenobuokamoto! That did the trick! Since, unfortunately, I do not understand what your script does... I am not able to explain why it works!

[hyp530]

Waiting for WSL to fix this issue.

[impzero]

same issue

[AlencarGabriel]

same issue, please MS, help us!

[keith-horton]

Hi there.

We have a few questions as we narrow down what is going on.

The below is referencing the troubleshooting docs in https://github.com/MicrosoftDocs/WSL/blob/main/WSL/troubleshooting.md

Sadly not all changes have been replicated yet to https://learn.microsoft.com/en-us/windows/wsl/troubleshooting :(

1. can folks who are hitting this confirm they do not have a separate Network Manager running?
   see "Docker container issues when its Network Manager is running" in the above troubleshooting article.

2. have folks who have hit this looked at the troubleshooting article we wrote? Can you give us feedback if that works for you, or if that cannot be done for a reason with your scenario?
   see: "Docker container issues in WSL2 with Mirrored networking mode enabled when running under the default networking namespace"

Thanks for your help!
We're trying to understand exactly what is going on.

[AlencarGabriel]

Hi there.

We have a few questions as we narrow down what is going on.

The below is referencing the troubleshooting docs in https://github.com/MicrosoftDocs/WSL/blob/main/WSL/troubleshooting.md

Sadly not all changes have been replicated yet to https://learn.microsoft.com/en-us/windows/wsl/troubleshooting :(

1. can folks who are hitting this confirm they do not have a separate Network Manager running?
   see "Docker container issues when its Network Manager is running" in the above troubleshooting article.
2. have folks who have hit this looked at the troubleshooting article we wrote? Can you give us feedback if that works for you, or if that cannot be done for a reason with your scenario?
   see: "Docker container issues in WSL2 with Mirrored networking mode enabled when running under the default networking namespace"

Thanks for your help! We're trying to understand exactly what is going on.

In my scenario, I don't have network-manager.service:

I use docker-ce, not docker-desktop. My case:

From WSL: curl localhost:8080 OK (service 8080 running in windows host)

From container docker: curl localhost:8080 = Couldn't connect to server - OK localhost here is the container loopback.

From container docker: curl 10.173.16.72:8080 = Couldn't connect to server. - 10.173.16.72 is IP of WSL and Windows Host.

[shigenobuokamoto]

seems like a new issue that is different from the current flow.

From container docker: curl 10.173.16.72:8080 = Couldn't connect to server. - 10.173.16.72 is IP of WSL and Windows Host.

this is the WSL ip address from the container's perspective, not WSL and Windows Host.

then, i tried to think of a way to implement this method.
with these steps, you can connect 10.173.16.72:8080 from the container to 8080 on the Windows host.

1. enable the target port to overlap with Windows Host

.wslconfig

[experimental]
ignoredPorts=8080

2. 10.173.16.72:8080 -> 127.0.0.1:8080 with socat

$ sudo socat TCP-LISTEN:8080,bind=10.173.16.72,reuseaddr,fork TCP:127.0.0.1:8080

[AlencarGabriel]

seems like a new issue that is different from the current flow.

From container docker: curl 10.173.16.72:8080 = Couldn't connect to server. - 10.173.16.72 is IP of WSL and Windows Host.

this is the WSL ip address from the container's perspective, not WSL and Windows Host.

then, i tried to think of a way to implement this method. with these steps, you can connect 10.173.16.72:8080 from the container to 8080 on the Windows host.

1. enable the target port to overlap with Windows Host

.wslconfig

[experimental]
ignoredPorts=8080

2. 10.173.16.72:8080 -> 127.0.0.1:8080 with socat

$ sudo socat TCP-LISTEN:8080,bind=10.173.16.72,reuseaddr,fork TCP:127.0.0.1:8080

Sorry @shigenobuokamoto , I mixed up the problems.

Regarding the 10.173.16.72 ip, it's from WSL and Windows Host. It's different because it's the IP when I'm connected to the VPN. I also tried the IP of the Wi-fi network interface but to no avail.

But I had the same port forwarding problem with my docker-ce and networkingMode=mirrored.

docker run --rm -it -p 8082:8082 alpine nc -p 8082 -l

Trying to connect via the browser or curl on the Windows machine doesn't work (http://localhost:8082). However, if I connect via WSL, it works.

I set {"iptables": false} in daemon.json and it worked. Without the iptables modification it only worked with --network-host.

Now I think we're talking about the same thing in this issue.

[shigenobuokamoto]

@AlencarGabriel, sorry if I offended you

mirrored network has a rather special structure. it would be nice to have an official document, but i have written my thoughts below, so please use them as a reference.

#11326 (comment)

for connections from Windows Host to docker on WSL, you can control traffic to the container using daemon.json or nftables. currently, it is not possible to create the same behavior as native linux without controlling it.

10.173.16.72 is its own address to linux. since it processes the traffic that comes here by itself, it rejects connect to 8080 that is not open. but Linux itself can connect to 10.173.16.72:8080 (on Windows Host). this is where the magic of mirrored network.

this issue and similar docker-related issues pointed to connect from Windows host to container service.
in that sense, i think that connecting from container to Windows poses a new issue. and a different (ad-hoc) solution is required than what has been shown so far.

[watchingfun]

Interestingly, the port forwarding does work from another machine on the same network as host. Just not on the host machine itself.

Machine A:

• Change networkingMode to mirrored
• docker run -d -p 8080:80 nginx:alpine
• Go to localhost:8080
• Cannot connect
• Set Windows firewall rules to allow 8080 inbound
• Go to localhost:8080
• Cannot connect

Machine B:

• Go to A:8080
• works

🤣Damn, I've been messing around all afternoon. I turned off the firewall and forwarded the ports, but I just don't know why the telnet connection won't connect. Later I saw what you said and found out that it doesn’t work locally.

[felipecrs]

What is your distribution? I'm tried configuring my WSL like yours, but I still can reproduce the issue.

WindowsTerminal_7GBLym2iKS.mp4

❯ wsl --version
WSL version: 2.2.3.0
Kernel version: 5.15.153.1-2
WSLg version: 1.0.61
MSRDC version: 1.2.5326
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26091.1-240325-1447.ge-release
Windows version: 10.0.22631.3447

[felipecrs]

BTW I can confirm that @shigenobuokamoto's workaround works like a charm.

[keith-horton]

To follow-up on this thread, we are actively working to address this. We are considering solution such as that shown by @shigenobuokamoto earlier in this thread. We are currently considering options and possible side-effects for the multitude of ways Linux might be configured by users.

Thanks everyone for the detailed feedback!

[underlines]

Wow, I couldn't get it to work in mirrored mode for 4 hours, then it randomly worked after restarting services, killing processes, restarting wsl, running docker desktop as admin etc.
The next day I couldn't open docker desktop, the engine would just not start, for 2 hours.

Modifying my config finally worked:

I added:

firewall=false
dnsTunneling=true

so my config now looks like this:

[wsl2]
networkingMode=mirrored
# localhostForwarding=true
firewall=false
dnsTunneling=true

[experimental]
autoMemoryReclaim=gradual
sparseVhd=true

[SiggiSmara]

I'm using docker-ce, behind a corporate proxy/VPN with mirrored networking (no other wsl settings) and the only thing that has worked so far that allows access to docker containers ports from the windows host side is to not use iptables on the dockerd side.

[tyeth]

Just got this running recently with @underlines config.

Forgot that mirrored mode requires a signed in user, which fails if offline (even signed in and online before reboot), which is beyond frustrating, and will cause me to recommend against docker desktop version of docker.

[spyro2000]

Network Mode mirrored does not work for me with docker. Looks like port 2375 is only reachable via IPv6 (netstat - nl | grep 2375 only returns a mapping for 'tcp6').

Why I can reach http://localhost:2375/version via Browser, e. g. Testcontainers does not find a docker environment (despite DOCKER_HOST set to 'tcp://localhost:2375').

Even after 8 years, docker is just a nightmare with WSL(2).I am literally the only dev using Windows anymore on my team, everyone else is using Macs now where it just works.