-
Notifications
You must be signed in to change notification settings - Fork 711
Use the VM’s IP address on the same subnet as the host OS. #4175
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Use the VM’s IP address on the same subnet as the host OS. #4175
Conversation
9c6d7cb
to
912ed32
Compare
89e8a70
to
dee7e4e
Compare
Now, SSH port forwarding works via accessing guest IP directly. edited: benchmark result is updated at #4175 (comment), following details are obsoleted.
- SSH port forwarding over direct IP
```console
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 62915 connected to 127.0.0.1 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 518 MBytes 4.32 Gbits/sec
...
[ 5] 9.00-10.00 sec 650 MBytes 5.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 6.07 GBytes 5.22 Gbits/sec sender
[ 5] 0.00-10.00 sec 6.07 GBytes 5.21 Gbits/sec receiver
iperf Done.
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 62929 connected to 127.0.0.1 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 294 MBytes 2.46 Gbits/sec
...
[ 5] 9.00-10.01 sec 301 MBytes 2.52 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 2.94 GBytes 2.52 Gbits/sec sender
[ 5] 0.00-10.14 sec 2.94 GBytes 2.49 Gbits/sec receiver
iperf Done. |
edited: benchmark result is updated at #4175 (comment), following details are obsoleted. $ socat TCP-LISTEN:5202,fork TCP:192.168.64.3:5201&
$ iperf3 -c 127.0.0.1 -p 5202
Connecting to host 127.0.0.1, port 5202
[ 5] local 127.0.0.1 port 63300 connected to 127.0.0.1 port 5202
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 1.14 GBytes 9.72 Gbits/sec
...
[ 5] 9.01-10.01 sec 1.16 GBytes 9.95 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 11.6 GBytes 9.93 Gbits/sec sender
[ 5] 0.00-10.01 sec 11.6 GBytes 9.93 Gbits/sec receiver
iperf Done.
2025/10/11 18:30:02 socat[15240] E write(5, 0x78f040000, 8192): Broken pipe Will |
83ede84
to
cc08d05
Compare
|
Access method | sender | receiver |
---|---|---|
GRPC forwarder | 2.88 | 2.89 |
SSH forwarder | 2.31 | 2.31 |
SSH forwarder over VSOCK | 3.67 | 3.66 |
[New] SSH fwrdr via vzNAT | 8.76 | 8.75 |
[New] Forwarder via vzNAT | 22.0 | 22.0 |
(e.g.) socat via vzNAT |
8.93 | 8.93 |
(e.g.) Direct IP via vzNAT | 40.6 | 40.6 |
Gbits/sec |
How to measure
Setup
MacBook Pro 14 inch, 2023
cpu: Apple M2 Pro
mem: 16GB
$ sw_vers
ProductName: macOS
ProductVersion: 26.0.1
BuildVersion: 25A362
$ limactl --version
limactl version 2.0.0-alpha.2-89-gcc08d051
$ limactl start template://ubuntu --name test --containerd=none --log-level error
$ yes|limactl shell test DEBIAN_FRONTEND=noninteractive sudo apt-get -U install -qqq -y iperf3 &>/dev/null
GRPC port forwarder
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 57887 connected to 127.0.0.1 port 5201
...
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 3.36 GBytes 2.88 Gbits/sec sender
[ 5] 0.00-9.97 sec 3.36 GBytes 2.89 Gbits/sec receiver
iperf Done.
SSH port forwarder
$ LIMA_SSH_PORT_FORWARDER=true LIMA_SSH_OVER_VSOCK=false limactl restart test --log-level error
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 57958 connected to 127.0.0.1 port 5201
...
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 2.69 GBytes 2.31 Gbits/sec sender
[ 5] 0.00-10.01 sec 2.69 GBytes 2.31 Gbits/sec receiver
iperf Done.
SSH port forwarder over VSOCK
$ LIMA_SSH_PORT_FORWARDER=true LIMA_SSH_OVER_VSOCK=true limactl restart test --log-level error
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 57970 connected to 127.0.0.1 port 5201
...
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 4.27 GBytes 3.67 Gbits/sec sender
[ 5] 0.00-10.00 sec 4.27 GBytes 3.66 Gbits/sec receiver
iperf Done.
[NEW] SSH port forwarder via vzNAT
$ limactl stop test --log-level error
$ LIMA_SSH_PORT_FORWARDER=true limactl start test --log-level error --network vzNAT
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 58015 connected to 127.0.0.1 port 5201
...
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 10.2 GBytes 8.76 Gbits/sec sender
[ 5] 0.00-10.00 sec 10.2 GBytes 8.75 Gbits/sec receiver
iperf Done.
[NEW] Port forwarder via vzNAT
$ limactl restart test --log-level error
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 58025 connected to 127.0.0.1 port 5201
...
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 25.7 GBytes 22.0 Gbits/sec sender
[ 5] 0.00-10.00 sec 25.6 GBytes 22.0 Gbits/sec receiver
iperf Done.
(e.g.)P socat
via vzNAT
$ socat TCP-LISTEN:5202,fork TCP:192.168.64.2:5201&
$ iperf3 -c 127.0.0.1 -p 5202
Connecting to host 127.0.0.1, port 5202
[ 5] local 127.0.0.1 port 58136 connected to 127.0.0.1 port 5202
...
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 10.4 GBytes 8.93 Gbits/sec sender
[ 5] 0.00-10.00 sec 10.4 GBytes 8.93 Gbits/sec receiver
iperf Done.
2025/10/12 14:54:15 socat[24552] E write(5, 0x8dac10000, 8192): Broken pipe
(e.g.) Direct IP via vzNAT
$ limactl list test
NAME STATUS SSH VMTYPE ARCH CPUS MEMORY DISK DIR
test Running 192.168.64.2:22 vz aarch64 4 4GiB 100GiB ~/.lima/test
$ iperf3 -c 192.168.64.2
Connecting to host 192.168.64.2, port 5201
[ 5] local 192.168.64.1 port 58127 connected to 192.168.64.2 port 5201
...
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 47.3 GBytes 40.6 Gbits/sec sender
[ 5] 0.00-10.01 sec 47.3 GBytes 40.6 Gbits/sec receiver
iperf Done.
cc08d05
to
4954852
Compare
It seems that #3715 is related to the failure of the test using |
Some tests failed because that requires forwarding port on 127.0.0.1 in the guest. |
Fixed by reverted to "github.com/containers/gvisor-tap-vsock/pkg/tcpproxy" |
c81c411
to
9fd03ae
Compare
Changed to check if guest IP is a direct accessible IP or unspecified when using port forwarding to a direct IP. |
29db5b5
to
f22a2f0
Compare
.github/workflows/test.yml
Outdated
run: brew install bash coreutils w3m socat | ||
- name: Uninstall qemu | ||
run: brew uninstall --ignore-dependencies --force qemu | ||
- name: "Allow limactl to firewall" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add the reason as a comment line.
Is this needed for all the port forwards, or just for a particular test?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, it's test.
Changed to draft.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can't resolve error on https://github.com/lima-vm/lima/actions/runs/18483981814/job/52663789633#step:11:532
The failure was recorded in ha.stderr.log
as:
2025/10/14 03:06:50 tcpproxy: for incoming conn 127.0.0.1:49276, error dialing "0.0.0.0:3022": dial tcp 192.168.65.1:0->192.168.65.2:3022: connect: no route to host
I guessed "no route to host" was caused by firewall, but it seems to be different.
🤔
Is this gRPC forwarder? |
No, it's using |
805e32c
to
f34aebb
Compare
I can't reproduce "connect: no route to host" errors on dialing with local Macs that are mentioned in #4175 (comment)
Firewall state seems not to affect the errors. |
f34aebb
to
dbe02f0
Compare
Added fallback to GRPC forwarder if dialing to a direct IP fails, as I can’t resolve the "connect: no route to host" error on CI. Also removed support for static port forwarding via direct access to guest IP, as I can’t find a good fallback method. |
Is this safe to be merged in v2.0 or do we want to postpone this to v2.1? Also wondering if we can have an env var similar to https://lima-vm.io/docs/config/environment-variables/#lima_ssh_over_vsock for toggling the behavior until we can be confident of the maturity of the feature |
Until we figure out why the test does not work in CI, it may be good to opt-in by environment variables. |
ea72245
to
4938689
Compare
LimaVersion string `json:"limaVersion"` | ||
Param map[string]string `json:"param,omitempty"` | ||
// Guest IP address directly accessible from the host. | ||
GuestIP net.IP `json:"guestIP,omitempty"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can't we reuse SSHAddress
in L47?
Currently this still refers to "127.0.0.1" even in direct IP mode
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also considered it, but as far as I reviewed the code and past PR, etc., SSHAddress
and instSSHAddress
are just SSH addresses, and the value acquisition path is also driver-dependent, so I felt that the meaning did not match slightly, so I made it a separate field.
pkg/hostagent/requirements.go
Outdated
}, | ||
) | ||
|
||
useDirectIP := false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We merged LIMA_SSH_OVER_VSOCK
with true
as the default.
Is it safe to make this default too?
If we are not confident, the default may still false until v2.1.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have never seen an example of CI failure due to "SSH over VSOCK" so far.
The fact that the VM cannot be restarted when testing LIMA_SSH_OVER_VSOCK
in CI is also failed, but the ssh communication itself is possible.
So, I think it's okay to leave true
as the default value of LIMA_SSH_OVER_VSOCK
.
Since SSH is working with direct IP on CI, it should change |
…ost OS. Guest IP address will be detected in requirement process. `hostagent.go`: - Add fields to `HostAgent` - `guestIfnameOnSameSubnetAsHost string` - `guestIPv4 net.IP` - `guestIPv6 net.IP` - Add methods `GuestIP()`, `GuestIPv4()`, and `GuestIPv6()` to `HostAgent` - Add `HostAgent.WriteSSHConfigFile()` helper to write SSHConfigFile - Add `HostAgent.sshAddressPort()` helper to provide ipAddress and port for SSH `requirements.go`: - Add `stdoutParser func(string) error` field to `requirement` - Add `HostAgent.detectGuestIfnameOnSameSubnetAtHost()` to parse `ip -j neighbor` command output - Add `HostAgent.detectGuestIPAddress()` to parse `ip -j addr` command output - Add two requirements to `HostAgent.essentialRequirements()` - "detect guest interface on same subnet as the host" - "detect guest IPv4 address" - "detect guest IPv6 address" Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
Update to use guestIPAddress: - `limactl shell` - `limactl show-ssh` - `limactl tunnel` pkg/limatype: - Add `GuestIP net.IP` to `Instance` - Add `Instance.SSHAddressPort()` helper to provide ipAddress and port for SSH Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
Support printing "Guest IP Address: %s" on `limactl start` Signed-off-by: Norio Nomura <norio.nomura@gmail.com> # Conflicts: # pkg/hostagent/requirements.go
…r than "127.0.0.1" affected functions: - Copy to host - Reverse SSHFS - SSH port forwarding Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
Change to use `github.com/inetaf/tcpproxy`: - pkg/driver/vz/vsock_forwarder.go - pkg/portfwd/client.go - pkg/portfwdserver/server.go pkg/portfwd: - Use `dialContext` instead of `client` to use `net.Conn` other than `GrpcClientRW`. - Change to create proxies from `dialContext` parameters instead of `conn`. - Add `DialContextToGRPCTunnel()` to return `DialContext` function that connects to GRPC tunnel. pkg/hostagent: - Add `HostAgent.DialContextToGuestIP()` to return `DialContext` function that connects to the guest IP directly. If the guest IP is not known, it returns nil. - Prefer `HostAgent.DialContextToGuestIP()` over `portfwd.DialContextToGRPCTunnel()`. Signed-off-by: Norio Nomura <norio.nomura@gmail.com> pkg/hostagent: Use `HostAgent.DialContextToGuestIP()` if the IP is accessible directly. Signed-off-by: Norio Nomura <norio.nomura@gmail.com> Revert to "github.com/containers/gvisor-tap-vsock/pkg/tcpproxy" Signed-off-by: Norio Nomura <norio.nomura@gmail.com> # Conflicts: # pkg/hostagent/hostagent.go pkg/hostagent: Aware forwarding guest address is IPv4 or IPv6 Signed-off-by: Norio Nomura <norio.nomura@gmail.com> pkg/hostagent: Fallback to GRPC forwarder if dialing to a direct ip fails Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
…work=vzNAT` Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
To following tests will be run with enabled. Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
…rt-forwarding.pl` Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
…ariable to enable direct IP port forwarding to the VM. ### Direct IP Port Forwarding To enable direct IP port forwarding, set the `_LIMA_DIRECT_IP_PORT_FORWARDER` environment variable to `true`: ```bash export _LIMA_DIRECT_IP_PORT_FORWARDER=true ``` This feature makes Lima to use direct IP port forwarding instead of gRPC port forwarding. When this feature is enabled, Lima tries to connect to the guest's IP address directly for port forwarding. #### Fallback to gRPC Port Forwarding Lima may fall back to gRPC port forwarding in the following cases: - If the guest's IP address is not available, Lima falls back to gRPC port forwarding. - If the guest's IP address is available but the connection to the guest's IP address fails, Lima also falls back to gRPC port forwarding. - If the guest's IP address is not accessible from the host (e.g. localhost on guest), Lima falls back to gRPC port forwarding as well. Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
4938689
to
d2d9cda
Compare
…sts other than macOS Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
GuestIPAddress
GuestIP
field toGET /v1/info
endpoint`GuestIPAddress string
GuestIP string
field toStatus
ssh
execution to support SSH address other than "127.0.0.1"Support static port forwarding via direct access to guest IPdefault.yaml
with--network=vzNAT
_LIMA_DIRECT_IP_PORT_FORWARDER
environment variable to enable direct IP port forwarding to the VM