Skip to content

Conversation

norio-nomura
Copy link
Contributor

@norio-nomura norio-nomura commented Oct 10, 2025

  • pkg/hostagent: Detect the VM’s IP address on the same subnet as the host OS
  • pkg/hostagent: Add GuestIPAddressGuestIP field to GET /v1/info endpoint`
  • pkg/hostagent/events: Add GuestIPAddress string GuestIP string field to Status
  • pkg/hostagent: Update all ssh execution to support SSH address other than "127.0.0.1"
  • Support port forwarding via direct access to guest IP
  • Support static port forwarding via direct access to guest IP
  • Add test using default.yaml with --network=vzNAT
  • Add the _LIMA_DIRECT_IP_PORT_FORWARDER environment variable to enable direct IP port forwarding to the VM

@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch from 9c6d7cb to 912ed32 Compare October 10, 2025 10:01
@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch from 89e8a70 to dee7e4e Compare October 11, 2025 08:36
@norio-nomura
Copy link
Contributor Author

norio-nomura commented Oct 11, 2025

Now, SSH port forwarding works via accessing guest IP directly.
iperf3 comparison:

edited: benchmark result is updated at #4175 (comment), following details are obsoleted.

- SSH port forwarding over direct IP ```console $ iperf3 -c 127.0.0.1 Connecting to host 127.0.0.1, port 5201 [ 5] local 127.0.0.1 port 62915 connected to 127.0.0.1 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 518 MBytes 4.32 Gbits/sec ... [ 5] 9.00-10.00 sec 650 MBytes 5.45 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 6.07 GBytes 5.22 Gbits/sec sender [ 5] 0.00-10.00 sec 6.07 GBytes 5.21 Gbits/sec receiver

iperf Done.

- Direct IP
```console
$ iperf3 -c 192.168.64.3
Connecting to host 192.168.64.3, port 5201
[  5] local 192.168.64.1 port 62913 connected to 192.168.64.3 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  2.76 GBytes  23.6 Gbits/sec                  
...
[  5]   9.01-10.01  sec  2.82 GBytes  24.2 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  28.1 GBytes  24.1 Gbits/sec                  sender
[  5]   0.00-9.99   sec  28.1 GBytes  24.1 Gbits/sec                  receiver

iperf Done.
  • SSH port forwarding over VSOCK
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[  5] local 127.0.0.1 port 62929 connected to 127.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   294 MBytes  2.46 Gbits/sec                  
...
[  5]   9.00-10.01  sec   301 MBytes  2.52 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  2.94 GBytes  2.52 Gbits/sec                  sender
[  5]   0.00-10.14  sec  2.94 GBytes  2.49 Gbits/sec                  receiver

iperf Done.

@norio-nomura norio-nomura marked this pull request as ready for review October 11, 2025 08:59
@norio-nomura
Copy link
Contributor Author

norio-nomura commented Oct 11, 2025

  • port forward via socat

edited: benchmark result is updated at #4175 (comment), following details are obsoleted.

$ socat TCP-LISTEN:5202,fork TCP:192.168.64.3:5201&
$ iperf3 -c 127.0.0.1 -p 5202
Connecting to host 127.0.0.1, port 5202
[  5] local 127.0.0.1 port 63300 connected to 127.0.0.1 port 5202
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  1.14 GBytes  9.72 Gbits/sec                  
...
[  5]   9.01-10.01  sec  1.16 GBytes  9.95 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  11.6 GBytes  9.93 Gbits/sec                  sender
[  5]   0.00-10.01  sec  11.6 GBytes  9.93 Gbits/sec                  receiver

iperf Done.
2025/10/11 18:30:02 socat[15240] E write(5, 0x78f040000, 8192): Broken pipe

Will limactl also be able to go this far?

@norio-nomura norio-nomura changed the title [WIP] Use the VM’s IP address on the same subnet as the host OS. Use the VM’s IP address on the same subnet as the host OS. Oct 11, 2025
@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch 2 times, most recently from 83ede84 to cc08d05 Compare October 12, 2025 04:33
@norio-nomura
Copy link
Contributor Author

iperf3 comparison

Access method sender receiver
GRPC forwarder 2.88 2.89
SSH forwarder 2.31 2.31
SSH forwarder over VSOCK 3.67 3.66
[New] SSH fwrdr via vzNAT 8.76 8.75
[New] Forwarder via vzNAT 22.0 22.0
(e.g.) socat via vzNAT 8.93 8.93
(e.g.) Direct IP via vzNAT 40.6 40.6
Gbits/sec
How to measure

Setup

MacBook Pro 14 inch, 2023
cpu: Apple M2 Pro
mem: 16GB

$ sw_vers
ProductName:		macOS
ProductVersion:		26.0.1
BuildVersion:		25A362
$ limactl --version
limactl version 2.0.0-alpha.2-89-gcc08d051
$ limactl start template://ubuntu --name test --containerd=none --log-level error
$ yes|limactl shell test DEBIAN_FRONTEND=noninteractive sudo apt-get -U install -qqq -y iperf3 &>/dev/null

GRPC port forwarder

$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[  5] local 127.0.0.1 port 57887 connected to 127.0.0.1 port 5201
...
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.36 GBytes  2.88 Gbits/sec                  sender
[  5]   0.00-9.97   sec  3.36 GBytes  2.89 Gbits/sec                  receiver

iperf Done.

SSH port forwarder

$ LIMA_SSH_PORT_FORWARDER=true LIMA_SSH_OVER_VSOCK=false limactl restart test --log-level error
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[  5] local 127.0.0.1 port 57958 connected to 127.0.0.1 port 5201
...
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  2.69 GBytes  2.31 Gbits/sec                  sender
[  5]   0.00-10.01  sec  2.69 GBytes  2.31 Gbits/sec                  receiver

iperf Done.

SSH port forwarder over VSOCK

$ LIMA_SSH_PORT_FORWARDER=true LIMA_SSH_OVER_VSOCK=true limactl restart test --log-level error
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[  5] local 127.0.0.1 port 57970 connected to 127.0.0.1 port 5201
...
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  4.27 GBytes  3.67 Gbits/sec                  sender
[  5]   0.00-10.00  sec  4.27 GBytes  3.66 Gbits/sec                  receiver

iperf Done.

[NEW] SSH port forwarder via vzNAT

$ limactl stop test --log-level error
$ LIMA_SSH_PORT_FORWARDER=true limactl start test --log-level error --network vzNAT
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[  5] local 127.0.0.1 port 58015 connected to 127.0.0.1 port 5201
...
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  10.2 GBytes  8.76 Gbits/sec                  sender
[  5]   0.00-10.00  sec  10.2 GBytes  8.75 Gbits/sec                  receiver

iperf Done.

[NEW] Port forwarder via vzNAT

$ limactl restart test --log-level error
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[  5] local 127.0.0.1 port 58025 connected to 127.0.0.1 port 5201
...
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  25.7 GBytes  22.0 Gbits/sec                  sender
[  5]   0.00-10.00  sec  25.6 GBytes  22.0 Gbits/sec                  receiver

iperf Done.

(e.g.)P socat via vzNAT

$ socat TCP-LISTEN:5202,fork TCP:192.168.64.2:5201&
$ iperf3 -c 127.0.0.1 -p 5202
Connecting to host 127.0.0.1, port 5202
[  5] local 127.0.0.1 port 58136 connected to 127.0.0.1 port 5202
...
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  10.4 GBytes  8.93 Gbits/sec                  sender
[  5]   0.00-10.00  sec  10.4 GBytes  8.93 Gbits/sec                  receiver

iperf Done.
2025/10/12 14:54:15 socat[24552] E write(5, 0x8dac10000, 8192): Broken pipe

(e.g.) Direct IP via vzNAT

$ limactl list test
NAME    STATUS     SSH                VMTYPE    ARCH       CPUS    MEMORY    DISK      DIR
test    Running    192.168.64.2:22    vz        aarch64    4       4GiB      100GiB    ~/.lima/test
$ iperf3 -c 192.168.64.2
Connecting to host 192.168.64.2, port 5201
[  5] local 192.168.64.1 port 58127 connected to 192.168.64.2 port 5201
...
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  47.3 GBytes  40.6 Gbits/sec                  sender
[  5]   0.00-10.01  sec  47.3 GBytes  40.6 Gbits/sec                  receiver

iperf Done.

@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch from cc08d05 to 4954852 Compare October 12, 2025 06:41
@norio-nomura
Copy link
Contributor Author

It seems that #3715 is related to the failure of the test using w3m in CI, but I want to know what to do.

@norio-nomura
Copy link
Contributor Author

Some tests failed because that requires forwarding port on 127.0.0.1 in the guest.
That cannot be supported by port forwarding to direct IP on the guest.

@norio-nomura
Copy link
Contributor Author

It seems that #3715 is related to the failure of the test using w3m in CI, but I want to know what to do.

Fixed by reverted to "github.com/containers/gvisor-tap-vsock/pkg/tcpproxy"

@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch from c81c411 to 9fd03ae Compare October 13, 2025 05:26
@norio-nomura
Copy link
Contributor Author

Some tests failed because that requires forwarding port on 127.0.0.1 in the guest. That cannot be supported by port forwarding to direct IP on the guest.

Changed to check if guest IP is a direct accessible IP or unspecified when using port forwarding to a direct IP.

@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch 4 times, most recently from 29db5b5 to f22a2f0 Compare October 14, 2025 01:14
run: brew install bash coreutils w3m socat
- name: Uninstall qemu
run: brew uninstall --ignore-dependencies --force qemu
- name: "Allow limactl to firewall"
Copy link
Member

@AkihiroSuda AkihiroSuda Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the reason as a comment line.
Is this needed for all the port forwards, or just for a particular test?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, it's test.
Changed to draft.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't resolve error on https://github.com/lima-vm/lima/actions/runs/18483981814/job/52663789633#step:11:532
The failure was recorded in ha.stderr.log as:

2025/10/14 03:06:50 tcpproxy: for incoming conn 127.0.0.1:49276, error dialing "0.0.0.0:3022": dial tcp 192.168.65.1:0->192.168.65.2:3022: connect: no route to host

I guessed "no route to host" was caused by firewall, but it seems to be different.
🤔

@norio-nomura norio-nomura marked this pull request as draft October 14, 2025 03:17
@AkihiroSuda
Copy link
Member

[New] Forwarder via vzNAT 22.0 22.0

Is this gRPC forwarder?

@norio-nomura
Copy link
Contributor Author

Is this gRPC forwarder?

No, it's using tcpproxy.DialProxy with DialContextToGuestIP as DialContext.

@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch 2 times, most recently from 805e32c to f34aebb Compare October 15, 2025 00:34
@norio-nomura
Copy link
Contributor Author

I can't reproduce "connect: no route to host" errors on dialing with local Macs that are mentioned in #4175 (comment)
The test-port-forwarding.pl test passed on local Macs:

  • macOS 26.0.1 with Apple M2
  • macOS 13.7.8 with Intel

Firewall state seems not to affect the errors.

@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch from f34aebb to dbe02f0 Compare October 15, 2025 10:14
@norio-nomura
Copy link
Contributor Author

Added fallback to GRPC forwarder if dialing to a direct IP fails, as I can’t resolve the "connect: no route to host" error on CI. Also removed support for static port forwarding via direct access to guest IP, as I can’t find a good fallback method.

@AkihiroSuda AkihiroSuda added this to the v2.0.0 milestone Oct 16, 2025
@AkihiroSuda AkihiroSuda linked an issue Oct 16, 2025 that may be closed by this pull request
@AkihiroSuda
Copy link
Member

Is this safe to be merged in v2.0 or do we want to postpone this to v2.1?

Also wondering if we can have an env var similar to https://lima-vm.io/docs/config/environment-variables/#lima_ssh_over_vsock for toggling the behavior until we can be confident of the maturity of the feature

@norio-nomura
Copy link
Contributor Author

Until we figure out why the test does not work in CI, it may be good to opt-in by environment variables.

@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch from ea72245 to 4938689 Compare October 16, 2025 09:16
@norio-nomura norio-nomura marked this pull request as ready for review October 16, 2025 09:54
LimaVersion string `json:"limaVersion"`
Param map[string]string `json:"param,omitempty"`
// Guest IP address directly accessible from the host.
GuestIP net.IP `json:"guestIP,omitempty"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we reuse SSHAddress in L47?

Currently this still refers to "127.0.0.1" even in direct IP mode

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also considered it, but as far as I reviewed the code and past PR, etc., SSHAddress and instSSHAddress are just SSH addresses, and the value acquisition path is also driver-dependent, so I felt that the meaning did not match slightly, so I made it a separate field.

},
)

useDirectIP := false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We merged LIMA_SSH_OVER_VSOCK with true as the default.
Is it safe to make this default too?
If we are not confident, the default may still false until v2.1.

Copy link
Contributor Author

@norio-nomura norio-nomura Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have never seen an example of CI failure due to "SSH over VSOCK" so far.

The fact that the VM cannot be restarted when testing LIMA_SSH_OVER_VSOCK in CI is also failed, but the ssh communication itself is possible.

So, I think it's okay to leave true as the default value of LIMA_SSH_OVER_VSOCK.

@norio-nomura
Copy link
Contributor Author

Since SSH is working with direct IP on CI, it should change LIMA_DIRECT_IP_ACCESS to affect only port forwarding via direct IP.
Sorry, I'll change to that.

…ost OS.

Guest IP address will be detected in requirement process.

`hostagent.go`:
- Add fields to `HostAgent`
  - `guestIfnameOnSameSubnetAsHost string`
  - `guestIPv4 net.IP`
  - `guestIPv6 net.IP`
- Add methods `GuestIP()`, `GuestIPv4()`, and `GuestIPv6()` to `HostAgent`
- Add `HostAgent.WriteSSHConfigFile()` helper to write SSHConfigFile
- Add `HostAgent.sshAddressPort()` helper to provide ipAddress and port for SSH

`requirements.go`:
- Add `stdoutParser func(string) error` field to `requirement`
- Add `HostAgent.detectGuestIfnameOnSameSubnetAtHost()` to parse `ip -j neighbor` command output
- Add `HostAgent.detectGuestIPAddress()` to parse `ip -j addr` command output
- Add two requirements to `HostAgent.essentialRequirements()`
  - "detect guest interface on same subnet as the host"
  - "detect guest IPv4 address"
  - "detect guest IPv6 address"

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
Update to use guestIPAddress:
- `limactl shell`
- `limactl show-ssh`
- `limactl tunnel`

pkg/limatype:
- Add `GuestIP net.IP` to `Instance`
- Add `Instance.SSHAddressPort()` helper to provide ipAddress and port for SSH

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
Support printing "Guest IP Address: %s" on `limactl start`

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>

# Conflicts:
#	pkg/hostagent/requirements.go
…r than "127.0.0.1"

affected functions:
- Copy to host
- Reverse SSHFS
- SSH port forwarding

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
Change to use `github.com/inetaf/tcpproxy`:
- pkg/driver/vz/vsock_forwarder.go
- pkg/portfwd/client.go
- pkg/portfwdserver/server.go

pkg/portfwd:
- Use `dialContext` instead of `client` to use `net.Conn` other than `GrpcClientRW`.
- Change to create proxies from `dialContext` parameters instead of `conn`.
- Add `DialContextToGRPCTunnel()` to return `DialContext` function that connects to GRPC tunnel.

pkg/hostagent:
- Add `HostAgent.DialContextToGuestIP()` to return `DialContext` function that connects to the guest IP directly.
  If the guest IP is not known, it returns nil.
- Prefer `HostAgent.DialContextToGuestIP()` over `portfwd.DialContextToGRPCTunnel()`.

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>

pkg/hostagent: Use `HostAgent.DialContextToGuestIP()` if the IP is accessible directly.

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>

Revert to "github.com/containers/gvisor-tap-vsock/pkg/tcpproxy"

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>

# Conflicts:
#	pkg/hostagent/hostagent.go

pkg/hostagent: Aware forwarding guest address is IPv4 or IPv6

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>

pkg/hostagent: Fallback to GRPC forwarder if dialing to a direct ip fails

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
…work=vzNAT`

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
To following tests will be run with enabled.

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
…rt-forwarding.pl`

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
…ariable to enable direct IP port forwarding to the VM.

### Direct IP Port Forwarding

To enable direct IP port forwarding, set the `_LIMA_DIRECT_IP_PORT_FORWARDER` environment variable to `true`:

```bash
export _LIMA_DIRECT_IP_PORT_FORWARDER=true
```
This feature makes Lima to use direct IP port forwarding instead of gRPC port forwarding.
When this feature is enabled, Lima tries to connect to the guest's IP address directly for port forwarding.

#### Fallback to gRPC Port Forwarding
Lima may fall back to gRPC port forwarding in the following cases:
- If the guest's IP address is not available, Lima falls back to gRPC port forwarding.
- If the guest's IP address is available but the connection to the guest's IP address fails, Lima also falls back to gRPC port forwarding.
- If the guest's IP address is not accessible from the host (e.g. localhost on guest), Lima falls back to gRPC port forwarding as well.

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch from 4938689 to d2d9cda Compare October 17, 2025 01:41
…sts other than macOS

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Report VM IP addresses to host agent

2 participants