Lima get stuck when certain frequency of DNS resolution occurs

[matsukaz]

Description

Actual Behavior

When I continue to execute nslookup/ping/dig or any other DNS name resolution operations within Lima, Lima get stuck after a certain number of times (lasting about 4 minutes).
This issue also occurs inside Docker container when using Rancher Desktop, so this is critial issue for any system that uses container linking, like Apache+PHP and MySQL.

Steps to Reproduce

1. Login to Lima and keep running nslookup.

$ rdctl shell
lima-rancher-desktop:/Users/xxx$ while true; do nslookup www.google.co.jp; done

2. While the above process is running, show a list of UDP open files that qemu-system-aarch64 handles on host OS.

$ lsof -p $(pgrep qemu-system-aarch64) | grep "UDP"
...
qemu-syst 6788 xxxx  119u  IPv4 0x2c6ecf140850ff5f         0t0                 UDP *:63544
qemu-syst 6788 xxxx  120u  IPv4 0x2c6ecf140851762f         0t0                 UDP *:63398

3. A number of UDP open files keep increasing and after it reaches to FD=1024u, Lima get stuck.

$ lsof -p $(pgrep qemu-system-aarch64) | grep "UDP"
...
qemu-syst 6788 xxxx  1023u  IPv4 0x2c6ecf14085191bf         0t0                 UDP *:54486
qemu-syst 6788 xxxx  1024u  IPv4 0x2c6ecf140852088f         0t0                 UDP *:62934

4. If you wait about 4 minutes, all UDP open files get released and Lima starts running again.

Workaround

If you set hostResolver.enabled=false , this issue doesn't occur.

hostResolver:
  enabled: false

Versions

• Lima 0.14.2
• Rancher Desktop 1.7.0

Operating System / Build Version / CPU

• MacOS Monterey 12.6 (M1 2020)
• MacOS Ventura 13.0.1 (Intel Core i5, 2019)

[jandubois]

I know we looked into this before; @Nino-K do you remember the current status?

[matsukaz]

@Nino-K
Thanks for trying to reproduce this issue.
Looks like this behavior occurs when using debian and alpine images used as example.
Ubuntu, which is used as default instance, didn't solve this problem when using workaround.

debian

images:
- location: "https://cloud.debian.org/images/cloud/bullseye/20221205-1220/debian-11-generic-amd64-20221205-1220.qcow2"
  arch: "x86_64"
  digest: "sha512:25c6f2b5d61b0897077533023cccb05badb2d3fd5adc9ca9071fb8f1fe5ffe682a09f12279ef8013dc5641d20ff16eedc09e3b4657d81bcf455bdb9b4cd8ba54"
- location: "https://cloud.debian.org/images/cloud/bullseye/20221205-1220/debian-11-generic-arm64-20221205-1220.qcow2"
  arch: "aarch64"
  digest: "sha512:9d09c70030bbbc4680f4ada358a2fc3e57f73d37789384a4230d4f374629a3edc04982ac67423e8dc4531cba9abf5f89891a974d51ebc8f48bfc626d4171f09b"

alpine

images:
- location: "https://github.com/lima-vm/alpine-lima/releases/download/v0.2.26/alpine-lima-std-3.17.0-x86_64.iso"↲
  arch: "x86_64"
  digest: "sha512:93bf8d52ad2b3a6ef401a1a565f67e4f06f8022ec8963cad36cac4a25253f056ac0b755d1ed56b8993b0261440e0d5d38bad15b271864ed33cd3b02272738672"
- location: "https://github.com/lima-vm/alpine-lima/releases/download/v0.2.26/alpine-lima-std-3.17.0-aarch64.iso"↲
  arch: "aarch64"
  digest: "sha512:aacd3a9e7a3231553bcee879c6770b379afbfad3dcfac7f274a80d6c490faf33f3c23b6130efa881f57c0c5235ced2877e2f45e3cdf4fefff11531d6023ae214"

[matsukaz]

Following is the full yaml file I used to reproduce this issue.

images:
- location: "https://github.com/lima-vm/alpine-lima/releases/download/v0.2.26/alpine-lima-std-3.17.0-x86_64.iso"
  arch: "x86_64"
  digest: "sha512:93bf8d52ad2b3a6ef401a1a565f67e4f06f8022ec8963cad36cac4a25253f056ac0b755d1ed56b8993b0261440e0d5d38bad15b271864ed33cd3b02272738672"
- location: "https://github.com/lima-vm/alpine-lima/releases/download/v0.2.26/alpine-lima-std-3.17.0-aarch64.iso"
  arch: "aarch64"
  digest: "sha512:aacd3a9e7a3231553bcee879c6770b379afbfad3dcfac7f274a80d6c490faf33f3c23b6130efa881f57c0c5235ced2877e2f45e3cdf4fefff11531d6023ae214"

mounts:
- location: "~"
- location: "/tmp/lima"
  writable: true

containerd:
  system: false
  user: false

hostResolver:
  enabled: false

[Nino-K]

@matsukaz thanks for reporting this again, the PR #1294 should address this issue. I believe the core of the issue was we left the response writer open after serving the query. This was the case for both TCP (connected sockets) and UDP (non-connected sockets). Although, there is no real connection in UDP, closing the response writer still allows the callee (the os stack) to release the file descriptor and not hold on to it.