Skip to content

feat(local-dev): dev tool local-dev support Linux - #7077

Open
Yicong-Huang wants to merge 1 commit into
apache:mainfrom
Yicong-Huang:fix/local-dev-linux
Open

feat(local-dev): dev tool local-dev support Linux #7077
Yicong-Huang wants to merge 1 commit into
apache:mainfrom
Yicong-Huang:fix/local-dev-linux

Conversation

@Yicong-Huang

@Yicong-Huang Yicong-Huang commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Makes bin/local-dev.sh work on Linux, not just macOS.

The script already carried some Linux support — _find_jdk17 globs /usr/lib/jvm/*, SDKMAN and asdf are probed, the docker install hint has a Linux: line — but the three platform probes underneath it were macOS-only. The first was fatal, the other two degraded silently.

1. Host LAN IP detection (the hard blocker). _detect_host_lan_ip probed route get default and ipconfig getifaddr en0..en10. Neither exists on Linux: route is net-tools with different syntax, ipconfig is macOS/Windows, and the interfaces are eth0/enp*. So _require_host_lan_ip aborted every up/auto:

Before:  linux + up -> FATAL: could not detect a host LAN IP
After:   linux + up -> LAN IP from `ip route` / `ip -4 addr`, stack comes up

Split into _detect_host_lan_ip_darwin / _detect_host_lan_ip_linux behind a uname dispatcher, so the macOS path is byte-for-byte what it was. The Linux probe follows the same two steps — interface backing the default route first, then a scan — and the scan's exclusion list is the interesting part: this address exists to be reachable from both the host JVMs and the lakekeeper container, so a container bridge (docker0, br-*, veth*) is wrong even though it is a perfectly good local IPv4, and an overlay/VPN address (tailscale, zerotier, wireguard) is not reachable from the docker bridge at all. On a typical box hostname -I prints 10.10.10.30 172.17.0.1, and picking the second would defeat the whole point of the variable.

The FATAL message also named only the macOS probes; on Linux none of them had run, which made the message actively misleading. It now names the probes that actually executed.

2. Artifact mtime. svc_artifact_mtime used BSD stat -f "%Sm" -t FMT. GNU coreutils reads -f as "file system info" and the format strings as filenames — it writes a diagnostic to stderr, prints filesystem fields on stdout and exits 1 — so watch's ARTIFACT MTIME column rendered that garbage (the JVM call site had no 2>/dev/null guard at all). Replaced with _file_mtime_str, which probes for the dialect rather than branching on uname, so GNU coreutils on macOS also works.

3. lsof dependency. listen_pid_for_port required lsof. It ships with macOS but is not a default package on Debian/Ubuntu/Fedora, and without it every native service read as stoppedup relaunched services that were already running, and down silently no-opped. Falls back to ss from iproute2, matching on the local-address column so :18080 doesn't answer for :8080. -H is deliberately not used (it postdates the iproute2 in older distros), and the "empty output means nothing is listening" contract that svc_running_pid / wait_for_port / the status table rely on is preserved.

Two smaller _install_hint fixes: the docker line named docker-compose-plugin, which only lives in Docker's own apt repo rather than stock Ubuntu (docker-compose-v2), and the node / yarn / bun cases had no Linux: line at all.

Docs. bin/local-dev/README.md gains a Supported platforms section: a table of which probe is used on which platform, the Linux prerequisites (iproute2, docker + compose v2, and the docker-group re-login that trips everyone up once), and why a docker-bridge address is not a valid HOST_LAN_IP even though it looks like a fine local IPv4. The bash suite's header comment also stopped claiming a real stack "needs a Mac", and now states that platform-specific checks report a skip rather than a pass.

Any related issues, documentation, discussions?

Closes #7065

How was this PR tested?

New coverage in the existing infra-job suite. The LAN-IP cases run against a fake ip so the assertions don't depend on the test machine's interfaces, and the port-listener cases run against a real listener with lsof shimmed out:

$ bash bin/local-dev/tests/test_local_dev_sh.sh
...
  ✓ linux LAN IP: default route interface wins → 10.10.10.30
  ✓ linux LAN IP: no default route: skips the docker bridge → 10.10.10.30
  ✓ linux LAN IP: no default route: skips bridges, veth and tailscale → 192.168.1.42
  ✓ linux LAN IP: default route interface has no global v4: falls back to scan → 192.168.1.42
  ✓ linux LAN IP: nothing but loopback (refuses)
  ✓ linux LAN IP: only excluded interfaces (refuses)
  ✓ linux LAN IP: no `ip` on PATH (refuses)
  ✓ _detect_host_lan_ip dispatches per platform
  ✓ host-LAN-IP failure message is not macOS-only
  ✓ _file_mtime_str formats YYYY-MM-DD HH:MM (2026-07-29 22:17)
  ✓ _file_mtime_str refuses a missing file quietly
  ✓ _file_mtime_str refuses a missing argument quietly
  ✓ BSD stat is confined to _file_mtime_str
  ✓ listen_pid_for_port finds the listener (46263 → 193102)
  ✓ listen_pid_for_port works without lsof (193102)
  ✓ listen_pid_for_port: unused port yields nothing
  ✓ docker hint names the distro compose package
  ✓ every install hint has a Linux line

68 passed, 0 failed
$ python -m pytest bin/local-dev/tests/ -q
1 failed, 42 passed

That failure is test_is_dirty_after_seed_then_edit, pre-existing and unrelated — it reproduces identically on a pristine upstream/main checkout on this machine, and this PR touches neither tui.py nor that test. Filed as #7075 and fixed there.

End-to-end on Ubuntu 24.04.4 / x86_64, docker 29.1.3, with HOST_LAN_IP deliberately not exported — i.e. the exact case that used to abort:

[step] env HOST_LAN_IP='<unset>'
[step] down
[rc] down=0
[step] up (unfiltered)
...
  ✓  postgres                         :5432    healthy
  ✓  minio                            :9000    healthy
  ✓  lakefs                           :8000    healthy
  ✓  lakekeeper                       :8181    healthy
  ✓  litellm                          :4000    healthy
  ✓  config-service                   :9094    healthy
  ✓  access-control-service           :9096    healthy
  ✓  file-service                     :9092    healthy
  ✓  workflow-compiling-service       :9090    healthy
  ✓  computing-unit-master            :8085    healthy
  ✓  computing-unit-managing-service  :8082    healthy
  ✓  texera-web                       :8080    healthy
  ✓  agent-service                    :3001    healthy
  ✓  frontend                         :4200    healthy

  ✓ 14 of 14 services healthy
[rc] up=0

The detected address matches the kernel's own answer and is reachable at the MinIO port, which is what the variable is for:

detected=10.10.10.30
$ ip -4 route get 1.1.1.1 | awk '{print $7; exit}'
10.10.10.30
$ curl http://10.10.10.30:9000/minio/health/live   ->  HTTP 200

lsof removed from PATH (shimmed to exit 127) now reports the same state rather than "everything stopped" — same shell, same PIDs, so the comparison is apples to apples:

########## status: real lsof ##########
  ● config-service   9094  150646  running
  ● texera-web       8080  151007  running
  Status: 14 of 14 services running

########## status: lsof shimmed out (exit 127) ##########
  ● config-service   9094  150646  running
  ● texera-web       8080  151007  running
  Status: 14 of 14 services running

And watch's ARTIFACT MTIME column renders a timestamp instead of GNU stat's diagnostics:

    SERVICE                          PORT    PID       ARTIFACT MTIME     SRC STATE
  ● config-service                  :9094   150646    2026-07-29 20:33       running
  ● access-control-service          :9096   150670    2026-07-29 20:33       running

macOS is unchanged by construction: the Darwin probe body is untouched, and a test asserts the dispatcher still routes to it. I have no Mac to run the suite on, so CI's infra (macos-latest) job is what covers that side. The checks that can only hold on one platform — the ss fallback in particular, since ss is Linux-only and macOS ships lsof in the base system — are guarded and report a skip there rather than a pass.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

`bin/local-dev.sh` was macOS-only in three places. It already carried some
Linux support — `_find_jdk17` globs `/usr/lib/jvm/*`, SDKMAN and asdf are
probed, the docker install hint has a `Linux:` line — but the platform probes
underneath were not, and the first of them was fatal.

`_detect_host_lan_ip` probed `route get default` and `ipconfig getifaddr
en0..en10`. Neither exists on Linux — `route` is net-tools with different
syntax, `ipconfig` is macOS/Windows, and the interfaces are `eth0`/`enp*` — so
`_require_host_lan_ip` aborted every `up`/`auto` there. Split the probe per
platform and add the iproute2 equivalent. The scan skips loopback and the
interfaces that would defeat the purpose of this address: a container bridge
(docker0, br-*, veth*) is reachable from the host but not from inside another
container's netns, and an overlay/VPN address (tailscale, zerotier) is not
reachable from the docker bridge at all. The FATAL now names the probes that
actually ran instead of always blaming the macOS ones.

`svc_artifact_mtime` used BSD `stat -f "%Sm" -t FMT`. GNU coreutils reads
`-f` as "file system info" and the format strings as filenames, so it wrote a
diagnostic to stderr, printed filesystem fields on stdout and exited 1 —
`watch`'s ARTIFACT MTIME column rendered that. `_file_mtime_str` probes for
the dialect rather than branching on `uname`, so GNU coreutils on macOS works
too.

`listen_pid_for_port` required lsof. It ships with macOS but is not a default
package on Debian/Ubuntu/Fedora, and without it every native service read as
stopped — `up` relaunched services that were already running and `down`
silently no-opped. Falls back to `ss` from iproute2, which is essential on
any modern Linux.

Also in `_install_hint`: the docker line named `docker-compose-plugin`, which
only exists in Docker's own apt repo rather than stock Ubuntu, and the node /
yarn / bun cases had no Linux line at all.

Documents the result in bin/local-dev/README.md: a per-platform table of which
probe is used where, the Linux prerequisites (iproute2, docker + compose v2,
the `docker` group re-login), and why a docker-bridge address is not a valid
HOST_LAN_IP even though it looks like one.

Closes apache#7065
@github-actions github-actions Bot added fix docs Changes related to documentations infra labels Jul 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Committers with relevant context: @parshimers
    You can request their reviews formally with /request-review @parshimers.

  • Contributors with relevant context: @aglinxinyuan
    You can notify them by mentioning @aglinxinyuan in a comment.

@Yicong-Huang Yicong-Huang changed the title feat(local-dev): support Linux feat(local-dev): dev tool local-dev support Linux Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Changes related to documentations fix infra

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bin/local-dev.sh cannot bring the stack up on Linux

2 participants