Skip to content

feat(local-dev): dev tool local-dev support Linux #7077

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Yicong-Huang wants to merge 1 commit into
base: main
Choose a base branch
from
+392 −18
Open

feat(local-dev): dev tool local-dev support Linux #7077

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions bin/local-dev/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,40 @@ bin/local-dev.sh --help # full reference
Everything in this folder is wired up by the wrapper at `bin/local-dev.sh`;
you should not need to invoke any file inside `bin/local-dev/` directly.

## Supported platforms

macOS and Linux. Where the two differ, `main.sh` picks the right probe by
platform — there is nothing to configure:

| Concern | macOS | Linux |
| --- | --- | --- |
| Host LAN IP (the MinIO endpoint) | `route get default`, `ipconfig getifaddr` | `ip route show default`, `ip -4 addr show scope global` |
| Artifact mtime (`watch`'s ARTIFACT MTIME column) | BSD `stat -f` | GNU `stat -c` |
| Port → PID | `lsof` | `lsof`, falling back to `ss` |

On top of the toolchain in [AGENTS.md](../../AGENTS.md) — JDK 17, Scala 2.13,
Python 3.12, Node 24 — plus sbt and docker, Linux needs:

* **iproute2** (`ip`, `ss`). Essential on every mainstream distro, so normally
already installed. `ip` is required: without it the host LAN IP cannot be
detected and `up` refuses to start rather than publish an endpoint that only
works from one side.
* **docker with the compose v2 plugin.** On stock Ubuntu that is
`apt install docker.io docker-compose-v2`; `docker-compose-plugin` only
exists in Docker's own apt repository.
* **your user in the `docker` group** — `sudo usermod -aG docker "$USER"`,
then log in again. Group membership does not apply to shells that are
already running, so a `docker compose` permission error right after that
command is expected.

`HOST_LAN_IP` is detected for you; export it only to override the choice, e.g.
to pin one interface on a multi-homed host. The address has to be reachable
from **both** the host JVMs and the containers, which is why the scan skips
container bridges (`docker0`, `br-*`, `veth*`) and overlay/VPN interfaces
(tailscale, zerotier, wireguard): `172.17.0.1` looks like a perfectly good
local IPv4 but is not reachable from inside another container's network
namespace.

## Layout

```
Expand Down
122 changes: 108 additions & 14 deletions bin/local-dev/main.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -462,22 +462,61 @@ fi
# `localhost` only works for the host. `texera-minio` only works inside the
# docker network. The host's LAN IP works from BOTH (host loopback for the
# host, docker NAT'd out-and-back for the container).
_detect_host_lan_ip() {
#
# Both platform probes follow the same two steps: the interface backing the
# default route first (most reliable on a laptop that may have wifi +
# thunderbolt + tailscale all active), then a scan as a fallback.
_detect_host_lan_ip_darwin() {
local iface="" ip=""
# 1. The interface backing the default route — most reliable on a
# laptop that may have wifi + thunderbolt + tailscale all active.
iface=$(route get default 2>/dev/null | awk '/interface:/{print $2; exit}')
if [[ -n "$iface" ]]; then
ip=$(ipconfig getifaddr "$iface" 2>/dev/null)
[[ -n "$ip" && "$ip" != 127.* ]] && { printf '%s\n' "$ip"; return 0; }
fi
# 2. Fallback: linux `hostname -I`-equivalent walk over en*.
for iface in en0 en1 en2 en3 en4 en5 en6 en7 en8 en9 en10; do
ip=$(ipconfig getifaddr "$iface" 2>/dev/null)
[[ -n "$ip" && "$ip" != 127.* ]] && { printf '%s\n' "$ip"; return 0; }
done
return 1
}

_detect_host_lan_ip_linux() {
command -v ip >/dev/null 2>&1 || return 1
local iface="" addr="" idx="" dev="" fam="" cidr=""
# 1. Interface backing the default route — take its first global IPv4,
# stripping the /prefix that `ip -o addr` appends.
iface=$(ip -4 route show default 2>/dev/null \
| awk '{ for (i = 1; i < NF; i++) if ($i == "dev") { print $(i+1); exit } }')
if [[ -n "$iface" ]]; then
addr=$(ip -4 -o addr show dev "$iface" scope global 2>/dev/null \
| awk '{ split($4, a, "/"); print a[1]; exit }')
[[ -n "$addr" && "$addr" != 127.* ]] && { printf '%s\n' "$addr"; return 0; }
fi
# 2. Scan every global IPv4, skipping the interfaces that would defeat the
# purpose of this address. A container bridge (docker0, br-*, veth*) is
# reachable from the host but not from inside another container's
# network namespace the way MinIO needs; an overlay/VPN address
# (tailscale, zerotier) is not reachable from the docker bridge at all.
while read -r idx dev fam cidr _rest; do
[[ "$fam" == "inet" ]] || continue
case "$dev" in
lo|docker*|br-*|bridge*|virbr*|veth*|tun*|tap*|cni*|flannel*|cali*|kube*|tailscale*|zt*|wg*)
continue ;;
esac
addr="${cidr%%/*}"
[[ -n "$addr" && "$addr" != 127.* ]] && { printf '%s\n' "$addr"; return 0; }
done < <(ip -4 -o addr show scope global 2>/dev/null)
return 1
}

_detect_host_lan_ip() {
case "$(uname -s 2>/dev/null)" in
Darwin) _detect_host_lan_ip_darwin ;;
Linux) _detect_host_lan_ip_linux ;;
# Anything else (BSD, WSL oddities): try both rather than give up.
*) _detect_host_lan_ip_darwin || _detect_host_lan_ip_linux ;;
esac
}
# Lazy resolver — called from subcommands that actually need to publish a
# host-reachable S3 endpoint (cmd_up, cmd_auto). Subcommands like
# `version`, `status`, `--help`, or `-i` don't talk to MinIO and shouldn't
Expand All @@ -486,10 +525,17 @@ _require_host_lan_ip() {
[[ -n "${HOST_LAN_IP:-}" ]] && return 0
HOST_LAN_IP="$(_detect_host_lan_ip)" || HOST_LAN_IP=""
if [[ -z "$HOST_LAN_IP" ]]; then
local probes=""
case "$(uname -s 2>/dev/null)" in
Darwin) probes="\`route get default\` / en0-en10" ;;
Linux) probes="\`ip route show default\` / \`ip -4 addr show scope global\`" ;;
*) probes="the macOS and Linux probes" ;;
esac
echo "FATAL: could not detect a host LAN IP." >&2
echo " MinIO needs an address reachable from both docker (lakekeeper" >&2
echo " does S3 ops) and the host (JVMs read signed URLs back); none" >&2
echo " of \`route get default\` / en0-en10 had a non-loopback IPv4." >&2
echo " of $probes offered a non-loopback IPv4" >&2
echo " outside the container bridges." >&2
echo " Connect to a network or export HOST_LAN_IP=<your-IP> explicitly." >&2
exit 1
fi
Expand Down Expand Up @@ -1133,20 +1179,23 @@ _install_hint() {
node)
printf " ${BOLD}install Node 20+ (needed for frontend & agent-service):${RESET}\n"
printf " macOS: brew install node\n"
printf " Linux: use a version manager below — distro packages\n"
printf " (apt/dnf nodejs) are older than the frontend needs\n"
printf " nvm: nvm install --lts && nvm use --lts\n"
printf " fnm: fnm install --lts\n"
printf " volta: volta install node\n"
;;
yarn)
printf " ${BOLD}install yarn (needed for the frontend):${RESET}\n"
printf " any OS: corepack enable ${DIM}# ships with Node; frontend pins yarn@4${RESET}\n"
printf " macOS: brew install yarn\n"
printf " npm: npm install -g yarn\n"
printf " corepack: corepack enable\n"
printf " Linux: npm install -g yarn\n"
;;
bun)
printf " ${BOLD}install bun (needed for agent-service):${RESET}\n"
printf " macOS: brew install oven-sh/bun/bun\n"
printf " curl: curl -fsSL https://bun.sh/install | bash\n"
printf " Linux: curl -fsSL https://bun.sh/install | bash\n"
printf " npm: npm install -g bun\n"
;;
sbt)
printf " ${BOLD}install sbt (needed to build the JVM services):${RESET}\n"
Expand All @@ -1156,7 +1205,8 @@ _install_hint() {
docker)
printf " ${BOLD}install Docker (needed for postgres/minio/lakefs/lakekeeper/litellm):${RESET}\n"
printf " macOS: download Docker Desktop from https://docker.com/products/docker-desktop\n"
printf " Linux: apt install docker.io docker-compose-plugin\n"
printf " Linux: apt install docker.io docker-compose-v2 ${DIM}# dnf: moby-engine docker-compose${RESET}\n"
printf " then sudo usermod -aG docker \"\$USER\" and log back in\n"
;;
*)
printf " ${DIM}no install hint for: %s${RESET}\n" "$tool"
Expand Down Expand Up @@ -1194,9 +1244,53 @@ _diagnose_node() {
}

# --------- helpers ---------
# PID of whatever is listening on a TCP port, or nothing. Always exits 0 —
# "empty output" is the contract for "nothing is listening", and callers
# (svc_running_pid, wait_for_port, the status table) rely on that.
#
# lsof first: it ships with macOS and is what this has always used. But it is
# not a default package on Debian/Ubuntu/Fedora, and when it is missing every
# native service reads as stopped — `up` then relaunches services that are
# already running and `down` silently no-ops. So fall back to `ss` from
# iproute2, which is essential on any modern Linux.
listen_pid_for_port() {
# || true so pipefail doesn't kill us when nothing is listening
lsof -nP -iTCP:"$1" -sTCP:LISTEN -t 2>/dev/null | head -1 || true
local port="$1" pid=""
if command -v lsof >/dev/null 2>&1; then
# || true so pipefail doesn't kill us when nothing is listening
pid=$(lsof -nP -iTCP:"$port" -sTCP:LISTEN -t 2>/dev/null | head -1 || true)
[[ -n "$pid" ]] && { printf '%s\n' "$pid"; return 0; }
fi
if command -v ss >/dev/null 2>&1; then
# `ss -lntp` prints e.g.
# LISTEN 0 511 127.0.0.1:8080 0.0.0.0:* users:(("java",pid=4827,fd=9))
# Match on the local-address column ending in :<port> so :18080 doesn't
# answer for :8080. No -H: it postdates the iproute2 in older distros.
pid=$(ss -lntp 2>/dev/null \
| awk -v suffix=":$port" '$4 ~ suffix"$" { print; exit }' \
| grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2 || true)
[[ -n "$pid" ]] && { printf '%s\n' "$pid"; return 0; }
fi
return 0
}

# Modification time of a file as "YYYY-MM-DD HH:MM", in whichever stat dialect
# is present. Non-zero and silent if the file is missing or unreadable, so
# callers can substitute their own placeholder.
#
# GNU coreutils and BSD disagree completely here: BSD's `stat -f FMT -t TIMEFMT`
# reads `-f` as "file system info" under GNU, which then treats the format
# strings as filenames — it writes a diagnostic to stderr, prints filesystem
# fields on stdout and exits 1. Probe rather than branch on `uname` so a
# GNU-coreutils macOS also works.
_file_mtime_str() {
local f="${1:-}"
[[ -n "$f" && -e "$f" ]] || return 1
if stat -c '%y' "$f" >/dev/null 2>&1; then
# GNU: "2026-07-29 20:06:12.123456789 +0000" → keep date + HH:MM
stat -c '%y' "$f" 2>/dev/null | cut -c1-16
else
stat -f "%Sm" -t "%Y-%m-%d %H:%M" "$f" 2>/dev/null
fi
}

# Branch + short-sha of the deploy source ($REPO_ROOT), tab-separated, each
Expand Down Expand Up @@ -1671,14 +1765,14 @@ svc_artifact_mtime() {
done <<< "$globbed"
fi
if [[ ${#main_jars[@]} -gt 0 ]]; then
stat -f "%Sm" -t "%Y-%m-%d %H:%M" "${main_jars[0]}"
_file_mtime_str "${main_jars[0]}" || echo "—"
return
fi
fi
echo "—"
;;
bun) stat -f "%Sm" -t "%Y-%m-%d %H:%M" "$(amap_get SVC_CWD "$svc")/bun.lock" 2>/dev/null || echo "—" ;;
yarn) stat -f "%Sm" -t "%Y-%m-%d %H:%M" "$(amap_get SVC_CWD "$svc")/yarn.lock" 2>/dev/null || echo "—" ;;
bun) _file_mtime_str "$(amap_get SVC_CWD "$svc")/bun.lock" || echo "—" ;;
yarn) _file_mtime_str "$(amap_get SVC_CWD "$svc")/yarn.lock" || echo "—" ;;
docker) echo "—" ;;
esac
}
Expand Down
Loading
Loading