fix(bin): replace lsof with /dev/tcp in server check_port; remove dea…#3105
fix(bin): replace lsof with /dev/tcp in server check_port; remove dea…#3105bitflicker64 wants to merge 4 commits into
Conversation
…d check_port from pd/store lsof -i :PORT scans file descriptors up to ulimit -n. In kind/k8s pods ulimit -n can be ~1e9, causing lsof to peg CPU and never finish, blocking server startup. Replace with bash /dev/tcp which queries the kernel socket table directly and is unaffected by the fd limit. check_port in pd and store was never called by their startup scripts (confirmed with maintainer), so remove it as dead code. Also fix command_available in pd/store where command substitution with >/dev/null 2>&1 always produced an empty result, making the function always return "available" regardless of whether the command existed. This affected get_ip and download in those files. Signed-off-by: Himanshu Verma <himnshuverma10152006@gmail.com>
dosubot
Bot
added
size:XS
`lsof -i :PORT` can become extremely slow when the file descriptor limit is very large, as seen in some kind/Kubernetes environments, blocking server startup. Replace it with Bash `/dev/tcp` to probe the configured address without invoking `lsof`. Parse the host from `restserver.url` and `gremlinserver.url` so the probe targets the configured bind address instead of always using `127.0.0.1`. Handle IPv4, bracketed IPv6 addresses such as `[::1]`, and wildcard addresses by probing their corresponding loopback address. Remove unused `check_port` implementations from PD and Store, since their startup scripts never call them. Also fix `command_available` in PD and Store. Its previous command substitution discarded redirected output and could incorrectly report missing commands as available, affecting `get_ip` and `download`. Signed-off-by: Himanshu Verma <himnshuverma10152006@gmail.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3105 +/- ##
============================================
- Coverage 37.43% 32.73% -4.71%
+ Complexity 520 498 -22
============================================
Files 808 819 +11
Lines 69581 70745 +1164
Branches 9165 9366 +201
============================================
- Hits 26048 23158 -2890
- Misses 40750 45019 +4269
+ Partials 2783 2568 -215 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@copilot review |
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The new probe can miss or stall on real port conflicts, and the corrected command detection exposes a broken curl-only path. Evidence: static review of 6395c0c; bash -n passed; codecov/project is failing on this head.
| if [ $(command -v $cmd >/dev/null 2>&1) ]; then | ||
| return 1 | ||
| else | ||
| if [[ -x "$(command -v "$cmd")" ]]; then |
There was a problem hiding this comment.
command_available makes the curl-only download() branch reachable, but that branch still uses -o ${path}/${link_url}. Its callers pass a full https://.../opentelemetry-javaagent.jar URL, so curl tries to write through nonexistent nested https:/... directories and telemetry startup still fails when wget is absent; the Store copy has the same behavior. Please fix both helpers to use a basename destination such as curl -L "$link_url" -o "$path/$(basename "$link_url")" and cover the curl-only path.
| fi | ||
|
|
||
| # Wildcard binds include loopback: normalize to loopback for the probe | ||
| if [[ -z "$host" || "$host" == "0.0.0.0" ]]; then |
There was a problem hiding this comment.
0.0.0.0 to 127.0.0.1 only probes loopback. A process listening on another local interface can make the loopback connect fail while the subsequent wildcard Java bind still fails with EADDRINUSE; the :: to ::1 path has the same multi-interface gap. Please make wildcard checks cover the actual bind scope (for example through the kernel listening table or all local addresses) and add IPv4/IPv6 regressions with a non-loopback listener.
| # Use bash /dev/tcp instead of lsof to avoid the FD-table walk that lsof | ||
| # performs under inflated nofile limits (e.g. kind / some Kubernetes runtimes), | ||
| # which pegs CPU and prevents Java from starting. | ||
| if (echo >/dev/tcp/"$host"/"$port") >/dev/null 2>&1; then |
There was a problem hiding this comment.
/dev/tcp has no deadline here. For a configured local/Pod address whose firewall or CNI drops SYN packets, this check waits for the kernel retry timeout and can still stall startup—the failure mode this change is intended to remove. Please use a probe with an explicit short timeout or inspect listening sockets without initiating a network connection, and cover the timeout path.
| # Use bash /dev/tcp instead of lsof to avoid the FD-table walk that lsof | ||
| # performs under inflated nofile limits (e.g. kind / some Kubernetes runtimes), | ||
| # which pegs CPU and prevents Java from starting. | ||
| if (echo >/dev/tcp/"$host"/"$port") >/dev/null 2>&1; then |
There was a problem hiding this comment.
🧹 echo writes a newline to whichever service already owns the port, although detecting a listener only requires opening the connection. This can create parse errors, logs, or error metrics in the existing REST/Gremlin service. Please open and close the descriptor without payload, for example with : >/dev/tcp/"$host"/"$port".
Refactors `check_port` and `download` functions to resolve startup hangs and incorrect file path generation.
check_port (server):
- Replace `lsof -i :PORT` with a layered probe to prevent startup hangs in Kubernetes/kind pods. Environments with a high ulimit (e.g., ~1e9) caused lsof to scan all file descriptors, pegging the CPU indefinitely.
- Implement a new sequential probe order:
1. `ss -ltn`: Queries the kernel socket table directly without an FD scan or timeout risk.
2. `netstat -ltn`: Serves as a fallback for systems missing the `ss` utility.
3. `bash /dev/tcp` with a 1-second timeout: Used as a last resort. Utilizes the `:` null command instead of `echo` to avoid sending a newline payload to existing services. The timeout protects against SYN-drop stalls on CNI or firewall paths.
download (pd, store):
- Fix the curl-only logic branch that incorrectly attempted to write to ${path}/${link_url}, resulting in the creation of nonexistent nested URL directories.
- Change the command to `curl -L "$link_url" -o "$path/$(basename "$link_url")"` to align with the correct implementation used by the server.
- Note: This branch was previously unreachable due to a bug in `command_available`. Fixing `command_available` exposed this pathing issue.
Signed-off-by: Himanshu Verma <himnshuverma10152006@gmail.com>
The macOS netstat fallback used (\.|:)${port}\b, which could match the port number inside an IP address octet (e.g. .80 in 192.168.80.1:443), resulting in a false "port in use" error.
Replace \b with ([[:space:]]|$) so the port only matches when followed by whitespace or the end of the line.
Also document the unguarded /dev/tcp fallback used when timeout is unavailable. Recommend installing coreutils to provide timeout and avoid potential stalls on firewalled ports.
Signed-off-by: Himanshu Verma <himnshuverma10152006@gmail.com>
2 participants
Purpose of the PR
Fix a startup hang in kind/Kubernetes environments caused by
lsofscanning an enormous file descriptor table.Main Changes
The startup script (
start-hugegraph.sh) callscheck_portbefore launching Java to verify the target port is free. The original implementation usedlsof -i :PORT.In kind/k8s pods,
ulimit -n(max open file descriptors) can be set to ~1 billion.lsoftries to scan all possible file descriptors up to that limit, pegging CPU at 100% and never completing — blocking server startup entirely. Plain Docker with the same image is unaffected because its defaultulimit -nis ~1024.hugegraph-server/.../util.shlsof -i :PORTwith Bash's/dev/tcppseudo-device. This checks whether the port accepts a TCP connection without invokinglsof, avoiding the expensive file-descriptor scan under very largenofilelimits and removing the dependency on the externallsofbinary.hugegraph-pd/.../util.shandhugegraph-store/.../util.shcheck_portentirely — it is never called by the PD or Store startup scripts .command_available: the original implementation used command substitution while redirecting the command output to/dev/null, so the substitution always produced an empty result. This caused the function to report commands as available regardless of whether they existed, affectingget_ipanddownload.Verifying these changes
ulimit -nset to a large value (e.g. 1e9). Confirm the server starts successfully without hanging at the port check step.Does this PR potentially affect the following parts?
Documentation Status
Doc - No Need