Skip to content

fix: instruction telemetry requires many file descriptors on high core-count systems #673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
harp-intel wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: instruction telemetry requires many file descriptors on high core-count systems #673

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions internal/script/scripts.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1492,6 +1492,8 @@ turbostat -i $interval $count
duration={{.Duration}}
pid={{.InstrMixPID}}

min_fd_limit=4096

cleanup_done=0

finalize() {
Expand All @@ -1514,6 +1516,38 @@ finalize() {
}
trap finalize INT TERM EXIT

# processwatch opens perf_event file descriptors per monitored CPU. On large
# systems, the default soft open-file limit can be too low and libbpf fails with
# "Too many open files" while attaching insn_collect.
cpu_count=$(getconf _NPROCESSORS_ONLN 2>/dev/null || nproc 2>/dev/null || echo 1)
required_fd_limit=$((cpu_count * 2 + 128))
if [ "$required_fd_limit" -lt "$min_fd_limit" ]; then
required_fd_limit=$min_fd_limit
fi

current_fd_limit=$(ulimit -Sn)
if [ "$current_fd_limit" = "unlimited" ]; then
current_fd_limit=$required_fd_limit
fi

if [ "$current_fd_limit" -lt "$required_fd_limit" ]; then
hard_fd_limit=$(ulimit -Hn)
if [ "$hard_fd_limit" = "unlimited" ] || [ "$hard_fd_limit" -ge "$required_fd_limit" ]; then
ulimit -Sn "$required_fd_limit" 2>/dev/null || true
fi
current_fd_limit=$(ulimit -Sn)
fi

if [ "$current_fd_limit" = "unlimited" ]; then
current_fd_limit=$required_fd_limit
fi

if [ "$current_fd_limit" -lt "$required_fd_limit" ]; then
echo "instruction telemetry requires at least $required_fd_limit open files for $cpu_count CPUs, but the current soft limit is $current_fd_limit" 1>&2
echo "increase the open file limit (ulimit -n) or reduce the number of monitored CPUs before retrying instruction telemetry" 1>&2
exit 1
fi

if [ $duration -ne 0 ] && [ $interval -ne 0 ]; then
count=$((duration / interval))
arg_count="-n $count"
Expand Down
Loading