Trilinos
Sample from tool showing Trilinos build target max RAM usage and other useful metrics #15303
bartlettroscoe
started this conversation in
Show and tell
Replies: 5 comments 5 replies
-
|
CC: @cgcgcg, @ccober6, @rppawlo, @sebrowne, @achauphan FYI: This is the tool and the data I mentioned at the Trilinos HPSF meeting yesterday. I can give limited access to the repo to access the tool inside of SNL. (We are waiting for the SNL copyright assertion and then it will be open-sourced on GitHub.) |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
That's pretty nice! I'll have a look at how to run it and get back to you. |
Beta Was this translation helpful? Give feedback.
-
|
CC: @cgcgcg, @ccober6, @rppawlo, @sebrowne, @achauphan FYI, I threw together a Jupyter Notebook with a container launch script and put it in the internal repo. The HTML file generated from that is attached. project_build_test_metrics_analysis.html Now we can run this type of analysis on any project with a single command. Lots of interesting info in this data (you really see which targets and which packages consume most resources). And there are Project Build Target and Test Process MetricsUse this notebook to analyze one build/test run captured by Run the setup cells first. Each report section below is its own Python cell so you can edit a computation, rerun one section, or insert your own cells between sections. The build summary JSON and build process JSONL are required. The test summary JSON and test process JSONL are optional when missing or when their filenames are set to empty strings. Input Settings
Legend
Equivalent jq/shelljq -n --arg project_name "$PROJECT_METRICS_PROJECT_NAME" \ --arg run_dir "$PROJECT_METRICS_RUN_DIR" \ --arg host_data_files_dir "$PROJECT_METRICS_HOST_DATA_FILES_DIR" \ --arg build_prefix "$PROJECT_METRICS_BUILD_OUTPUT_PREFIX" \ --arg package_regex "$PROJECT_METRICS_PACKAGE_PATH_REGEX" \ '{ project_name: $project_name, run_dir: $run_dir, host_data_files_dir: $host_data_files_dir, build_output_prefix: $build_prefix, package_path_regex: $package_regex }' Contents
Input Files
Legend
Equivalent jq/shell# Set these to the resolved paths shown in the table.for path in "$BUILD_SUMMARY_JSON" "$BUILD_PROCS_JSONL" "$TEST_SUMMARY_JSON" \ "$TEST_PROCS_JSONL" "$MAKE_OUT" "$CTEST_OUT"; do if [[ -f "${path}" && "${path}" == *.jsonl ]]; then printf '%s\t%s JSONL rows\n' "${path}" "$(jq -sc length "${path}")" elif [[ -f "${path}" ]]; then printf '%s\t%s bytes\n' "${path}" "$(stat -c%s "${path}")" else printf '%s\tmissing\n' "${path}" fidone System Usage Summary
Legend
Equivalent jq/shelljq -n --slurpfile build "$BUILD_SUMMARY_JSON" --slurpfile test "$TEST_SUMMARY_JSON" ' def stat($s; $group; $stat; $legacy): $s[$group][$stat] // $s[$legacy]; [["Build", $build[0]], ["Test", $test[0]]] | map( (.[1]["duration-sec"] // null) as $duration_sec | { phase: .[0], summary: .[1], duration_min: (if $duration_sec == null then null else ($duration_sec / 60) end), avg_load_frac: stat(.[1]; "load-frac"; "avg"; "avg-load-frac"), max_load_frac: stat(.[1]; "load-frac"; "max"; "max-load-frac"), avg_ram_frac: stat(.[1]; "ram-frac"; "avg"; "avg-ram-frac"), max_ram_frac: stat(.[1]; "ram-frac"; "max"; "max-ram-frac"), avg_sample_sec: stat(.[1]; "sample-duration-sec"; "avg"; null), max_sample_sec: stat(.[1]; "sample-duration-sec"; "max"; null) } )' Build Process LogBuild Target Coverage
Legend
Equivalent jq/shellif [[ -f "$MAKE_OUT" ]]; then object_targets="$(grep -c " object ${BUILD_OUTPUT_PREFIX}/" "$MAKE_OUT" || true)" library_targets="$(grep -c " library ${BUILD_OUTPUT_PREFIX}/" "$MAKE_OUT" || true)" executable_targets="$(grep -c " executable ${BUILD_OUTPUT_PREFIX}/" "$MAKE_OUT" || true)"else object_targets=0 library_targets=0 executable_targets=0fi jq -s --arg prefix "$BUILD_OUTPUT_PREFIX" --arg build_dir "$BUILD_WORKING_DIR" \ --argjson object_targets "$object_targets" \ --argjson library_targets "$library_targets" \ --argjson executable_targets "$executable_targets" ' def is_build_target: (.["cmnd-output"] != null) and (.["output-file-size-mb"] | type == "number") and (.["duration-sec"] | type == "number") and (($prefix == "") or (.["cmnd-output"] | startswith($prefix))) and (($build_dir == "") or ((.["working-dir"] // "" | rtrimstr("/")) == $build_dir)); { tracked_process_rows: length, ninja_object_targets: $object_targets, ninja_library_targets: $library_targets, ninja_executable_targets: $executable_targets, tracked_build_targets: (map(select(is_build_target)) | length), tracked_object_targets: (map(select(is_build_target and (.["cmnd-output"] | endswith(".o")))) | length), non_build_process_rows: (map(select(is_build_target | not)) | length) }' "$BUILD_PROCS_JSONL" Build Target RAM Totals
Legend
Equivalent jq/shelltotal_ram_gb="$(jq '."total-ram-gb" // 0' "$BUILD_SUMMARY_JSON")"jq -s --argjson total "$total_ram_gb" \ --arg prefix "$BUILD_OUTPUT_PREFIX" --arg build_dir "$BUILD_WORKING_DIR" ' def ram_frac: .["ram-frac"].max // .["max-ram-frac"] // 0; def ram_gb: .["ram-gb"].max // .["max-ram-gb"] // (ram_frac * $total); def is_build_target: (.["cmnd-output"] != null) and (.["output-file-size-mb"] | type == "number") and (.["duration-sec"] | type == "number") and (($prefix == "") or (.["cmnd-output"] | startswith($prefix))) and (($build_dir == "") or ((.["working-dir"] // "" | rtrimstr("/")) == $build_dir)); map(select(is_build_target)) as $build | map(select(is_build_target and (.["cmnd-output"] | endswith(".o")))) as $object | { total_ram_gb: $total, build_ram_frac: ($build | map(ram_frac) | add // 0), build_ram_gb: ($build | map(ram_gb) | add // 0), object_ram_frac: ($object | map(ram_frac) | add // 0), object_ram_gb: ($object | map(ram_gb) | add // 0), build_targets: $build, object_targets: $object }' "$BUILD_PROCS_JSONL" Build Target Timing Totals
Legend
Equivalent jq/shellwall_clock="$(jq '."duration-sec" // 0' "$BUILD_SUMMARY_JSON")"total_cores="$(jq '."total-num-cores" // 0' "$BUILD_SUMMARY_JSON")"jq -s --argjson wall "$wall_clock" --argjson cores "$total_cores" \ --arg prefix "$BUILD_OUTPUT_PREFIX" --arg build_dir "$BUILD_WORKING_DIR" ' def is_build_target: (.["cmnd-output"] != null) and (.["output-file-size-mb"] | type == "number") and (.["duration-sec"] | type == "number") and (($prefix == "") or (.["cmnd-output"] | startswith($prefix))) and (($build_dir == "") or ((.["working-dir"] // "" | rtrimstr("/")) == $build_dir)); (map(select(is_build_target)) | map(.["duration-sec"]) | add // 0) as $sum | ($sum / ($wall | if . == 0 then null else . end)) as $speedup | { wall_clock_build_time_min: ($wall / 60), sum_tracked_build_duration_min: ($sum / 60), sum_tracked_build_duration_h: ($sum / 3600), speedup: $speedup, parallel_efficiency: (if $speedup == null or $cores == 0 then null else $speedup / $cores end), tracked_build_targets: map(select(is_build_target)) }' "$BUILD_PROCS_JSONL" Top Build Targets By Max RAM
Legend
Equivalent jq/shelltotal_ram_gb="$(jq '."total-ram-gb" // 0' "$BUILD_SUMMARY_JSON")"# SORT_KEY: ram_gb, duration_sec, or output_mb.# OBJECT_ONLY: true only for the object-target table.# FULL_OUTPUT: true for full cmnd-output paths, false for basenames.# INCLUDE_PACKAGE_DIR: true for tables with a Package dir column.jq -s --argjson total "$total_ram_gb" --argjson top_n "$TOP_N" \ --arg prefix "$BUILD_OUTPUT_PREFIX" --arg build_dir "$BUILD_WORKING_DIR" \ --arg sort_key "$SORT_KEY" --argjson object_only "$OBJECT_ONLY" \ --argjson full_output "$FULL_OUTPUT" \ --argjson include_package_dir "$INCLUDE_PACKAGE_DIR" ' def ram_frac: .["ram-frac"].max // .["max-ram-frac"] // 0; def ram_gb: .["ram-gb"].max // .["max-ram-gb"] // (ram_frac * $total); def duration_sec: .["duration-sec"] // 0; def output_mb: .["output-file-size-mb"] // 0; def package_dir: (.["cmnd-output"] // "" | split("/")) as $parts | ($prefix | rtrimstr("/") | split("/")) as $prefix_parts | if $prefix != "" and ($parts[:($prefix_parts | length)] == $prefix_parts) and (($parts | length) > ($prefix_parts | length)) then $parts[$prefix_parts | length] else null end; def target_label: if $full_output then .["cmnd-output"] else (.["cmnd-output"] | split("/")[-1]) end; def is_build_target: (.["cmnd-output"] != null) and (.["output-file-size-mb"] | type == "number") and (.["duration-sec"] | type == "number") and (($prefix == "") or (.["cmnd-output"] | startswith($prefix))) and (($build_dir == "") or ((.["working-dir"] // "" | rtrimstr("/")) == $build_dir)); def sort_value: if $sort_key == "duration_sec" then duration_sec elif $sort_key == "output_mb" then output_mb else ram_gb end; map(select(is_build_target)) | if $object_only then map(select(.["cmnd-output"] | endswith(".o"))) else . end | sort_by(sort_value) | reverse | .[:$top_n]' "$BUILD_PROCS_JSONL" Top Object Targets By Max RAM
Legend
Equivalent jq/shelltotal_ram_gb="$(jq '."total-ram-gb" // 0' "$BUILD_SUMMARY_JSON")"# SORT_KEY: ram_gb, duration_sec, or output_mb.# OBJECT_ONLY: true only for the object-target table.# FULL_OUTPUT: true for full cmnd-output paths, false for basenames.# INCLUDE_PACKAGE_DIR: true for tables with a Package dir column.jq -s --argjson total "$total_ram_gb" --argjson top_n "$TOP_N" \ --arg prefix "$BUILD_OUTPUT_PREFIX" --arg build_dir "$BUILD_WORKING_DIR" \ --arg sort_key "$SORT_KEY" --argjson object_only "$OBJECT_ONLY" \ --argjson full_output "$FULL_OUTPUT" \ --argjson include_package_dir "$INCLUDE_PACKAGE_DIR" ' def ram_frac: .["ram-frac"].max // .["max-ram-frac"] // 0; def ram_gb: .["ram-gb"].max // .["max-ram-gb"] // (ram_frac * $total); def duration_sec: .["duration-sec"] // 0; def output_mb: .["output-file-size-mb"] // 0; def package_dir: (.["cmnd-output"] // "" | split("/")) as $parts | ($prefix | rtrimstr("/") | split("/")) as $prefix_parts | if $prefix != "" and ($parts[:($prefix_parts | length)] == $prefix_parts) and (($parts | length) > ($prefix_parts | length)) then $parts[$prefix_parts | length] else null end; def target_label: if $full_output then .["cmnd-output"] else (.["cmnd-output"] | split("/")[-1]) end; def is_build_target: (.["cmnd-output"] != null) and (.["output-file-size-mb"] | type == "number") and (.["duration-sec"] | type == "number") and (($prefix == "") or (.["cmnd-output"] | startswith($prefix))) and (($build_dir == "") or ((.["working-dir"] // "" | rtrimstr("/")) == $build_dir)); def sort_value: if $sort_key == "duration_sec" then duration_sec elif $sort_key == "output_mb" then output_mb else ram_gb end; map(select(is_build_target)) | if $object_only then map(select(.["cmnd-output"] | endswith(".o"))) else . end | sort_by(sort_value) | reverse | .[:$top_n]' "$BUILD_PROCS_JSONL" Top Build Targets By Duration
Legend
Equivalent jq/shelltotal_ram_gb="$(jq '."total-ram-gb" // 0' "$BUILD_SUMMARY_JSON")"# SORT_KEY: ram_gb, duration_sec, or output_mb.# OBJECT_ONLY: true only for the object-target table.# FULL_OUTPUT: true for full cmnd-output paths, false for basenames.# INCLUDE_PACKAGE_DIR: true for tables with a Package dir column.jq -s --argjson total "$total_ram_gb" --argjson top_n "$TOP_N" \ --arg prefix "$BUILD_OUTPUT_PREFIX" --arg build_dir "$BUILD_WORKING_DIR" \ --arg sort_key "$SORT_KEY" --argjson object_only "$OBJECT_ONLY" \ --argjson full_output "$FULL_OUTPUT" \ --argjson include_package_dir "$INCLUDE_PACKAGE_DIR" ' def ram_frac: .["ram-frac"].max // .["max-ram-frac"] // 0; def ram_gb: .["ram-gb"].max // .["max-ram-gb"] // (ram_frac * $total); def duration_sec: .["duration-sec"] // 0; def output_mb: .["output-file-size-mb"] // 0; def package_dir: (.["cmnd-output"] // "" | split("/")) as $parts | ($prefix | rtrimstr("/") | split("/")) as $prefix_parts | if $prefix != "" and ($parts[:($prefix_parts | length)] == $prefix_parts) and (($parts | length) > ($prefix_parts | length)) then $parts[$prefix_parts | length] else null end; def target_label: if $full_output then .["cmnd-output"] else (.["cmnd-output"] | split("/")[-1]) end; def is_build_target: (.["cmnd-output"] != null) and (.["output-file-size-mb"] | type == "number") and (.["duration-sec"] | type == "number") and (($prefix == "") or (.["cmnd-output"] | startswith($prefix))) and (($build_dir == "") or ((.["working-dir"] // "" | rtrimstr("/")) == $build_dir)); def sort_value: if $sort_key == "duration_sec" then duration_sec elif $sort_key == "output_mb" then output_mb else ram_gb end; map(select(is_build_target)) | if $object_only then map(select(.["cmnd-output"] | endswith(".o"))) else . end | sort_by(sort_value) | reverse | .[:$top_n]' "$BUILD_PROCS_JSONL" Top Build Targets By Output File Size
Legend
Equivalent jq/shelltotal_ram_gb="$(jq '."total-ram-gb" // 0' "$BUILD_SUMMARY_JSON")"# SORT_KEY: ram_gb, duration_sec, or output_mb.# OBJECT_ONLY: true only for the object-target table.# FULL_OUTPUT: true for full cmnd-output paths, false for basenames.# INCLUDE_PACKAGE_DIR: true for tables with a Package dir column.jq -s --argjson total "$total_ram_gb" --argjson top_n "$TOP_N" \ --arg prefix "$BUILD_OUTPUT_PREFIX" --arg build_dir "$BUILD_WORKING_DIR" \ --arg sort_key "$SORT_KEY" --argjson object_only "$OBJECT_ONLY" \ --argjson full_output "$FULL_OUTPUT" \ --argjson include_package_dir "$INCLUDE_PACKAGE_DIR" ' def ram_frac: .["ram-frac"].max // .["max-ram-frac"] // 0; def ram_gb: .["ram-gb"].max // .["max-ram-gb"] // (ram_frac * $total); def duration_sec: .["duration-sec"] // 0; def output_mb: .["output-file-size-mb"] // 0; def package_dir: (.["cmnd-output"] // "" | split("/")) as $parts | ($prefix | rtrimstr("/") | split("/")) as $prefix_parts | if $prefix != "" and ($parts[:($prefix_parts | length)] == $prefix_parts) and (($parts | length) > ($prefix_parts | length)) then $parts[$prefix_parts | length] else null end; def target_label: if $full_output then .["cmnd-output"] else (.["cmnd-output"] | split("/")[-1]) end; def is_build_target: (.["cmnd-output"] != null) and (.["output-file-size-mb"] | type == "number") and (.["duration-sec"] | type == "number") and (($prefix == "") or (.["cmnd-output"] | startswith($prefix))) and (($build_dir == "") or ((.["working-dir"] // "" | rtrimstr("/")) == $build_dir)); def sort_value: if $sort_key == "duration_sec" then duration_sec elif $sort_key == "output_mb" then output_mb else ram_gb end; map(select(is_build_target)) | if $object_only then map(select(.["cmnd-output"] | endswith(".o"))) else . end | sort_by(sort_value) | reverse | .[:$top_n]' "$BUILD_PROCS_JSONL" Packages Sorted By Build RAM
Legend
Equivalent jq/shelltotal_ram_gb="$(jq '."total-ram-gb" // 0' "$BUILD_SUMMARY_JSON")"# SORT_KEY: ram_frac, duration_frac, or output_mb.jq -s --argjson total "$total_ram_gb" \ --arg prefix "$BUILD_OUTPUT_PREFIX" --arg build_dir "$BUILD_WORKING_DIR" \ --arg pkg_re "$PACKAGE_PATH_REGEX" --arg sort_key "$SORT_KEY" ' def ram_frac: .["ram-frac"].max // .["max-ram-frac"] // 0; def ram_gb: .["ram-gb"].max // .["max-ram-gb"] // (ram_frac * $total); def regex_package($path): try ($path | match($pkg_re).captures[0].string) catch null; def package_dir: (.["cmnd-output"] // "") as $output | (($prefix | rtrimstr("/") | split("/")) as $prefix_parts | ($output | ltrimstr("/") | split("/")) as $parts | regex_package($output) // regex_package("/" + ($output | ltrimstr("/"))) // (if $prefix != "" and ($parts[:($prefix_parts | length)] == $prefix_parts) and (($parts | length) > ($prefix_parts | length)) then $parts[$prefix_parts | length] else null end)); def is_build_target: (.["cmnd-output"] != null) and (.["output-file-size-mb"] | type == "number") and (.["duration-sec"] | type == "number") and (($prefix == "") or (.["cmnd-output"] | startswith($prefix))) and (($build_dir == "") or ((.["working-dir"] // "" | rtrimstr("/")) == $build_dir)) and (package_dir != null); map(select(is_build_target) | . + {package_dir: package_dir}) | group_by(.package_dir) | map({ package_dir: .[0].package_dir, ram_gb: (map(ram_gb) | add // 0), ram_frac_sum: (map(ram_frac) | add // 0), duration_sec: (map(.["duration-sec"]) | add // 0), output_mb: (map(.["output-file-size-mb"]) | add // 0) }) as $rows | ($rows | map(.ram_frac_sum) | add // 1) as $total_ram_frac | ($rows | map(.duration_sec) | add // 1) as $total_duration_sec | ($rows | map(.output_mb) | add // 1) as $total_output_mb | $rows | map(. + { ram_frac: (.ram_frac_sum / $total_ram_frac), duration_frac: (.duration_sec / $total_duration_sec), output_frac: (.output_mb / $total_output_mb) }) | sort_by( if $sort_key == "duration_frac" then .duration_frac elif $sort_key == "output_mb" then .output_mb else .ram_frac end) | reverse | map(. + {duration_min: (.duration_sec / 60)})' "$BUILD_PROCS_JSONL" Packages Sorted By Build Duration
Legend
Equivalent jq/shelltotal_ram_gb="$(jq '."total-ram-gb" // 0' "$BUILD_SUMMARY_JSON")"# SORT_KEY: ram_frac, duration_frac, or output_mb.jq -s --argjson total "$total_ram_gb" \ --arg prefix "$BUILD_OUTPUT_PREFIX" --arg build_dir "$BUILD_WORKING_DIR" \ --arg pkg_re "$PACKAGE_PATH_REGEX" --arg sort_key "$SORT_KEY" ' def ram_frac: .["ram-frac"].max // .["max-ram-frac"] // 0; def ram_gb: .["ram-gb"].max // .["max-ram-gb"] // (ram_frac * $total); def regex_package($path): try ($path | match($pkg_re).captures[0].string) catch null; def package_dir: (.["cmnd-output"] // "") as $output | (($prefix | rtrimstr("/") | split("/")) as $prefix_parts | ($output | ltrimstr("/") | split("/")) as $parts | regex_package($output) // regex_package("/" + ($output | ltrimstr("/"))) // (if $prefix != "" and ($parts[:($prefix_parts | length)] == $prefix_parts) and (($parts | length) > ($prefix_parts | length)) then $parts[$prefix_parts | length] else null end)); def is_build_target: (.["cmnd-output"] != null) and (.["output-file-size-mb"] | type == "number") and (.["duration-sec"] | type == "number") and (($prefix == "") or (.["cmnd-output"] | startswith($prefix))) and (($build_dir == "") or ((.["working-dir"] // "" | rtrimstr("/")) == $build_dir)) and (package_dir != null); map(select(is_build_target) | . + {package_dir: package_dir}) | group_by(.package_dir) | map({ package_dir: .[0].package_dir, ram_gb: (map(ram_gb) | add // 0), ram_frac_sum: (map(ram_frac) | add // 0), duration_sec: (map(.["duration-sec"]) | add // 0), output_mb: (map(.["output-file-size-mb"]) | add // 0) }) as $rows | ($rows | map(.ram_frac_sum) | add // 1) as $total_ram_frac | ($rows | map(.duration_sec) | add // 1) as $total_duration_sec | ($rows | map(.output_mb) | add // 1) as $total_output_mb | $rows | map(. + { ram_frac: (.ram_frac_sum / $total_ram_frac), duration_frac: (.duration_sec / $total_duration_sec), output_frac: (.output_mb / $total_output_mb) }) | sort_by( if $sort_key == "duration_frac" then .duration_frac elif $sort_key == "output_mb" then .output_mb else .ram_frac end) | reverse | map(. + {duration_min: (.duration_sec / 60)})' "$BUILD_PROCS_JSONL" Packages Sorted By Build Output File Size
Legend
Equivalent jq/shelltotal_ram_gb="$(jq '."total-ram-gb" // 0' "$BUILD_SUMMARY_JSON")"# SORT_KEY: ram_frac, duration_frac, or output_mb.jq -s --argjson total "$total_ram_gb" \ --arg prefix "$BUILD_OUTPUT_PREFIX" --arg build_dir "$BUILD_WORKING_DIR" \ --arg pkg_re "$PACKAGE_PATH_REGEX" --arg sort_key "$SORT_KEY" ' def ram_frac: .["ram-frac"].max // .["max-ram-frac"] // 0; def ram_gb: .["ram-gb"].max // .["max-ram-gb"] // (ram_frac * $total); def regex_package($path): try ($path | match($pkg_re).captures[0].string) catch null; def package_dir: (.["cmnd-output"] // "") as $output | (($prefix | rtrimstr("/") | split("/")) as $prefix_parts | ($output | ltrimstr("/") | split("/")) as $parts | regex_package($output) // regex_package("/" + ($output | ltrimstr("/"))) // (if $prefix != "" and ($parts[:($prefix_parts | length)] == $prefix_parts) and (($parts | length) > ($prefix_parts | length)) then $parts[$prefix_parts | length] else null end)); def is_build_target: (.["cmnd-output"] != null) and (.["output-file-size-mb"] | type == "number") and (.["duration-sec"] | type == "number") and (($prefix == "") or (.["cmnd-output"] | startswith($prefix))) and (($build_dir == "") or ((.["working-dir"] // "" | rtrimstr("/")) == $build_dir)) and (package_dir != null); map(select(is_build_target) | . + {package_dir: package_dir}) | group_by(.package_dir) | map({ package_dir: .[0].package_dir, ram_gb: (map(ram_gb) | add // 0), ram_frac_sum: (map(ram_frac) | add // 0), duration_sec: (map(.["duration-sec"]) | add // 0), output_mb: (map(.["output-file-size-mb"]) | add // 0) }) as $rows | ($rows | map(.ram_frac_sum) | add // 1) as $total_ram_frac | ($rows | map(.duration_sec) | add // 1) as $total_duration_sec | ($rows | map(.output_mb) | add // 1) as $total_output_mb | $rows | map(. + { ram_frac: (.ram_frac_sum / $total_ram_frac), duration_frac: (.duration_sec / $total_duration_sec), output_frac: (.output_mb / $total_output_mb) }) | sort_by( if $sort_key == "duration_frac" then .duration_frac elif $sort_key == "output_mb" then .output_mb else .ram_frac end) | reverse | map(. + {duration_min: (.duration_sec / 60)})' "$BUILD_PROCS_JSONL" Test Process LogTest Process Totals
Legend
Equivalent jq/shelltotal_ram_gb="$(jq '."total-ram-gb" // 0' "$TEST_SUMMARY_JSON")"excludes_json="$(printf '%s\n' "$TEST_COMMAND_EXCLUDES" \ | jq -R 'split(",") | map(select(length > 0))')"jq -s --argjson total "$total_ram_gb" --argjson excludes "$excludes_json" \ --arg build_dir "$BUILD_WORKING_DIR" --arg pkg_re "$PACKAGE_PATH_REGEX" ' def ram_frac: .["ram-frac"].max // .["max-ram-frac"] // 0; def ram_gb: .["ram-gb"].max // .["max-ram-gb"] // (ram_frac * $total); def package_dir: try (.["working-dir"] | match($pkg_re).captures[0].string) catch null; def is_test_proc: (.cmnd != null) and (.["duration-sec"] | type == "number") and (.cmnd as $cmd | ($excludes | index($cmd) | not)) and (($build_dir == "") or ((.["working-dir"] // "") | startswith($build_dir + "/"))) and (package_dir != null); map(select(is_test_proc)) as $tests | { tracked_test_process_rows: ($tests | length), ram_usage_by_all_tests_gb: ($tests | map(ram_gb) | add // 0), sum_tracked_test_duration_min: (($tests | map(.["duration-sec"]) | add // 0) / 60), test_processes: $tests }' "$TEST_PROCS_JSONL" Top Test Processes By Max RAM
Legend
Equivalent jq/shelltotal_ram_gb="$(jq '."total-ram-gb" // 0' "$TEST_SUMMARY_JSON")"excludes_json="$(printf '%s\n' "$TEST_COMMAND_EXCLUDES" \ | jq -R 'split(",") | map(select(length > 0))')"# SORT_KEY: ram_gb or duration_sec.jq -s --argjson total "$total_ram_gb" --argjson top_n "$TOP_N" \ --argjson excludes "$excludes_json" --arg build_dir "$BUILD_WORKING_DIR" \ --arg pkg_re "$PACKAGE_PATH_REGEX" --arg sort_key "$SORT_KEY" ' def ram_frac: .["ram-frac"].max // .["max-ram-frac"] // 0; def ram_gb: .["ram-gb"].max // .["max-ram-gb"] // (ram_frac * $total); def package_dir: try (.["working-dir"] | match($pkg_re).captures[0].string) catch null; def is_test_proc: (.cmnd != null) and (.["duration-sec"] | type == "number") and (.cmnd as $cmd | ($excludes | index($cmd) | not)) and (($build_dir == "") or ((.["working-dir"] // "") | startswith($build_dir + "/"))) and (package_dir != null); def sort_value: if $sort_key == "duration_sec" then .["duration-sec"] else ram_gb end; map(select(is_test_proc)) | sort_by(sort_value) | reverse | .[:$top_n]' "$TEST_PROCS_JSONL" Top Test Processes By Duration
Legend
Equivalent jq/shelltotal_ram_gb="$(jq '."total-ram-gb" // 0' "$TEST_SUMMARY_JSON")"excludes_json="$(printf '%s\n' "$TEST_COMMAND_EXCLUDES" \ | jq -R 'split(",") | map(select(length > 0))')"# SORT_KEY: ram_gb or duration_sec.jq -s --argjson total "$total_ram_gb" --argjson top_n "$TOP_N" \ --argjson excludes "$excludes_json" --arg build_dir "$BUILD_WORKING_DIR" \ --arg pkg_re "$PACKAGE_PATH_REGEX" --arg sort_key "$SORT_KEY" ' def ram_frac: .["ram-frac"].max // .["max-ram-frac"] // 0; def ram_gb: .["ram-gb"].max // .["max-ram-gb"] // (ram_frac * $total); def package_dir: try (.["working-dir"] | match($pkg_re).captures[0].string) catch null; def is_test_proc: (.cmnd != null) and (.["duration-sec"] | type == "number") and (.cmnd as $cmd | ($excludes | index($cmd) | not)) and (($build_dir == "") or ((.["working-dir"] // "") | startswith($build_dir + "/"))) and (package_dir != null); def sort_value: if $sort_key == "duration_sec" then .["duration-sec"] else ram_gb end; map(select(is_test_proc)) | sort_by(sort_value) | reverse | .[:$top_n]' "$TEST_PROCS_JSONL" Packages Sorted By Test RAM
Legend
Equivalent jq/shelltotal_ram_gb="$(jq '."total-ram-gb" // 0' "$TEST_SUMMARY_JSON")"excludes_json="$(printf '%s\n' "$TEST_COMMAND_EXCLUDES" \ | jq -R 'split(",") | map(select(length > 0))')"# SORT_KEY: ram_frac or duration_frac.jq -s --argjson total "$total_ram_gb" \ --argjson excludes "$excludes_json" --arg build_dir "$BUILD_WORKING_DIR" \ --arg pkg_re "$PACKAGE_PATH_REGEX" --arg sort_key "$SORT_KEY" ' def ram_frac: .["ram-frac"].max // .["max-ram-frac"] // 0; def ram_gb: .["ram-gb"].max // .["max-ram-gb"] // (ram_frac * $total); def package_dir: try (.["working-dir"] | match($pkg_re).captures[0].string) catch null; def is_test_proc: (.cmnd != null) and (.["duration-sec"] | type == "number") and (.cmnd as $cmd | ($excludes | index($cmd) | not)) and (($build_dir == "") or ((.["working-dir"] // "") | startswith($build_dir + "/"))) and (package_dir != null); map(select(is_test_proc) | . + {package_dir: package_dir}) | group_by(.package_dir) | map({ package_dir: .[0].package_dir, ram_gb: (map(ram_gb) | add // 0), ram_frac_sum: (map(ram_frac) | add // 0), duration_sec: (map(.["duration-sec"]) | add // 0) }) as $rows | ($rows | map(.ram_frac_sum) | add // 1) as $total_ram_frac | ($rows | map(.duration_sec) | add // 1) as $total_duration_sec | $rows | map(. + { ram_frac: (.ram_frac_sum / $total_ram_frac), duration_frac: (.duration_sec / $total_duration_sec) }) | sort_by(if $sort_key == "duration_frac" then .duration_frac else .ram_frac end) | reverse | map(. + {duration_min: (.duration_sec / 60)})' "$TEST_PROCS_JSONL" Packages Sorted By Test Duration
Legend
Equivalent jq/shelltotal_ram_gb="$(jq '."total-ram-gb" // 0' "$TEST_SUMMARY_JSON")"excludes_json="$(printf '%s\n' "$TEST_COMMAND_EXCLUDES" \ | jq -R 'split(",") | map(select(length > 0))')"# SORT_KEY: ram_frac or duration_frac.jq -s --argjson total "$total_ram_gb" \ --argjson excludes "$excludes_json" --arg build_dir "$BUILD_WORKING_DIR" \ --arg pkg_re "$PACKAGE_PATH_REGEX" --arg sort_key "$SORT_KEY" ' def ram_frac: .["ram-frac"].max // .["max-ram-frac"] // 0; def ram_gb: .["ram-gb"].max // .["max-ram-gb"] // (ram_frac * $total); def package_dir: try (.["working-dir"] | match($pkg_re).captures[0].string) catch null; def is_test_proc: (.cmnd != null) and (.["duration-sec"] | type == "number") and (.cmnd as $cmd | ($excludes | index($cmd) | not)) and (($build_dir == "") or ((.["working-dir"] // "") | startswith($build_dir + "/"))) and (package_dir != null); map(select(is_test_proc) | . + {package_dir: package_dir}) | group_by(.package_dir) | map({ package_dir: .[0].package_dir, ram_gb: (map(ram_gb) | add // 0), ram_frac_sum: (map(ram_frac) | add // 0), duration_sec: (map(.["duration-sec"]) | add // 0) }) as $rows | ($rows | map(.ram_frac_sum) | add // 1) as $total_ram_frac | ($rows | map(.duration_sec) | add // 1) as $total_duration_sec | $rows | map(. + { ram_frac: (.ram_frac_sum / $total_ram_frac), duration_frac: (.duration_sec / $total_duration_sec) }) | sort_by(if $sort_key == "duration_frac" then .duration_frac else .ram_frac end) | reverse | map(. + {duration_min: (.duration_sec / 60)})' "$TEST_PROCS_JSONL" CTest OutputThis section provides summaries of the information from running CTest itself as provided by the CTest Totals
Legend
Equivalent jq/shelljq -R -s ' def lines: split("\n"); def tests: lines | map(capture("^\\s*[0-9]+/[0-9]+\\s+Test\\s+#\\s*(?<id>[0-9]+):" + "\\s+(?<name>.*?)\\s+\\.{3,}\\s+(?<status>.+?)\\s+" + "(?<sec>[0-9.]+)\\s+sec\\s*$")? // empty); def subprojects: reduce lines[] as $line ({in:false, rows:[]}; if $line == "Subproject Time Summary:" then .in = true elif $line == "Label Time Summary:" then .in = false elif .in then (($line | capture("^(?<name>.+?)\\s*=\\s*(?<sec>[0-9.]+)" + "\\s+sec[*]proc\\s+[(](?<tests>[0-9]+) tests?[)]")?) // empty) as $m | if $m == null then . else .rows += [$m] end else . end) | .rows; (capture("Total Test time [(]real[)] =\\s*(?<sec>[0-9.]+) sec")?.sec | tonumber?) as $total_real_sec | { tests_parsed: (tests | length), pass_summary: (capture("(?<pct>[0-9]+)% tests passed, " + "(?<failed>[0-9]+) tests failed out of (?<total>[0-9]+)")?), sum_individual_test_min: ((tests | map(.sec | tonumber) | add // 0) / 60), sum_subproject_min_proc: ((subprojects | map(.sec | tonumber) | add // 0) / 60), total_real_min: (if $total_real_sec == null then null else ($total_real_sec / 60) end) }' "$CTEST_OUT" CTest Subprojects By min*proc
Legend
Equivalent jq/shelljq -R -s ' split("\n") | reduce .[] as $line ({in:false, rows:[]}; if $line == "Subproject Time Summary:" then .in = true elif $line == "Label Time Summary:" then .in = false elif .in then (($line | capture("^(?<name>.+?)\\s*=\\s*(?<sec>[0-9.]+)" + "\\s+sec[*]proc\\s+[(](?<tests>[0-9]+) tests?[)]")?) // empty) as $m | if $m == null then . else .rows += [[ ($m.name | gsub("^\\s+|\\s+$"; "")), ($m.sec | tonumber), ($m.tests | tonumber) ]] end else . end) | .rows | sort_by(.[1]) | reverse | map({ subproject: .[0], sec_proc: .[1], min_proc: (.[1] / 60), num_tests: .[2] })' "$CTEST_OUT" CTest Tests By Duration
Legend
Equivalent jq/shelljq -R -s --argjson top_n "$TOP_N" ' split("\n") | map(capture("^\\s*[0-9]+/[0-9]+\\s+Test\\s+#\\s*(?<id>[0-9]+):" + "\\s+(?<name>.*?)\\s+\\.{3,}\\s+(?<status>.+?)\\s+" + "(?<sec>[0-9.]+)\\s+sec\\s*$")? // empty) | map(. + {id: (.id | tonumber), duration_sec: (.sec | tonumber)}) | sort_by(.duration_sec) | reverse | .[:$top_n]' "$CTEST_OUT" CTest Versus Process-Log Package Duration
Legend
Equivalent jq/shell# This computes the JSONL package side; join with the CTest subproject command above# by normalizing package/subproject names with non-alphanumeric characters removed.total_ram_gb="$(jq '."total-ram-gb" // 0' "$TEST_SUMMARY_JSON")"excludes_json="$(printf '%s\n' "$TEST_COMMAND_EXCLUDES" \ | jq -R 'split(",") | map(select(length > 0))')"jq -s --argjson excludes "$excludes_json" --arg build_dir "$BUILD_WORKING_DIR" \ --arg pkg_re "$PACKAGE_PATH_REGEX" ' def package_dir: try (.["working-dir"] | match($pkg_re).captures[0].string) catch null; def normalized($s): $s | ascii_downcase | gsub("[^a-z0-9]"; ""); def is_test_proc: (.cmnd != null) and (.["duration-sec"] | type == "number") and (.cmnd as $cmd | ($excludes | index($cmd) | not)) and (($build_dir == "") or ((.["working-dir"] // "") | startswith($build_dir + "/"))) and (package_dir != null); map(select(is_test_proc) | . + {package_dir: package_dir}) | group_by(.package_dir) | map({ package_dir: .[0].package_dir, normalized_package_dir: normalized(.[0].package_dir), process_duration_sec: (map(.["duration-sec"]) | add // 0), process_duration_min: ((map(.["duration-sec"]) | add // 0) / 60) })' "$TEST_PROCS_JSONL" |
Beta Was this translation helpful? Give feedback.
4 replies
-
|
This is a lot of data, but it would be good to give some indication about how to improve these metrics. I have used the script from |
Beta Was this translation helpful? Give feedback.
-
|
How sure are we that these statistics are correct and/or meaningful? I spot checked Tempus on my system and I am getting and In the tables above I see that the same test took 93s. |
Beta Was this translation helpful? Give feedback.
-
The data is the data, and there are a lot of targets and tests and lots of metrics/stats you need to know to analyze what is happening. But
This is just data. It is up to you to decide how to use it. But if it were me, I would start by looking at the *.o targets for code that you control that consume the most RAM and runtime and decide if you want to look into breaking those up into smaller translation units or doing other types of refactorings and see what that does. And if you do experiment with making changes, do the builds before and after your changes and compare these metrics and see how they change. For example, if you are trying to make the NOTE: While build times will tend to be noising, the max RAM per target should be pretty stable, even on a loaded machine. Therefore, max RAM per target is likely a better metric to start with. And max RAM is the single biggest impediment to making the builds go faster because if you hit an out-of-memory state, you are screwed. If you overload the cores, no big deal. I would live the Trilinos PR builds to use this tool and post the results to CDash so we can see how well the test machines are actually being utilized. What are the min/max/average loads and RAM fractions on these machines?
I have looked at a lot of this and it seems to be pretty accurate. The only weakness I can se is that you don't catch all of the smaller targets and test processes that take very little RAM and run very fast. But we don't really care about those anyway. We care about the big stuff. Note that in general, it is hard to map individual test processes to specific CTest tests. You can have multiple tests that run the same binary but have radically different runtimes. But with the tests you mention above, this is pretty easy because there seems to be a single test binary for each tests. Also, note that the timings are gotten by sampling First, see the file:
which shows that when I ran these tests on my machine and grabbed these metrics, CTest showed. Looking in the file:
with: cat high-ram-usage-procs.test.jsonl \
| jq -s 'map(select(."cmnd" | startswith("Tempus_BDF2_CDR")))' | pprint_json.py -w 120that shows: [
{ "proc-id": 3759072, "starttime-utc": "2026-05-25T23:02:56Z", "duration-sec": 15.657724,
"cmnd": "Tempus_BDF2_CDR.exe",
"working-dir": "/mounted_from_host/Trilinos/BUILDS/clang-19-genconfig-cov/packages/tempus/test/BDF2",
"cmnd-line-matches": [], "cmnd-output": null, "output-file-size-mb": null, "num-proc-samples": 30,
"load-frac": {"avg": 0.036071, "min": 0.0, "max": 0.04},
"ram-frac": {"avg": 0.000639, "min": 0.000614, "max": 0.000656}, "load": {"avg": 1.009988, "min": 0.0, "max": 1.12},
"ram-gb": {"avg": 0.039938, "min": 0.038375, "max": 0.041}, "num-system-samples": 30,
"system-load-frac": {"start": 0.5675, "end": 0.4525, "avg": 0.512155, "min": 0.4525, "max": 0.5675},
"system-ram-frac": {"start": 0.080374, "end": 0.082149, "avg": 0.081628, "min": 0.080371, "max": 0.083572},
"system-buff-cache-ram-frac": { "start": 0.922471, "end": 0.919446, "avg": 0.92022, "min": 0.919372, "max": 0.922478
},
"system-load": {"start": 15.89, "end": 12.67, "avg": 14.340333, "min": 12.67, "max": 15.89},
"system-ram-gb": {"start": 5.023374, "end": 5.134328, "avg": 5.101726, "min": 5.023191, "max": 5.223236},
"system-buff-cache-ram-gb": { "start": 57.662838, "end": 57.473763, "avg": 57.522118, "min": 57.469078,
"max": 57.663292
}
}
]That looks like a pretty good match at 15.39 vs 15.657724 seconds? And for the test as reported by CTest: when I run: cat high-ram-usage-procs.test.jsonl \
| jq -s 'map(select(."cmnd" | startswith("Tempus_ExplicitRK_ASA")))' | pprint_json.py -w 120I see: [
{ "proc-id": 3650227, "starttime-utc": "2026-05-25T22:54:42Z", "duration-sec": 93.072398,
"cmnd": "Tempus_ExplicitRK_ASA.exe",
"working-dir": "/mounted_from_host/Trilinos/BUILDS/clang-19-genconfig-cov/packages/tempus/test/ExplicitRK",
"cmnd-line-matches": [], "cmnd-output": null, "output-file-size-mb": null, "num-proc-samples": 167,
"load-frac": {"avg": 0.034821, "min": 0.0, "max": 0.036429},
"ram-frac": {"avg": 0.000699, "min": 0.000445, "max": 0.000817},
"load": {"avg": 0.974988, "min": 0.0, "max": 1.020012},
"ram-gb": {"avg": 0.043687, "min": 0.027812, "max": 0.051063}, "num-system-samples": 167,
"system-load-frac": {"start": 0.334643, "end": 0.882857, "avg": 0.661946, "min": 0.334643, "max": 0.882857},
"system-ram-frac": {"start": 0.110816, "end": 0.110833, "avg": 0.121922, "min": 0.089166, "max": 0.213233},
"system-buff-cache-ram-frac": {"start": 0.38068, "end": 0.867123, "avg": 0.733912, "min": 0.38068, "max": 0.893924},
"system-load": {"start": 9.37, "end": 24.72, "avg": 18.534491, "min": 9.37, "max": 24.72},
"system-ram-gb": {"start": 6.926017, "end": 6.927093, "avg": 7.620094, "min": 5.572897, "max": 13.327045},
"system-buff-cache-ram-gb": { "start": 23.795982, "end": 54.203068, "avg": 45.876191, "min": 23.795982,
"max": 55.878357
}
}
]Again, 93.20 vs. 93.072398 looks like a pretty good match? Please pull down trilinos-build-target-metrics.tar.gz and analyze the data for yourself. Let me know if you see something that is off in the data. And the whole point is the ability to run this tool yourself and analyze the data in your local builds. The Jupyter Notebook (hopefully) shows you how to do that. And you can run the tool yourself pretty easily in your own local builds as described at: Let me know if you can't get that working or have any questions about that. |
Beta Was this translation helpful? Give feedback.
-
|
@cgcgcg, also, note that when I run this for the Trilinos container LLVM/clang build, I try to grab all of the build targets, not matter their RAM usage with something like: rm -r packages/
rm -r /home/runner/ccache-dir/*
time ./do-configure -DTrilinos_ENABLE_ALL_PACKAGES=ON -DTrilinos_PARALLEL_LINK_JOBS_LIMIT=5 \
&> configure.all-packages.out
log_system_and_proc_usage_args=(-C 5.0 -p -1 -M -1
-r '^clang$' '^clang[+][+]$' '^ld$' '^llvm-ar$' '^f951$'
-R 'log_system_and_proc_usage[.]py'
-w "${PWD}")
log_system_and_proc_usage.py "${log_system_and_proc_usage_args[@]}" \
-o log_system_and_proc_usage.build.json -P high-ram-usage-procs.build.jsonl -X &
log_system_and_proc_usage_pid=$!
time ninja -j $(nproc) &> make.out
shutdown-and-wait.sh ${log_system_and_proc_usage_pid}
time sleep 60s # Let 1-min load die down
log_system_and_proc_usage_args=(-C 5.0 -p -1 -M 0.001 -R 'log_system_and_proc_usage[.]py')
log_system_and_proc_usage.py "${log_system_and_proc_usage_args[@]}" \
-o log_system_and_proc_usage.test.json -P high-ram-usage-procs.test.jsonl -X &
log_system_and_proc_usage_pid=$!
time ctest -j $(nproc) &> ctest.out
shutdown-and-wait.sh ${log_system_and_proc_usage_pid}Note that above, I don't have a nice way to filter for test processes only so I just track all of them that use more than |
Beta Was this translation helpful? Give feedback.
-
|
FYI: See related comment: |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
FYI, I updated the Jupyter Notebook to add histograms, bar charts, sums, other metrics, generated system RAM and load plots, etc. See attached examples. Just download the *.html files and open in your browser. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
@trilinos/developers |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Trilinos Build Target Metrics using log_system_and_proc_usage.py
Table of Contents:
log_system_and_proc_usage.pywith 6 parallel link jobsOverview
There is a tool I recently created called
log_system_and_proc_usage.py. It can be used to gather data about any build or test process and runs completely separate from your build and test process. It will track system RAM and load usage and track all processes that use more than-M <min-ram-mb>in RAM (as sampled). The repo this tool is in is in the process of being open-sourced. But until then, I wanted to post an analysis I did of some data produced with this tool. The data and how it was analyzed is given in:(This was produced in a container so nothing sensitive there.)
And there is more updated data in that tarred up directory. It is just not being analyzed like the summary I show below.
I would like to create a Jupyter Notebook for this type of analysis and then people could copy it it and mess around with it.
Summary of build and test info from log_system_and_proc_usage.py
In this section, we give a tight summary of build and test processes given info from the
log_system_and_proc_usage.pyscript for a full Trilinos build.Summary of build process log info
Here is a summary of what we learned from this build data ...
Summary of RAM and load across the entire build
From
log_system_and_proc_usage.build.json:{ "starttime-utc": "2026-05-22T01:31:33Z", "duration-sec": 6605.005419, "num-samples": 6179, "total-num-cores": 28, "total-ram-gb": 62.5, "max-load-frac": 1.074286, "avg-load-frac": 0.745843, "max-ram-frac": 0.883351, "avg-ram-frac": 0.424406, "max-ram-processes": [...] }That is 75% core utilization without running out of RAM. (NOTE: If you don't limit the build jobs based, you get OOM states.)
Fraction of tracked build targets to all ninja build targets
So we are tracking 90% of the build targets (all taking more than 1.0M of max RAM).
Total RAM usage for all tracked build targets summary
So you could spread out the object files to build them all at the same time and utilize 2.8T of RAM!
Build processes taking the most RAM summary
The top-10 tracked build processes taking the most max RAM where:
NOTE: Sorted by "Max RAM (GB)"
Object build processes taking the most RAM summary
The top-10 tracked object build processes taking the most max RAM where:
NOTE: Sorted by "Max RAM (GB)"
Build processes taking the most CPU time summary
NOTE: Sorted by "Duration (min)"
Build targets by output file size summary
Summary of size of tracked files vs. build dir size:
The top-10 largets output files for tracked targets:
NOTE: Sorted by "Output file size (GB)"
Summary of test process log info
Here is a summary of what we learned about running the tests.
Summary of RAM and load across the entire test suite
From
log_system_and_proc_usage.test.json:{ "starttime-utc": "2026-05-22T15:52:29Z", "duration-sec": 555.005861, "num-samples": 539, "total-num-cores": 28, "total-ram-gb": 62.5, "max-load-frac": 1.022857, "avg-load-frac": 0.816635, "max-ram-frac": 0.784487, "avg-ram-frac": 0.199954, "max-ram-processes": [ ... ] }That is good core utilization of 82%, but the max RAM usage of 78% is surprising!
Test processes using the most RAM summary
The top-10 test processes using the most RAM are:
Packages using the most RAM summary
The total RAM usage for tests by package is given by:
Test time time by package as determined by CTest
The
ctestSTDOUT, sorted by time gives:Summary of test process time as determined by log_system_and_proc_usage.py
NOTE: This does not see test processes that take less than 1 sec or use less than 1.0M in RAM. But you can see that top packages have very similar duration numbers.
Top test processes using the most CPU time
Top tests using the most CPU time as determined by CTest
If you compare that to the list of processes shown above, you can see they match up pretty well. Very inteestring. You could match up the GB usage by matching up the test target binaries from the
ctest --show-only=json-v1output if you wanted.Full Trilinos build with updated and faster
log_system_and_proc_usage.pywith 6 parallel link jobsNOTE: This was done on 5/25/2026 with an updated version of
log_system_and_proc_usage.pythat is faster and has more data and is restructured.Run a ful Trilinos build from scratch with updated settings, getting all of the interesting build object.
That showed:
I copied that to:
The build passed, and log_system_and_proc_usage.build.json with:
shows:
{ "starttime-utc": "2026-05-25T21:07:16Z", "duration-sec": 6360.629117, "num-samples": 11170, "total-num-cores": 28, "total-ram-gb": 62.5, "load-frac": { "start": 0.152857, "end": 0.180357, "avg": 0.74054, "min": 0.152857, "max": 0.996071 }, "ram-frac": { "start": 0.168211, "end": 0.102206, "avg": 0.429736, "min": 0.102206, "max": 0.860161 }, "buff-cache-ram-frac": { "start": 0.132326, "end": 0.336174, "avg": 0.388754, "min": 0.132326, "max": 0.809222 }, "load": { "start": 4.28, "end": 5.05, "avg": 20.735118, "min": 4.28, "max": 27.89 }, "ram-gb": { "start": 10.513173, "end": 6.387884, "avg": 26.858491, "min": 6.387884, "max": 53.760034 }, "buff-cache-ram-gb": { "start": 8.271553, "end": 21.013927, "avg": 24.300671, "min": 8.271553, "max": 50.583733 }, "sample-duration-sec": { "start": 0.015599, "end": 0.021745, "avg": 0.068867, "min": 0.015599, "max": 2.9828 }, "num-curr-tracked-procs": 8, "num-logged-procs": 11671, "num-total-tracked-procs": 11761, "proc-ram-frac": {"avg": 0.007998, "min": 0.0, "max": 0.242549}, "proc-ram-gb": {"avg": 0.499884, "min": 0.0, "max": 15.159312}, "proc-load": {"avg": 0.777907, "min": 0.0, "max": 4.765992} }The plots of the load and RAM usage are given in:
Build RAM and load Plot:
Test RAM and load Plot:
And the new data there is "end-system-buff-ram-frac".
The new and interesting data is the sample duration for
log_system_and_proc_usage.pywhich is:So while the max was 2.98, the average was 0.069. That means we could do
-s 0.1on this machine with this build and keep up. And we about the right number of samples 6360 / 11170 = 0.57 sec.So I think we could reduce this to
-s 0.1and pick up more processes. But I think we should likely not care to track build or test processes that take less that 1.5 sec to run. The CMake instrumentation system already has durations for all processes and tests. We could just augment that info with this info (at least for the build targets) and upload that to CDash. But trying to associated the test processes with indvidual CTest tests would be very difficult. I don't even want to deal with that right now.This looks excellent!
Beta Was this translation helpful? Give feedback.
All reactions