Repository navigation
PNWC Hardware Testing Scripts Manjaro Linux Intel CPU
Python orchestration scripts for automated hardware diagnostics, benchmarking, and reliability stress testing on Manjaro Linux test benches (Z790/i9 platform, broadly compatible).
Documents the Python test suite: automated CPU/memory/storage/GPU test orchestration, result logging and comparison, thermal monitoring during tests, pass/fail thresholds, and HTML report generation for test results.
Automate the full hardware qualification test suite so results are reproducible, logged, and comparable across systems without manual command-by-command execution.
- Running a full automated hardware qualification suite on a new system
- Comparing test results across multiple systems for consistency
- Generating documented test reports for hardware procurement or RMA
- Scheduling periodic automated health checks on lab hardware
Python orchestration scripts for diagnostics, benchmarking, and reliability stress testing on Manjaro Linux test benches. Designed for Z790/i9 platforms but works on any modern PC hardware.
These scripts wrap robust CLI tools (stress-ng, fio, glmark2, vkmark, memtester, sysbench, etc.), stream their output live so you always see progress, and compile results into clean client-facing Markdown reports with a clear PASS / FAIL verdict.
Part of the PNWC ULTIMATE-CYBERSECURITY-MASTER-GUIDE ยท Pacific Northwest Computers ยท Vancouver, WA
| Script | Purpose | Requires sudo |
|---|---|---|
full_hw_suite.py |
Full diagnostic pass - system info, CPU, RAM, storage, GPU | โ |
standalone_gpu_tester.py |
Universal GPU benchmark (OpenGL/Vulkan validation, memtest, kernel fault scanning) | โ |
amd_gpu_tester.py |
AMD Radeon GPU diagnostic with amdgpu telemetry & Vulkan/OpenGL testing | โ |
nvidia_gpu_tester.py |
NVIDIA GPU diagnostic with nvidia-smi telemetry & Vulkan/OpenGL testing | โ |
standalone_ram_tester.py |
RAM-only - bandwidth (sysbench) + stability (memtester) | โ |
stress_soak.py |
Reliability burn-in - simultaneous CPU+RAM+storage+GPU stress with continuous thermal logging | โ |
GPU-only scripts run as your normal desktop user - not
sudo. Launching the GUI benchmarks (glmark2,vkmark) undersudocan block access to the display server. Their kernel-fault scan needs read access to the kernel journal, so run as a user in thesystemd-journalgroup (or as root in a session where the display is reachable); otherwise that one check is reported as REVIEW rather than PASS. The full-suite, RAM, and soak scripts do needsudoand internally route the GPU workload back to the desktop user.
Quick rule of thumb: Run the diagnostic scripts first to establish a baseline and catch obvious failures. Run
stress_soak.pybefore returning hardware to a client to validate long-term reliability.
sudo pacman -Syu --needed \
base-devel git python \
stress-ng fio memtester sysbench \
inxi dmidecode smartmontools nvme-cli pciutils usbutils \
lm_sensors intel-gpu-tools \
vulkan-tools mesa-utils vkmark glmark2 \
nvidia-utils cuda mesa vulkan-radeon \
amdsmi amdgpu_top radeontopNotes:
-
nvidia-smiis not a separate package - it ships withnvidia-utils. (Installingnvidia-smidirectly will fail with "target not found".) -
cudais large (~3 GB) and is only needed if you plan to run CUDA/gpu-burncompute stress. Drop it on AMD/Intel-only benches. - Install the cross-vendor set above only on a universal bench. Otherwise install just the GPU tools that match your hardware (see Step 3).
You can install the full AMD + NVIDIA + Intel tool set in one shot using the install_testbench_tools.sh script.
sudo sensors-detect --autoWithout this, sensors returns no data and post-load thermal snapshots will be empty.
Intel iGPU (all Z790/i9 systems - already covered by intel-gpu-tools above)
AMD Radeon:
sudo pacman -S --needed mesa vulkan-radeon amdsmi amdgpu_top radeontopNVIDIA:
sudo pacman -S --needed nvidia-utils cudaVerify the NVIDIA tools are present:
nvidia-smi --versionfor bin in inxi dmidecode sysbench memtester fio stress-ng \
smartctl nvme sensors glmark2 vkmark intel_gpu_top \
vulkaninfo glxinfo; do
printf "%-20s %s\n" "$bin" "$(command -v $bin || echo 'โ NOT FOUND')"
doneAny โ NOT FOUND line means that tool needs to be installed before running the relevant script. (Note: glmark2 and vkmark are natively packaged in Manjaro/Arch.)
Runs a complete diagnostic pass sequentially: System info โ CPU benchmark โ RAM (bandwidth + stability) โ Storage (SMART + fio IOPS) โ GPU benchmark. Produces a single Markdown report with a report-level verdict.
Run from the mount point of the drive you want to test.
sudo python3 full_hw_suite.py --client "Client Name"Output: ~/Full_Hardware_Report_YYYYMMDD_HHMMSS.md
Key v2.1 improvements:
- Safer timeout and process-group handling to prevent zombie processes.
- Better storage testing using temporary
fiofiles rather than repo-folder test files. - Deep SMART analysis via
smartctl. - Renderer validation for both Vulkan and OpenGL environments.
- Proper desktop GPU command handling when running under
sudo(avoids display server blocks). - Kernel hardware error snapshots embedded directly in the report.
- Clear report-level PASS / FAIL verdict.
Tests the GPU in isolation. This is an excellent general first-pass GPU script for any vendor (Intel, AMD, NVIDIA). For a serious suspected AMD or NVIDIA card failure, run the vendor-specific AMD/NVIDIA script afterward - those collect richer per-second telemetry.
No sudo required. Run from a desktop terminal session (X11 or Wayland - not SSH).
Features:
- Keeps the PNWC ASCII banner branding.
- Checks GPU hardware with
inxiandlspci. - Validates Vulkan (
vulkaninfo --summary) and OpenGL (glxinfo -B). - Warns/fails if it detects software rendering (e.g., llvmpipe, lavapipe, softpipe, or software rasterizer).
- Optionally runs
memtest_vulkanfor VRAM stability if installed. - Optionally runs
vkmarkfor Vulkan load testing if installed. - Keeps
glmark2as the core OpenGL benchmark/load test. - Watches kernel logs during the run for NVIDIA Xid/NVRM events, amdgpu resets/ring timeouts/VM faults, Intel i915 hangs, DRM errors, and PCIe AER errors.
- Uses process-group timeout cleanup so hung GPU tests are killed more reliably.
Example runs:
# Standard run:
python3 standalone_gpu_tester.py --client "Client Name"
# Quick smoke test:
python3 standalone_gpu_tester.py --quick --client "Client Name"
# Longer OpenGL timed soak:
python3 standalone_gpu_tester.py --glmark2-run-forever --glmark2-timeout 900 --client "Client Name"
# Skip optional tests if needed:
python3 standalone_gpu_tester.py --no-memtest --no-vkmark --client "Client Name"Output: ~/GPU_Report_YYYYMMDD_HHMMSS.md
Dedicated, vendor-specific diagnostic paths for deep GPU telemetry.
-
AMD -
amd_gpu_tester.py(v1.0.0): reads theamdgpukernel driver's sysfs/hwmon interface directly (edge/junction/memory temps, power, clocks, fan, VRAM, PCIe link), with optionalrocm-smi/amdgpu_topfor richer product/VBIOS context. -
NVIDIA -
nvidia_gpu_tester.py(v2.3.2): dynamicnvidia-smifield probing for per-second telemetry, with the 590-eraclocks_event_reasons.*throttle-field rename handled, hardenedgpu-burnresult parsing, a kernel-journal access probe, and hybrid/Optimus OpenGL routing.
Run from a desktop session as your normal user (see the sudo note near the top):
# AMD run:
python3 amd_gpu_tester.py --client "Client Name"
# NVIDIA run:
python3 nvidia_gpu_tester.py --client "Client Name"
# List detected AMD cards first (handy on multi-GPU benches):
python3 amd_gpu_tester.py --list-gpusOutput (written to the current working directory):
AMD_GPU_Report_YYYYMMDD_HHMMSS.md or NVIDIA_GPU_Report_YYYYMMDD_HHMMSS.md, plus the matching per-second telemetry CSV (amd_load_*.csv / nvidia_load_*.csv).
Features:
- Leverages
memtest_vulkanfor thorough VRAM stability testing (default 360 seconds). - Standardized load testing using both
vkmark(Vulkan) andglmark2(OpenGL/ES). - Collects continuous vendor telemetry -
amdgpusysfs/hwmon (plus optionalrocm-smi/amdgpu_top) on AMD, or dynamicnvidia-smifield probing on NVIDIA - once per second during all tests. - Scans kernel logs specifically for GPU/PCIe faults (Xid/NVRM, amdgpu resets/ring timeouts/VM faults, PCIe AER).
- Tri-state verdicts (PASS / FAIL / REVIEW) so an unmeasured check is never silently reported as a pass.
- Optional aggressive stress modes (
--glmark2-run-forever --glmark2-timeout 900), optional CUDA/compute stress (--gpu-burnon NVIDIA,--compute-stresson AMD), and optional legacy torture mode (--furmark).
Tests RAM only. Reads hardware topology from dmidecode, runs sysbench memory bandwidth, then memtester for bit-pattern stability. Output streams live so you can watch for FAILED lines in real time.
sudo python3 standalone_ram_tester.py --client "Client Name" --memtester-size 4G --passes 3Output: ~/RAM_Report_YYYYMMDD_HHMMSS.md and full memtester log.
Key v2.1 improvements:
- Keeps the PNWC banner and adds robust CLI options (
--client,--memtester-size,--passes,--auto-size). - Dynamically checks available memory before launching
memtesterto prevent hard system locks. - Saves a complete raw
memtesterlog alongside the Markdown report. - Watches kernel logs specifically for MCE, EDAC, OOM, and memory faults during the run.
- Produces a real PASS / FAIL verdict table.
Reliability burn-in tester. Fundamentally different from the diagnostic scripts - it exists to answer the question "will this machine hold up under real sustained load?" not just "is this machine functional right now?"
The diagnostic scripts run each subsystem sequentially and for seconds at a time. A machine can pass all of them and still fail 45 minutes into a real workload when:
- Thermals soak through a marginal cooler mount.
- XMP/EXPO instability surfaces under sustained memory pressure.
- A PSU with marginal capacity can't sustain simultaneous CPU + GPU + storage load.
- A VRM thermal limit kicks in that only appears under combined load.
stress_soak.py addresses this by:
- Running CPU, RAM, storage, and GPU stressors simultaneously for hours.
- Logging every thermal reading to CSV throughout the entire run.
- Watching the kernel ring buffer continuously for throttle events and hardware errors.
- Running
memtesterbefore the combined load so RAM errors surface early. - Producing a PASS / FAIL verdict with one row per check.
# Quick smoke test (~15 min) - good for first run on new hardware
sudo python3 stress_soak.py --mode quick --client "Client Name"
# Standard burn-in before returning hardware to a client (~4 hrs)
sudo python3 stress_soak.py --mode standard --client "Client Name"
# Overnight burn-in for rebuilt, re-pasted, or overclocked systems
sudo python3 stress_soak.py --mode overnight --client "Client Name"
# Skip GPU stress (headless, or CPU/RAM focus only)
sudo python3 stress_soak.py --mode standard --client "Client Name" --skip-gpu| Mode | Total | memtester | Stress | Cooldown |
|---|---|---|---|---|
quick |
~15 min | 2G ร 1 pass | 9 min | 3 min |
short |
~1 hr | 4G ร 2 passes | 45 min | 5 min |
standard |
~4 hr | 8G ร 3 passes | 3 hr | 15 min |
extended |
~8 hr | 16G ร 5 passes | 7 hr | 30 min |
overnight |
~24 hr | 16G ร 10 passes | 23 hr | 60 min |
Phase 1 - RAM validation (memtester)
Runs first with dedicated RAM. Any FAILED line is flagged immediately. If memtester finds errors, you know within the first few minutes rather than at the end of a 4-hour run.
Phase 2 - Combined stress soak (stress-ng + fio + glmark2)
-
stress-ng --cpu N --cpu-method all- cycles every stressor method across all threads. -
stress-ng --vm 2 --vm-bytes 60%- two RAM workers running bit-flip patterns simultaneously. -
fio- 4K random read/write on storage, running alongside CPU/RAM stress. -
glmark2 --run-forever- continuous GPU render loop. (v1.1: GPU execution is properly routed under the original desktop user to bypass root display-server blocks.) -
v1.1: Adds strict return-code checks for
fioandstress-ng.
Phase 3 - Cooldown monitoring Post-stress thermal monitoring. A slow cooldown with still-elevated temps indicates a cooler mount problem.
Continuous throughout all phases:
-
SensorPoller- pollssensors(now with better CPU package sensor detection in v1.1) every 5 seconds, writing every reading to a timestamped CSV. -
KernelWatcher- pollsjournalctl -kevery 30 seconds for tighter kernel fault patterns (throttle events, MCE, GPU hangs). - v1.1: Improved GPU FPS degradation logic to catch thermal throttling.
~/Soak_Report_YYYYMMDD_HHMMSS.md - full report with PASS/FAIL verdict
~/soak_temps_YYYYMMDD_HHMMSS.csv - complete thermal log, one row per 5-second sample
The report opens with a verdict table - one row per check. A single โ flags the machine as unreliable for client deployment.
| Result | Check | Detail |
|---|---|---|
| โ | RAM (memtester) | No FAILED lines detected |
| โ | CPU Temperature | Peak 74.2ยฐC (limit 100ยฐC) |
| โ | CPU Throttle Events | 12 events detected (threshold: 5) |
| โ | Hardware Errors | None detected |
| โ | stress-ng Stability | Completed full duration |
| โ | GPU Stability (glmark2) | FPS stable (max drop: 3.1%) |
Ctrl+C at any point terminates all stress child processes cleanly (v1.1 adds better process cleanup), stops monitoring threads, and writes a partial report marked โ ๏ธ INCOMPLETE. Thermal CSV data collected up to the interrupt point is preserved.
| Constant | Default | Meaning |
|---|---|---|
CPU_TEMP_WARN |
90ยฐC |
Warning flag in report (non-fatal) |
CPU_TEMP_FAIL |
100ยฐC |
Verdict FAIL - sustained throttle territory |
THROTTLE_FAIL |
5 |
Max kernel throttle events before FAIL |
HW_ERROR_FAIL |
1 |
Any kernel hardware error = FAIL |
GPU_FPS_DEGRADE_PCT |
25% |
FPS drop from firstโlast 3 scenes = GPU throttle flag |
SENSOR_POLL_S |
5 |
Seconds between sensor readings |
VM_BYTES_PCT |
60 |
% of RAM stress-ng vm workers consume |
For manual commands, package manager reference, sensor detection, turbostat, Phoronix Test Suite, fio deep-dives, and BIOS/kernel best practices, see the Manjaro_TestBench.md cheat sheet in the parent directory.
Pacific Northwest Computers ยท pnwcomputers.com ยท Vancouver, WA Last updated: 06-11-2026
- ../README.md - HardwareTesting section index
- ../Manjaro_Intel_TestBench.md - Manual test procedures these scripts automate
๐ Source: HardwareTesting/py/README.md | โ๏ธ Suggest an edit | โฌ
๏ธ Back to Hardware Testing
- ๐บ๏ธ Browse by Topic
- ๐ All Repository Files
- ๐งญ Learning Paths
- ๐ Master Guide
- ๐ค Glossary
- โ๏ธ Legal & Disclaimer
๐ Foundations & Learning
๐ก๏ธ Defense & Investigations
- Incident Response
- Open Source Intelligence (OSINT)
- Reverse Engineering
- Operations Security (OPSEC)
- Compliance
โ๏ธ Platforms & Infrastructure
๐ก Hardware, Wireless & Field Projects
- PNWC Cyberdeck
- Hardware Testing
- Hardware Hacking
- Software-Defined Radio
- Wardriving
- uConsole
- Space Security
๐งฐ Assessment & Field Reference
๐ฆ Downloads & Project Resources
๐๏ธ Additional Topics