AI Infra Debug

Strategy

1. Baseline measurements: “Before changing anything, I measured X, Y, Z. The bottleneck appears to be A.”
2. Concrete improvement: For example: reduced latency, improved GPU utilization, removed a memory bottleneck, improved batching, reduced redundant work, or found a config issue.
3. Final writeup: 1–2 page writeup

Report

Summary:

• Investigated [system/component].
• Found bottleneck in [area].
• Implemented/tested [change].
• Result: [metric before] -> [metric after].
• Remaining opportunities: [ranked list].

Method:

• Baseline setup
• Metrics collected
• Tools used
• Assumptions

Findings:

1. ...
2. ...
3. ...

Changes made:

• ...

Risks / tradeoffs:

• ...

Recommended next steps:

1. ...
2. ...
3. ...

Strategy

Identify Metrics:

• What matters most:
  • latency
  • throughput
  • GPU utilization
  • memory
  • cost
  • reliability
  • developer simplicity
• What is the current behavior, and what would count as a useful improvement

Useful Commands:

• watch -n 1 nvidia-smi: Shows GPU memory, utilization, power, running processes.
• htop: Shows CPU usage.
• iostat -x 1: Shows disk bottlenecks, if available.
• df -h: Checks disk space.
• du -sh .: Checks directory size.

For Python timing

import time

t0 = time.perf_counter()
# work here
t1 = time.perf_counter()

print(f"Elapsed: {t1 - t0:.3f}s")

For GPU timing, use CUDA synchronization

Without torch.cuda.synchronize(), GPU timing can lie because CUDA work is asynchronous.

import time
import torch

torch.cuda.synchronize()
t0 = time.perf_counter()

# GPU work here

torch.cuda.synchronize()
t1 = time.perf_counter()

print(f"GPU elapsed: {t1 - t0:.3f}s")