Project: Lecture Performance Benchmarking & Monitoring System

[mmcky]

Project: Lecture Performance Benchmarking & Monitoring

Motivation

As our lecture repositories evolve — with code changes, software upgrades in environment.yml, and new releases (via publish-* tags) — we currently have no systematic way to track how these changes affect build and execution performance over time.

We need a monitoring system that can:

1. Track notebook execution times across releases and environment changes
2. Correlate performance changes with specific code commits, environment.yml updates, and publish-* releases
3. Detect regressions early when software upgrades cause slowdowns
4. Provide historical trends for each lecture across all lecture repositories

Scope

This project will serve all lecture repositories including:

• lecture-python-intro
• lecture-python-programming.myst
• lecture-python-advanced.myst
• lecture-jax
• lecture-stats
• lecture-tools-techniques
• lecture-dynamics
• and others

Proposed Approach

Create a new repository lecture-benchmark that houses:

1. A benchmarking tool (possibly integrated with jupyter-book as a Sphinx extension, or a standalone tool) that captures per-notebook execution metrics during builds
2. A storage system for benchmark results (git-friendly format like JSON/CSV, or a lightweight database)
3. Reporting & visualization — either as part of the jupyter-book build (benchmark pages) or as a standalone dashboard
4. GitHub Actions integration — automatically run benchmarks on publish-* releases and environment.yml changes

Key Design Decisions

• Integration vs standalone: Should we build a Sphinx/jupyter-book extension (enabling benchmark result pages in the built lectures) or a separate CLI tool?
• Storage format: Git-friendly flat files (JSON/CSV) vs lightweight DB (SQLite)?
• Metrics to capture: Execution time, memory usage, cell-level timings, output sizes?
• Triggering: On every cache build? On publish only? On environment.yml changes?

Related Work

The existing benchmarks/ repository contains hardware profiling and JAX-specific benchmarks. This project would be complementary — focused on lecture-level performance tracking over time rather than micro-benchmarks.

Next Steps

• Create tool-lecture-benchmark repository with a detailed PLAN.md
• Prototype the data collection approach
• Define the storage schema
• Build integration with existing CI/CD workflows