Reorganize parallel programming lectures and improve content flow

[jstac]

Overview

This PR comprehensively reorganizes the parallel programming lectures to improve pedagogical flow, consolidate related material, and modernize the coverage of parallel computing technologies.

Major Changes

1. Expanded and Reorganized need_for_speed.md

Added new parallelization content:

• Moved "Overview" and "Types of Parallelization" sections from old parallelization.md
• Restructured into a new "Parallelization" section with subsections:
  • "Parallelization on CPUs" (covering multiprocessing and multithreading)
  • "Hardware Accelerators" (new section on GPUs and TPUs)
• Added visual aids with GPU images (geforce.png for single GPU, dgx.png for multi-GPU server)

Key improvements:

• Better progression from CPU parallelization to hardware accelerators
• Modern coverage of GPU/TPU computing for scientific workloads
• Discussion of single GPU vs multi-GPU server architectures
• Clear connection to JAX and other modern libraries

2. Renamed parallelization.md → numpy_vs_numba_vs_jax.md

Content reorganization:

• New structure with two main sections: "Vectorized operations" and "Sequential operations"
• Each section has "Speed" and "Syntax and Semantics" subsections
• Added JAX imports that were previously missing

Moved content IN:

• "A Multithreaded Ufunc" from numpy.md (shows NumPy's implicit multithreading)
• "Writing vectorized code" from jax_intro.md (demonstrates JAX vectorization with vmap)

Result:

• Creates a focused comparison lecture showing different approaches to the same computational problems
• Natural progression: NumPy → Numba → Numba with parallelization → JAX

3. Enhanced numba.md

Content additions:

• Moved "Multithreaded Loops in Numba" section from old parallelization.md
• Includes household wealth simulation example
• Demonstrates prange for CPU parallelization
• Moved associated exercises (numba_ex3, numba_ex4)

Content removals:

• Deleted "Alternatives to Numba" section (Cython and F2Py coverage)
• These technologies are less relevant to modern workflows

Organizational improvements:

• Renamed "Summary and Comments" → "Dangers and Limitations"
• Consolidated all exercises at end of lecture
• Removed reference to "following lecture on parallelization" (now self-contained)

4. Streamlined numpy.md

Content moved OUT:

• "A Multithreaded Ufunc" section → moved to numpy_vs_numba_vs_jax.md
• Kept "Implicit Multithreading in NumPy" section with matrix operation example

Result:

• More focused on core NumPy functionality
• Comparison material now consolidated in dedicated comparison lecture

5. Cleaned up jax_intro.md

Content moved OUT:

• "Writing vectorized code" section → moved to numpy_vs_numba_vs_jax.md

Link updates:

• Updated reference to point to numba.md instead of old parallelization.md

Result:

• More focused introduction to JAX fundamentals
• Vectorization comparison now in dedicated comparison lecture

6. Updated _toc.yml

• Changed reference from parallelization to numpy_vs_numba_vs_jax
• Maintains correct ordering in "High Performance Computing" section

7. Cross-repository link updates

In lecture-jax repository:

• Updated jax_intro.md link from parallelization.html → numba.html
• Points to correct exercise location

Pedagogical Benefits

1. Clearer learning progression: Students now see CPU parallelization before GPU acceleration
2. Better consolidation: Comparison material gathered in one dedicated lecture
3. Modern coverage: Added GPU/TPU content reflecting current scientific computing practices
4. Reduced redundancy: Eliminated outdated alternative tools (Cython, F2Py)
5. Improved discoverability: Related content now co-located rather than scattered

Testing

All modified lectures have been converted to Python and executed successfully:

• ✅ need_for_speed.md
• ✅ numpy.md
• ✅ numba.md
• ✅ numpy_vs_numba_vs_jax.md
• ✅ jax_intro.md

Expected errors in cells with raises-exception tags were confirmed to work correctly.

Breaking Changes

• Lecture file renamed: parallelization.md → numpy_vs_numba_vs_jax.md
• URL change: parallelization.html → numpy_vs_numba_vs_jax.html
• External link updated in lecture-jax repository

Files Changed

Modified:

• lectures/need_for_speed.md - Expanded parallelization coverage
• lectures/numpy.md - Streamlined content
• lectures/numba.md - Added multithreading, reorganized exercises
• lectures/numpy_vs_numba_vs_jax.md - Created from parallelization.md with new structure
• lectures/jax_intro.md - Removed vectorization section
• lectures/_toc.yml - Updated file reference

Added:

• lectures/_static/lecture_specific/need_for_speed/geforce.png
• lectures/_static/lecture_specific/need_for_speed/dgx.png

Deleted:

• lectures/parallelization.md (renamed to numpy_vs_numba_vs_jax.md)

External:

• lecture-jax/lectures/jax_intro.md - Updated link

[github-actions]

🚀 Deployed on https://69211c90b5b5cd4886129f22--epic-agnesi-957267.netlify.app

[jstac]

JAX Intro Lecture Improvements

This PR now includes significant improvements to jax_intro.md:

Content Additions

• ✅ Completed the "Random numbers and pure functions" section with comprehensive examples
• ✅ Added detailed comparison between NumPy's impure random generation and JAX's pure approach
• ✅ Demonstrated why explicit state management is necessary for functional purity

Structural Reorganization

Reorganized the lecture for better pedagogical flow:

New section order:

1. JAX as a NumPy Replacement (basics)
2. Functional Programming ← moved earlier
3. Random Numbers ← enhanced with integrated purity discussion
4. JIT Compilation ← includes "Compiling non-pure functions" subsection
5. Gradients
6. Exercises

Why this improves the lecture:

• Students learn the philosophy (pure functions) before seeing features that embody it
• Random number API no longer seems verbose—students understand it maintains purity
• JIT compilation benefits are clearer when students already understand pure functions
• Smooth transitions guide readers through: basics → why → how

Quality Improvements

• Fixed spelling errors (discusson, explict, parallelizaton, hardward, sleve, targetting)
• Fixed grammar errors ("uses use", "short that", "function will", etc.)
• Fixed missing jax. prefixes in code examples
• Fixed syntax errors (missing colons in timer blocks)
• Tested by converting to .py and running successfully

The lecture now provides a clear, logical progression that makes JAX's design choices intuitive rather than mysterious.

[jstac]

Latest Update: Pedagogical Improvements and Content Reorganization

This update includes improvements to sentence flow and content organization across three lectures. All changes have been tested by converting to Python with jupytext and executing successfully.

📝 numba.md - Flow Improvements

Internal Linking:

• Changed multithreading reference from Wikipedia to internal link pointing to the "Multithreaded Loops in Numba" section within the same lecture
• Added (multithreading)= label to enable the internal reference

Sentence Flow Improvements:

• Added transition phrase "Beyond speed gains from compilation" to better connect JIT compilation benefits with Numba's additional features
• Clarified NumPy arrays work well with Numba by adding "which have well-defined types"
• Changed "For example" to "Notably" when introducing class compilation (better fits the context of expanding capabilities)
• Added "Conversely" transition when contrasting prange vs ordinary range usage

Content Trimming:

• Removed "Numba will be a key part of our lectures..." sentence to keep focus on technical content

📝 numpy.md - Streamlining and Reorganization

Content Organization:

• Added "### Basics" subheading to clearly mark introductory material
• Added "### Mutability" subheading for better content structure
• Changed "Vectorized Functions" to "Universal Functions" (more accurate terminology)

Terminology Improvements:

• Emphasized key terms with bold: flat arrays, vectorized functions, ufuncs, universal functions
• Added clarifying comments to code examples (e.g., shape attribute examples now have inline explanations)

Content Removed (Redundant/Excessive):

• Removed np.asarray vs np.array comparison examples
• Removed np.genfromtxt reference (kept only np.loadtxt)
• Removed note about zero-based indices (already well-established Python convention)
• Removed redundant function examples (np.sum, np.mean)
• Removed references to older Python versions for matrix multiplication
• Removed redundant @ operator examples
• Removed manual for-loop equivalent of broadcasting operation
• Removed higher-dimensional broadcasting code examples
• Removed higher-dimensional ValueError example

Content Moved:

• Moved entire "Speed Comparisons" section to numpy_vs_numba_vs_jax.md
• Moved "Implicit Multithreading in NumPy" section to numpy_vs_numba_vs_jax.md

Enhancements:

• Added note about JAX's powerful np.vectorize capability
• Improved formatting of searchsorted() description

📝 numpy_vs_numba_vs_jax.md - Major Reorganization

Title Change:

• Changed from "Parallelization" to "NumPy vs Numba vs JAX" (more descriptive and accurate)

Bug Fix:

• Added missing imports: random, mpl_toolkits.mplot3d.axes3d.Axes3D, matplotlib.cm
• Added jax to pip install command

Content Added (from numpy.md):

• "Speed Comparisons" section
• "Vectorization vs Loops" section
• "Universal Functions" section with grid search example
• "Implicit Multithreading in NumPy" section with eigenvalue computation example

Minor Improvements:

• Changed "some examples" to "an example" for accuracy in multithreading section

✅ Testing

All three lecture files have been validated:

• ✓ Converted to Python using jupytext --to py
• ✓ Executed successfully with python
• ✓ All code cells run without errors (except intentional raises-exception tagged cells)

The reorganization improves pedagogical flow by:

1. Making numpy.md more focused on NumPy fundamentals without performance comparison distractions
2. Creating numpy_vs_numba_vs_jax.md as a dedicated comparison lecture
3. Improving readability with better transitions and clearer structure

[jstac]

Updates to need_for_speed.md

This commit includes grammar fixes and content reorganization to improve clarity and flow:

Grammar & Spelling Fixes

• Fixed incomplete sentence on line 229: added missing word "compiler"
• Fixed header level inconsistency: changed "Multi-GPU Servers" from ### to ##### to match "Single GPU Systems"

Content Reorganization

• Overview section: Restructured with clearer introduction and numbered list format
• Major Scientific Libraries: Separated "Why do we need them?" into its own subsection
• Pure Python is slow: Reorganized from "The Need for Speed" with improved logical flow
• Accelerating Python: New section grouping vectorization and JIT compiler discussions
• Vectorization: Added concrete speed comparison example with code cells
• Parallelization: Improved organization and clarity of GPU/TPU accelerator discussion

Content Improvements

• Added concrete code examples for vectorization comparison (NumPy vs pure Python loop)
• Clarified GPU/TPU benefits for scientific computing
• Improved transitions between sections
• Removed redundant explanations
• Streamlined discussion of hardware accelerators

All changes maintain the technical accuracy while improving readability and pedagogical flow.

[jstac]

Fix: Missing random module import

Fixed execution error in need_for_speed.md:

Issue: Code used random.uniform() without importing the random module, causing a NameError when running the notebook.

Fix: Added import random to the imports section (line 66).

Testing: Converted to Python using jupytext and ran successfully without errors.

[jstac]

Fix: Header hierarchy inconsistencies

Fixed invalid header structure that was causing build failures.

Issue: Four headers were using level 5 (#####) directly under level 3 (###) headers, which skips level 4 and creates an invalid hierarchy.

Headers fixed:

• "GPUs and TPUs" (line 500)
• "Why TPUs/GPUs Matter" (line 522)
• "Single GPU Systems" (line 540)
• "Multi-GPU Servers" (line 552)

Solution: Changed all four headers from ##### to #### to create proper hierarchy:

### Hardware Accelerators
    #### GPUs and TPUs
    #### Why TPUs/GPUs Matter

### Single GPUs vs GPU Servers
    #### Single GPU Systems
    #### Multi-GPU Servers

This should resolve the build failure.

[jstac]

Completed numpy_vs_numba_vs_jax lecture

This commit completes the numpy_vs_numba_vs_jax.md lecture with comprehensive timing comparisons and educational content.

What was added:

Vectorized Operations Section:

• Simple Numba loop version for comparison with NumPy vectorization
• Parallelized Numba version using prange on outer loop
• Nested parallel version using prange on both loops (demonstrates limitations)
• Discussion of parallelization overhead and when it helps vs. hurts
• Explanation of reduction problem challenges (shared variable contention)
• All versions include timing with JIT compilation overhead noted

Sequential Operations Section:

• Numba version of quadratic map with timing (first run + cached)
• JAX version using lax.scan with @partial(jax.jit, static_argnums=(1,))
• Clean implementation that avoids nested function workaround
• Timing comparisons showing both approaches are competitive
• Summary discussion emphasizing code readability differences

Key Design Decisions:

1. Machine-independent discussion: All performance commentary avoids specific numbers or ratios, focusing on patterns that will hold across different hardware
2. Educational focus: Examples deliberately show cases where "more parallelization" doesn't help, teaching users to think critically
3. Blank lines between sentences: Follows lecture convention throughout
4. Both compile and cached timing: Shows users the real overhead and benefits of JIT compilation

Performance Insights (from testing):

• Nested prange works but doesn't help for this reduction problem
• JAX and Numba are neck-and-neck for sequential operations after compilation
• JAX's cleaner syntax with static_argnums performs better than nested function approach
• Reduction operations face inherent parallelization challenges due to shared state

All code has been tested and runs successfully end-to-end.

[jstac]

Latest Update

Just pushed improvements to formatting and clarity across the parallel computing lectures:

• need_for_speed.md: Standardized header capitalization (e.g., "Where are the Bottlenecks?" → "Where are the bottlenecks?") and improved formatting consistency
• numba.md: Updated code cell types to ipython3 for consistency and removed redundant parallelization warning section
• numpy_vs_numba_vs_jax.md: Enhanced explanatory text, improved code clarity, and added missing validation checks

These changes improve the overall polish and readability of the lectures without affecting the core content.

[jstac]

@mmcky would you mind looking at this?

Was hoping to get these lectures finished this morning 😭

[jstac]

Context: i merged your changes to the environment into this PR. Might that be the issue?

[mmcky]

@jstac not sure why you're unlucky with the http connection issues! (gotta be a reason but can't think of it just yet) -- I triggered a new build and it seems to be running now.

[jstac]

Thanks @mmcky !! 🥳 🥳

[jstac]

Merging!