Skip to content

Warmup changes: only warm a few batches; extract to separate method in trainer class#43

Merged
PatrickRMiles merged 20 commits intoLBANN:mainfrom
PatrickRMiles:miles30/warmup_batches
Apr 2, 2026
Merged

Warmup changes: only warm a few batches; extract to separate method in trainer class#43
PatrickRMiles merged 20 commits intoLBANN:mainfrom
PatrickRMiles:miles30/warmup_batches

Conversation

@PatrickRMiles
Copy link
Copy Markdown
Collaborator

@PatrickRMiles PatrickRMiles commented Apr 2, 2026

  • Warms up only a few batches, not full epochs. Number of warmup batches is user-configurable. Maintains support for legacy "warmup_epochs".
  • Extracts warmup to a separate method so we can more easily distinguish warmup from training in performance traces. Adds adiak region markers around warmup call in worker.py.
  • Adds adiak region markers around self.cleanup_or_resume() in worker.py.

Based on #40

@PatrickRMiles PatrickRMiles changed the title Warmup a few batches, not full epochs Warmup changes: only warm a few batches; extract to separate method in trainer class Apr 2, 2026
@PatrickRMiles PatrickRMiles force-pushed the miles30/warmup_batches branch from ae3f84d to 66e366c Compare April 2, 2026 20:48
@PatrickRMiles PatrickRMiles merged commit b4d202c into LBANN:main Apr 2, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants