-
Notifications
You must be signed in to change notification settings - Fork 104
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement streaming algorithm #6316
Conversation
Codecov Report
@@ Coverage Diff @@
## main #6316 +/- ##
==========================================
- Coverage 82.68% 82.65% -0.04%
==========================================
Files 347 347
Lines 21330 21338 +8
Branches 862 862
==========================================
Hits 17637 17637
- Misses 3394 3402 +8
Partials 299 299
... and 4 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
060f77a
to
643b70c
Compare
8ffd6d9
to
d0bd32c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, does not hurt if @oyvindeide takes a look too
Instead of loading all parameter groups into memory before updating, the streaming algorithm loads one parameter group into memory, updates it, and writes it back to disk. This is particularly useful when multiple large fields are updated.
d0bd32c
to
613d1fe
Compare
Based on input from @oyvindeide , I have removed the usage of |
Instead of loading all parameter groups into memory before updating, the streaming algorithm loads one parameter group into memory, updates it, and writes it back to disk.
This is particularly useful when multiple large fields are updated.
Issue
Resolves #5905
Pre review checklist
Ground Rules),
and changes to existing code have good test coverage.
Pre merge checklist