Skip to content

Conversation

@garrettwrong
Copy link
Collaborator

This fixes the OOM condition I recently encountered with full size experiments. I found a place where I was not batching during the evaluation from the stacked Coef to Image. I previously just skirted by, but no longer do with the recently added GPU calls prior to this area.

This adds a batch_size to the related classes and implements a batched evaluation loop.

I'll need to review it more. Got sidetracked with the annoying logger/warnings issue.

@garrettwrong garrettwrong added bug Something isn't working enhancement New feature or request cleanup GPU labels Jan 30, 2025
@garrettwrong garrettwrong self-assigned this Jan 30, 2025
@codecov
Copy link

codecov bot commented Jan 30, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.63%. Comparing base (ec6a5d7) to head (18a58a9).
Report is 1 commits behind head on develop.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #1226   +/-   ##
========================================
  Coverage    90.63%   90.63%           
========================================
  Files          132      132           
  Lines        13698    13705    +7     
========================================
+ Hits         12415    12422    +7     
  Misses        1283     1283           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@j-c-c j-c-c left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@garrettwrong garrettwrong marked this pull request as ready for review January 31, 2025 14:56
@garrettwrong garrettwrong requested a review from janden as a code owner January 31, 2025 14:56
@garrettwrong
Copy link
Collaborator Author

Going to merge this to move things along.

@garrettwrong garrettwrong merged commit bb67480 into develop Jan 31, 2025
44 checks passed
@garrettwrong garrettwrong deleted the dbg_oom branch January 31, 2025 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cleanup enhancement New feature or request GPU

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants