Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update CUDA out of memory mesage with private pool info #124673

Closed
wants to merge 6 commits into from
Closed

Update CUDA out of memory mesage with private pool info #124673

Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b634475
Update
isuruf Apr 22, 2024
2110e02
Update
isuruf Apr 22, 2024
f289659
Update
isuruf Apr 23, 2024
88b8e42
Update
isuruf Apr 23, 2024
7176983
Update
isuruf Apr 24, 2024
ca6dc21
Update
isuruf May 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 19 additions & 3 deletions c10/cuda/CUDACachingAllocator.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1125,6 +1125,19 @@ class DeviceCachingAllocator {
device_free);
}

size_t allocated_in_private_pools = 0;
auto get_size_block = [](const BlockPool& pool) {
size_t res = 0;
for (const auto& block : pool.blocks) {
res += block->size;
}
return res;
};
for (const auto& p : graph_pools) {
allocated_in_private_pools += get_size_block(p.second->large_blocks);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This likely needs to happen before the mutex unlock on line 1119.

allocated_in_private_pools += get_size_block(p.second->small_blocks);
}

// "total capacity": total global memory on GPU
// "allowed": memory is allowed to use, which set by fraction.
// "already allocated": memory allocated by the program using the
Expand Down Expand Up @@ -1157,9 +1170,12 @@ class DeviceCachingAllocator {
" is free. ",
proc_info,
"Of the allocated memory ",
format_size(allocated_bytes),
" is allocated by PyTorch, and ",
format_size(reserved_bytes - allocated_bytes),
format_size(allocated_bytes + allocated_in_private_pools),
" is allocated by PyTorch, with ",
format_size(allocated_in_private_pools),
" allocated in private pools, and ",
format_size(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should print this if there aren't any allocated private pools. Also, maybe worth explicitly mentioning cudagraphs here? for e.g. users doing torch.compile(mode='reduce-overhead') they might not make the connection

reserved_bytes - allocated_bytes - allocated_in_private_pools),
" is reserved by PyTorch but unallocated.",
" If reserved but unallocated memory is large try setting",
" PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid"
Expand Down
Loading