Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix for OOM problems on GPU at large datasets. #525

Merged
merged 5 commits into from
Oct 27, 2020
Merged

Fix for OOM problems on GPU at large datasets. #525

merged 5 commits into from
Oct 27, 2020

Conversation

PhilipMay
Copy link
Contributor

@PhilipMay PhilipMay commented Oct 26, 2020 •
edited
Loading

see #522 and #487

TODO

  • do testing
  • get review & feedback

@PhilipMay PhilipMay mentioned this pull request Oct 26, 2020
Closed
@PhilipMay
Copy link
Contributor Author

Internal tests with large data sets show that the code fixed the oom problem.
I made the following changes:

  1. I detatch the whole embeddings once instead of each single emb - this should be more efficient
  2. If convert_to_numpy I move the whole embeddings to cpu.

IMO this PR is ready for review and merging.

@nreimers
Copy link
Member

Thanks for the work. I will have a look today afternoon and merge it

@PhilipMay PhilipMay mentioned this pull request Oct 27, 2020
Closed
@nreimers nreimers merged commit 199e5f7 into UKPLab:master Oct 27, 2020
@PhilipMay PhilipMay deleted the gpu_oom_result_fix branch October 28, 2020 20:34
@chschroeder chschroeder mentioned this pull request Dec 19, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@PhilipMay @nreimers