Skip to content

Fix for OOM problems on GPU at large datasets.#525

Merged
nreimers merged 5 commits intohuggingface:masterfrom
PhilipMay:gpu_oom_result_fix
Oct 27, 2020
Merged

Fix for OOM problems on GPU at large datasets.#525
nreimers merged 5 commits intohuggingface:masterfrom
PhilipMay:gpu_oom_result_fix

Conversation

@PhilipMay
Copy link
Contributor

@PhilipMay PhilipMay commented Oct 26, 2020

see #522 and #487

TODO

  • do testing
  • get review & feedback

@PhilipMay
Copy link
Contributor Author

Internal tests with large data sets show that the code fixed the oom problem.
I made the following changes:

  1. I detatch the whole embeddings once instead of each single emb - this should be more efficient
  2. If convert_to_numpy I move the whole embeddings to cpu.

IMO this PR is ready for review and merging.

@nreimers
Copy link
Collaborator

Thanks for the work. I will have a look today afternoon and merge it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants