Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: Graceful Degradation to CPU if CUDA isn't available #30

Merged
merged 2 commits into from
Jan 28, 2024

Conversation

KernAlan
Copy link
Contributor

Issue

#29

Solution

Include a check using torch if CUDA is available. If it's not, gracefully degrade to cpu.

Testing

Config json set to auto. But the torch cuda check fails, and the model gracefully degrades to cpu:

image

@savbell
Copy link
Owner

savbell commented Jan 26, 2024

Thanks for the PR! :)

One request: Can you add back the compute_type parameter when running on the CPU? That way, someone can set it to float32 to avoid the warning that comes up (more info: https://opennmt.net/CTranslate2/quantization.html). Thanks!

@KernAlan
Copy link
Contributor Author

Done!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants