Import bitsandbytes only if it is used #546
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
borzunov
force-pushed
the
optional-bnb-import
branch
3 times, most recently
from
b145a36
to
35bc63a
Compare
borzunov
force-pushed
the
optional-bnb-import
branch
from
35bc63a
to
c4da854
Compare
Codecov Report
@@ Coverage Diff @@
## master #546 +/- ##
==========================================
- Coverage 75.96% 75.94% -0.02%
==========================================
Files 81 81
Lines 7979 7978 -1
==========================================
- Hits 6061 6059 -2
- Misses 1918 1919 +1
|
mryab
reviewed
Jan 6, 2023
| @@ -9,13 +9,11 @@ | |||
| import numpy as np | |||
| import torch | |||
|
|
|||
| if importlib.util.find_spec("bitsandbytes") is not None: | |||
There was a problem hiding this comment.
importlib.util is no longer used, so its import can be removed
hivemind/compression/quantization.py
Outdated
| @@ -132,6 +130,9 @@ class BlockwiseQuantization(Quantization): | |||
| def quantize( | |||
| self, tensor: torch.Tensor, allow_inplace: bool = False | |||
| ) -> Tuple[np.ndarray, Tuple[np.ndarray, np.ndarray]]: | |||
| # This runs actual import only on the 1st call, copies references after that | |||
| from bitsandbytes.functional import quantize_blockwise | |||
There was a problem hiding this comment.
Since this is no longer under a conditional import (and will thus cause ImportError instead of NameError), can we wrap this statement instead of the quantize_blockwise call below with try-except? The same applies to extract
mryab
approved these changes
Jan 6, 2023
borzunov
changed the title
mryab
pushed a commit
that referenced
this pull request
Jan 6, 2023
When bitsandbytes is installed, hivemind always tries to import it even if it doesn't work. This leads to a number of undesirable behaviors:
- `import hivemind` fails if bitsandbytes is installed but fails to resolve CUDA dynamic libraries or load its own dynamic libraries. This happens all the time:
- on CPU-only hosts with CUDA installed (e.g., Colabs)
- on hosts with misconfigured CUDA (e.g., most nora envs)
- on macOS (even when the daemon is compiled correctly and it should work), etc.
- Users see the irrelevant bitsandbytes welcome message even if they don't use it. This message is currently impossible to suppress. Multiple users has reported that this is not expected when they import hivemind/petals.
This PR fixes it, so now hivemind's behavior matches with, e.g., HF transformers (import bitsandbytes only if it's necessary).
mryab
pushed a commit
that referenced
this pull request
Jan 7, 2023
When bitsandbytes is installed, hivemind always tries to import it even if it doesn't work. This leads to a number of undesirable behaviors:
- `import hivemind` fails if bitsandbytes is installed but fails to resolve CUDA dynamic libraries or load its own dynamic libraries. This happens all the time:
- on CPU-only hosts with CUDA installed (e.g., Colabs)
- on hosts with misconfigured CUDA (e.g., most nora envs)
- on macOS (even when the daemon is compiled correctly and it should work), etc.
- Users see the irrelevant bitsandbytes welcome message even if they don't use it. This message is currently impossible to suppress. Multiple users has reported that this is not expected when they import hivemind/petals.
This PR fixes it, so now hivemind's behavior matches with, e.g., HF transformers (import bitsandbytes only if it's necessary).
(cherry picked from commit e18868f)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When bitsandbytes is installed, hivemind always tries to import it even if it doesn't work. This leads to a number of undesirable behaviors:
import hivemindfails if bitsandbytes is installed but fails to resolve CUDA dynamic libraries or load its own dynamic libraries. This happens all the time:This PR fixes it, so now hivemind's behavior matches with, e.g., HF transformers (import bitsandbytes only if it's necessary).