Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nadav/llm 803 integrate the optimized models in testing package #2682

Merged
merged 16 commits into from
Oct 17, 2023
Merged

Nadav/llm 803 integrate the optimized models in testing package #2682

Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
f1b6966
onnx models
Nadav-Barak Oct 16, 2023
3f55090
validate onnx availability
Nadav-Barak Oct 16, 2023
d4634e0
bug fix
Nadav-Barak Oct 16, 2023
da69b98
bug fix 2
Nadav-Barak Oct 16, 2023
a192daf
bug fix 3
Nadav-Barak Oct 16, 2023
de1b636
debug_test
Nadav-Barak Oct 16, 2023
418d245
debug_test 2
Nadav-Barak Oct 16, 2023
c0296bf
debug_test 3
Nadav-Barak Oct 16, 2023
b5a5ae2
debug_test 4
Nadav-Barak Oct 16, 2023
13d647f
debug_test 5
Nadav-Barak Oct 16, 2023
e99f8d7
Final version with onnx
Nadav-Barak Oct 16, 2023
373fea7
Add arguments to text data property calc
Nadav-Barak Oct 16, 2023
42b819d
comment out test
Nadav-Barak Oct 16, 2023
c360a0c
Dont empty gpu
Nadav-Barak Oct 17, 2023
f599ad2
isort
Nadav-Barak Oct 17, 2023
7f5719a
Merge branch 'main' into nadav/llm-803-integrate-the-optimized-models…
Nadav-Barak Oct 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
11 changes: 0 additions & 11 deletions deepchecks/nlp/utils/text_properties.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@
from deepchecks.nlp.utils.text import cut_string, hash_text, normalize_text, remove_punctuation
from deepchecks.nlp.utils.text_properties_models import get_cmudict_dict, get_fasttext_model, get_transformer_pipeline
from deepchecks.utils.function import run_available_kwargs
from deepchecks.utils.gpu_utils import empty_gpu
from deepchecks.utils.strings import SPECIAL_CHARACTERS, format_list

__all__ = ['calculate_builtin_properties', 'get_builtin_properties_types']
Expand Down Expand Up @@ -833,7 +832,6 @@ def calculate_builtin_properties(
textblob_cache.clear()
words_cache.clear()
sentences_cache.clear()
empty_gpu(device)

if not calculated_properties:
raise RuntimeError('Failed to calculate any of the properties.')
Expand All @@ -844,15 +842,6 @@ def calculate_builtin_properties(
if k in calculated_properties
}

if cache_models:
# Move the transformers models to CPU RAM memory
for model_name in ['toxicity_classifier', 'formality_classifier', 'fluency_classifier']:
if model_name in kwargs:
kwargs[model_name].model.to('cpu')

# Clean all remaining RAM:
empty_gpu(device)

return calculated_properties, properties_types


Expand Down