score() returns information with predictions and labels #70
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The initialization of _features is done once in __init__() It is not necessary to initialize it again when _get_features_hook() is called.
At feature extractor construction, the output_layers_id are used to return features. The model output, i.e the logits, are now also returned. As a consequence, predict_tensor() always returns a list of features + logits, even for a single tensor. Note that, for PyTorch feature extractor, a new member `self._hook_layers_id` is introduced: it is simply the output_layers_id + the last layer id (-1) for the model logits. This new member variable is used to register the hooks in prepare_extractor() and to return the features+logits in predict_tensor().
This commit prepares the new architecture of predict() where a dictionary will be returned with extra information, such as labels and logits.
predict() now returns a 2-tuple of features and extra information. Extra information contains the logits of the model and the labels.
Since predict() now returns a tuple (features, info), all calls to predict() are updated.
The two methods `score()` and `_score_tensor()` of OOD detectors now return a tuple: - a NumPy array of OOD scores - a dictionary containing extra information e.g. logits and labels. This dictionary is the same as in extractor `predict()`.
cofri
force-pushed
the
feat/score_and_predict_together
branch
from
9e8ddb5
to
0675fd4
Compare
The argument `return_labels` has been removed and predict() now returns a dict containing logits and labels. The unit tests are updated to reflect these changes.
The `score()` function of OOD detectors now returns a tuple of scores and a dictionary containing extra information. The tests now reflect these changes.
cofri
force-pushed
the
feat/score_and_predict_together
branch
from
0675fd4
to
b2cc11f
Compare
cofri
changed the title
We now use the info["logits"] to compute ood scores for these detectors
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified Files
|
paulnovello
requested changes
Aug 30, 2023
y-prudent
reviewed
Aug 30, 2023
* Now _score_tensor outputs scores and logits while labels are accessed in the score function * For consistency and to spare VRAM, labels and logits are converted to np arrays
paulnovello
approved these changes
Aug 31, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR aims at returning both score and predictions when calling
score()function. Many changes have been introduced:predict_tensor()in FeatureExtractor now returns the intermediate features + the model output (logits) as a list of tensors. The logits tensor is always the last element of the list.predict()in FeatureExtractor now returns a tuple of 2 elements: the features (legacy behaviour) and a dictionary containing the logits and the labels.return_labelsinpredict()was removed. This flag is now useless since labels are in the new dictionary.score()in detectors returns a tuple of 2 elements: the OOD scores (legacy behaviour) and the same dictionary as above.output_layers_idhas been renamed tofeature_layers_id