Give ability for users to get uncertainty values. #144
Conversation
| @@ -64,9 +70,18 @@ def step(self, pool=None) -> bool: | |||
| if len(pool) > 0: | |||
| probs = self.get_probabilities(pool, **self.kwargs) | |||
| if probs is not None and (isinstance(probs, types.GeneratorType) or len(probs) > 0): | |||
| to_label = self.heuristic(probs) | |||
| to_label, uncertainty = self.heuristic.get_ranks(probs) | |||
There was a problem hiding this comment.
maybe put a flag to save uncertainties or not since it might take more time if people dont need it?
There was a problem hiding this comment.
We do not save if the path is None
|
|
||
| if partial_multi_bald_b.max() < MIN_SPREAD: | ||
| COUNT += 1 | ||
| if COUNT > 50 or len(history) >= predictions.shape[0]: | ||
| break | ||
|
|
||
| return np.array(history) | ||
|
|
||
| def reorder_indices(self): |
There was a problem hiding this comment.
why did we remove this exception?
There was a problem hiding this comment.
We raise that exception somewhere else in get_ranks
|
|
||
| def reorder_indices(self, predictions): |
There was a problem hiding this comment.
are we randomly sampling the predictions here instead of finding the variance of the iterations ?
There was a problem hiding this comment.
In Random we just sample randomly
There was a problem hiding this comment.
ah I thought it was the Variance one. my bad
tests/active/active_loop_test.py
Outdated
| _ = active_loop.step() | ||
| assert len(os.listdir(tmpdir)) == 1 | ||
| file = pjoin(tmpdir, os.listdir(tmpdir)[0]) | ||
| assert "pool=90" in file and "labelled=10" |
There was a problem hiding this comment.
the part after and is always true, no?
There was a problem hiding this comment.
It depends on how many items are labelled
Summary:
Allow the user to store uncertainties on disk when calling
ActiveLearningLoop.step.An issue that this create is that BatchBALD indices are not correlated with their uncertainty so this breaks a test. Not sure what is the best course of action.
Features:
Fix issue where NLPDataset was always imported.
Checklist:
tests/documentation_test.py).