Refactor/feature layers id in fit

.bumpversion.cfg

@@ -1,8 +1,8 @@
 [bumpversion]
-current_version = 0.0.1
+current_version = 0.1.0
 commit = True
 tag = True
 
-[bumpversion:file:oodeel/setup.py]
+[bumpversion:file:setup.py]
 
 [bumpversion:file:oodeel/__init__.py]

README.md

@@ -44,8 +44,6 @@ mls.fit(model) # A tensorflow or torch model
 scores, info = mls.score(ds) # ds is a tf.data.Dataset or a torch.DataLoader
 ```
 
-**Disclaimer**: It is still very much a work in progress, see issues and [development roadmap](#development-roadmap). Please use the lib carefully!
-
 # Table of contents
 
 - [Installation](#installation)
@@ -61,15 +59,19 @@ scores, info = mls.score(ds) # ds is a tf.data.Dataset or a torch.DataLoader
 
 # Installation
 
-**Oodeel** requires some stuff and several libraries including Numpy. Installation can be done using:
+Installation can be done using:
 
 ```bash
-git clone https://github.com/deel-ai/oodeel.git
-cd oodeel
-make install-env-${BACKEND}
-
+pip install oodeel
 ```
-where `${BACKEND}` is either `torch` or `tensorflow` depending on the user's backend.
+
+oodeel requires either `tensorflow` or `pytorch` to be already installed (it will not install them automatically not to mess-up with existing installations). It is regularly tested with:
+
+|Python version|Pytorch version|Tensorflow version|
+|---|---|---|
+|`3.8`| `1.11`| `2.5`|
+|`3.9`|`1.13` | `2.8`|
+|`3.10`| `2.00` | `2.11`|
 
 # Quick Start
 
@@ -98,7 +100,7 @@ Load a dataset and split it into an in-distribution dataset and ou-of-distributi
 from oodeel.datasets import OODDataset
 
 in_labels = [0, 1, 2, 3, 4]
-oods_in, oods_out = oods_test.assign_ood_labels_by_class(in_labels=in_labels)
+oods_in, oods_out = oods_test.split_by_class(in_labels=in_labels)
 ds_in = oods_in.prepare(batch_size=batch_size)
 ds_out = oods_out.prepare(batch_size=batch_size)
 ```
@@ -127,6 +129,31 @@ metrics = bench_metrics(
     )
 ```
 
+### And visualize the results!
+
+2D t-SNE (3D is also available).
+
+```python
+plot_2D_features(
+    model=model,
+    in_dataset=ds_in,
+    out_dataset=ds_out,
+    output_layer_id=-2,
+)
+```
+<p align="center">
+  <img src="docs/assets/tsne.png" alt="TSNE" />
+</p>
+
+Classical histograms and AUROC curve.
+```python
+plot_ood_scores(scores_in, scores_out, log_scale=False)
+plot_roc_curve(scores_in, scores_out)
+```
+<p align="center">
+  <img src="docs/assets/auroc.png" alt="AUROC" />
+</p>
+
 # Tutorials
 
 We propose some tutorials to get familiar with the library and its API. See the Tutorial section of the [doc](https://deel-ai.github.io/oodeel/)
@@ -161,17 +188,8 @@ Currently, **oodeel** includes the following baselines:
 **Oodeel** also includes standard training functions with data augmentation and learning rate scheduler for toy convnet models or models from `keras.applications` in [tf_training_tools.py](https://github.com/deel-ai/oodeel/tree/master/oodeel/utils/tf_training_tools.py) and `torchvision.models` in [torch_training_tools.py](https://github.com/deel-ai/oodeel/tree/master/oodeel/utils/torch_training_tools.py) files. These functions come in handy for benchmarks like *leave-k-classes-out* that requires retraining models on a subset of dataset classes.
 # Development Roadmap
 
-## Roadmap to first release:
-- [x] The library works for `keras` models
-- [x] Unification of tutorial notebooks
-- [x] Validation of all methods for pytorch using `TorchOperator`, making oodeel compatible with both tensorflow and pytorch models.
-- [x] Integration of `TorchDataHandler` to alleviate the need of `tf.data.Dataset` when using pytorch. At this stage, oodeel will no more require any tensorflow components when using pytorch, and vice-versa.
-- [x] Revise docstring and type hinting
-- [x] Set up the doc
-
-## What's next?
 - [ ] More baselines!
-- [ ] A module for thorough visualizations (result plots and feature space visualizations)
+- [x] A module for thorough visualizations (result plots and feature space visualizations)
 - [ ] Integrate model loading and uploading with [hugginface's transformers](https://huggingface.co/docs/transformers/index) library for pretraining
 - [ ] Extend the library to more diverse tasks like object detection, segmentation, NLP ...
 - [ ] Towards OOD Generalization?

docs/api/plots.md

@@ -0,0 +1,14 @@
+::: oodeel.eval.plots.features
+    options:
+        show_root_toc_entry: True
+        inherited_members: True
+
+::: oodeel.eval.plots.metrics
+    options:
+        show_root_toc_entry: True
+        inherited_members: True
+
+::: oodeel.eval.plots.plotly
+    options:
+        show_root_toc_entry: True
+        inherited_members: True

docs/index.md

@@ -37,12 +37,10 @@
 from oodeel.methods import MLS
 
 mls = MLS()
-mls.fit(model)
-scores, info = mls.score(ds)
+mls.fit(model) # A tensorflow or torch model
+scores, info = mls.score(ds) # ds is a tf.data.Dataset or a torch.DataLoader
 ```
 
-**Disclaimer**: It is still very much a work in progress, see issues and [development roadmap](#development-roadmap). Please use the lib carefully!
-
 # Table of contents
 
 - [Installation](#installation)
@@ -58,15 +56,18 @@ scores, info = mls.score(ds)
 
 # Installation
 
-**Oodeel** requires some stuff and several libraries including Numpy. Installation can be done using:
+Installation can be done using:
 
 ```bash
-git clone https://github.com/deel-ai/oodeel.git
-cd oodeel
-make install-env-${BACKEND}
-
+pip install oodeel
 ```
-where `${BACKEND}` is either `torch` or `tensorflow` depending on the user's backend.
+oodeel requires either `tensorflow` or `pytorch` to be already installed (it will not install them automatically not to mess-up with existing installations). It is regularly tested with:
+
+|Python version|Pytorch version|Tensorflow version|
+|---|---|---|
+|`3.8`| `1.11`| `2.5`|
+|`3.9`|`1.13` | `2.8`|
+|`3.10`| `2.00` | `2.11`|
 
 # Quick Start
 
@@ -95,7 +96,7 @@ Load a dataset and split it into an in-distribution dataset and ou-of-distributi
 from oodeel.datasets import OODDataset
 
 in_labels = [0, 1, 2, 3, 4]
-oods_in, oods_out = oods_test.assign_ood_labels_by_class(in_labels=in_labels)
+oods_in, oods_out = oods_test.split_by_class(in_labels=in_labels)
 # info contains model predictions and labels if avail
 ds_in = oods_in.prepare(batch_size=batch_size)
 ds_out = oods_out.prepare(batch_size=batch_size)
@@ -123,6 +124,30 @@ metrics = bench_metrics(
     metrics = ["auroc", "fpr95tpr"],
     )
 ```
+### And visualize the results!
+
+2D t-SNE (3D is also available).
+
+```python
+plot_2D_features(
+    model=model,
+    in_dataset=ds_in,
+    out_dataset=ds_out,
+    output_layer_id=-2,
+)
+```
+<p align="center">
+  <img src="assets/tsne.png" alt="TSNE" />
+</p>
+
+Classical histograms and AUROC curve.
+```python
+plot_ood_scores(scores_in, scores_out, log_scale=False)
+plot_roc_curve(scores_in, scores_out)
+```
+<p align="center">
+  <img src="assets/auroc.png" alt="AUROC" />
+</p>
 
 # Tutorials
 
@@ -156,15 +181,6 @@ Currently, **oodeel** includes the following baselines:
 **Oodeel** also includes standard training functions with data augmentation and learning rate scheduler for toy convnet models or models from `keras.applications` in [tf_training_tools.py](https://github.com/deel-ai/oodeel/tree/master/oodeel/utils/tf_training_tools.py) and `torchvision.models` in [torch_training_tools.py](https://github.com/deel-ai/oodeel/tree/master/oodeel/utils/torch_training_tools.py) files. These functions come in handy for benchmarks like *leave-k-classes-out* that requires retraining models on a subset of dataset classes.
 # Development Roadmap
 
-## Roadmap to first release:
-- [x] The library works for `keras` models
-- [x] Unification of tutorial notebooks
-- [x] Validation of all methods for pytorch using `TorchOperator`, making oodeel compatible with both tensorflow and pytorch models.
-- [x] Integration of `TorchDataHandler` to alleviate the need of `tf.data.Dataset` when using pytorch. At this stage, oodeel will no more require any tensorflow components when using pytorch, and vice-versa.
-- [x] Revise docstring and type hinting
-- [x] Set up the doc
-
-## What's next?
 - [ ] More baselines!
 - [ ] A module for thorough visualizations (result plots and feature space visualizations)
 - [ ] Integrate model loading and uploading with [hugginface's transformers](https://huggingface.co/docs/transformers/index) library for pretraining

docs/notebooks/tensorflow/demo_dknn_tf.ipynb

@@ -116,8 +116,8 @@
                 "\n",
                 "# 2- Split ID / OOD data depending on label value:\n",
                 "# in-distribution: MNIST[0-4] / out-of-distribution: MNIST[5-9]\n",
-                "ds_train, _ = ds_train.assign_ood_labels_by_class(in_labels)\n",
-                "oods_in, oods_out = data_test.assign_ood_labels_by_class(in_labels)\n",
+                "ds_train, _ = ds_train.split_by_class(in_labels)\n",
+                "oods_in, oods_out = data_test.split_by_class(in_labels)\n",
                 "\n",
                 "\n",
                 "# 3- Prepare data (preprocess, shuffle, batch)\n",
@@ -246,8 +246,8 @@
             ],
             "source": [
                 "# === DKNN scores ===\n",
-                "dknn = DKNN(nearest=50, feature_layers_idd=[-2])\n",
-                "dknn.fit(model, ds_train)\n",
+                "dknn = DKNN(nearest=50)\n",
+                "dknn.fit(model, feature_layers_id=[-2], fit_dataset=ds_train)\n",
                 "scores_in, _ = dknn.score(ds_in)\n",
                 "scores_out, _ = dknn.score(ds_out)\n",
                 "\n",
@@ -428,8 +428,8 @@
             ],
             "source": [
                 "# === DKNN scores ===\n",
-                "dknn = DKNN(nearest=50, feature_layers_idd=[-2])\n",
-                "dknn.fit(model, ds_fit)\n",
+                "dknn = DKNN(nearest=50)\n",
+                "dknn.fit(model, feature_layers_id=[-2], fit_dataset=ds_fit)\n",
                 "scores_in, _ = dknn.score(ds_in)\n",
                 "scores_out, _ = dknn.score(ds_out)\n",
                 "\n",
@@ -471,7 +471,7 @@
             "name": "python",
             "nbconvert_exporter": "python",
             "pygments_lexer": "ipython3",
-            "version": "3.8.17"
+            "version": "3.9.12"
         },
         "orig_nbformat": 4,
         "vscode": {

docs/notebooks/tensorflow/demo_entropy_tf.ipynb

@@ -116,8 +116,8 @@
                 "\n",
                 "# 2- Split ID / OOD data depending on label value:\n",
                 "# in-distribution: MNIST[0-4] / out-of-distribution: MNIST[5-9]\n",
-                "ds_train, _ = ds_train.assign_ood_labels_by_class(in_labels)\n",
-                "oods_in, oods_out = data_test.assign_ood_labels_by_class(in_labels)\n",
+                "ds_train, _ = ds_train.split_by_class(in_labels)\n",
+                "oods_in, oods_out = data_test.split_by_class(in_labels)\n",
                 "\n",
                 "\n",
                 "# 3- Prepare data (preprocess, shuffle, batch)\n",
@@ -469,7 +469,7 @@
             "name": "python",
             "nbconvert_exporter": "python",
             "pygments_lexer": "ipython3",
-            "version": "3.8.17"
+            "version": "3.9.12"
         },
         "orig_nbformat": 4,
         "vscode": {

docs/notebooks/tensorflow/demo_mahalanobis_tf.ipynb

@@ -116,8 +116,8 @@
                 "\n",
                 "# 2- Split ID / OOD data depending on label value:\n",
                 "# in-distribution: MNIST[0-4] / out-of-distribution: MNIST[5-9]\n",
-                "ds_train, _ = ds_train.assign_ood_labels_by_class(in_labels)\n",
-                "oods_in, oods_out = data_test.assign_ood_labels_by_class(in_labels)\n",
+                "ds_train, _ = ds_train.split_by_class(in_labels)\n",
+                "oods_in, oods_out = data_test.split_by_class(in_labels)\n",
                 "\n",
                 "\n",
                 "# 3- Prepare data (preprocess, shuffle, batch)\n",
@@ -246,8 +246,8 @@
             ],
             "source": [
                 "# === Mahalanobis scores ===\n",
-                "mahalanobis = Mahalanobis(eps=0.002, feature_layers_id=[-2])\n",
-                "mahalanobis.fit(model, ds_train)\n",
+                "mahalanobis = Mahalanobis(eps=0.002)\n",
+                "mahalanobis.fit(model, feature_layers_id=[-2], fit_dataset=ds_train)\n",
                 "scores_in, _ = mahalanobis.score(ds_in)\n",
                 "scores_out, _ = mahalanobis.score(ds_out)\n",
                 "clear_output()\n",
@@ -407,8 +407,8 @@
             "outputs": [],
             "source": [
                 "# === Mahalanobis OOD scores ===\n",
-                "mahalanobis = Mahalanobis(eps=0.002, feature_layers_id=[-2])\n",
-                "mahalanobis.fit(model, ds_fit)\n",
+                "mahalanobis = Mahalanobis(eps=0.002)\n",
+                "mahalanobis.fit(model, feature_layers_id=[-2], fit_dataset=ds_fit)\n",
                 "scores_in, _ = mahalanobis.score(ds_in)\n",
                 "scores_out, _ = mahalanobis.score(ds_out)\n",
                 "\n",
@@ -450,7 +450,7 @@
             "name": "python",
             "nbconvert_exporter": "python",
             "pygments_lexer": "ipython3",
-            "version": "3.9.18"
+            "version": "3.9.12"
         },
         "orig_nbformat": 4,
         "vscode": {

docs/notebooks/tensorflow/demo_mls_msp_tf.ipynb

@@ -121,8 +121,8 @@
                 "\n",
                 "# 2- Split ID / OOD data depending on label value:\n",
                 "# in-distribution: MNIST[0-4] / out-of-distribution: MNIST[5-9]\n",
-                "ds_train, _ = ds_train.assign_ood_labels_by_class(in_labels)\n",
-                "oods_in, oods_out = data_test.assign_ood_labels_by_class(in_labels)\n",
+                "ds_train, _ = ds_train.split_by_class(in_labels)\n",
+                "oods_in, oods_out = data_test.split_by_class(in_labels)\n",
                 "\n",
                 "\n",
                 "# 3- Prepare data (preprocess, shuffle, batch)\n",
@@ -616,7 +616,7 @@
             "name": "python",
             "nbconvert_exporter": "python",
             "pygments_lexer": "ipython3",
-            "version": "3.8.17"
+            "version": "3.9.12"
         },
         "orig_nbformat": 4,
         "vscode": {

docs/notebooks/tensorflow/demo_odin_tf.ipynb

@@ -114,8 +114,8 @@
                 "\n",
                 "# 2- Split ID / OOD data depending on label value:\n",
                 "# in-distribution: MNIST[0-4] / out-of-distribution: MNIST[5-9]\n",
-                "ds_train, _ = ds_train.assign_ood_labels_by_class(in_labels)\n",
-                "oods_in, oods_out = data_test.assign_ood_labels_by_class(in_labels)\n",
+                "ds_train, _ = ds_train.split_by_class(in_labels)\n",
+                "oods_in, oods_out = data_test.split_by_class(in_labels)\n",
                 "\n",
                 "\n",
                 "# 3- Prepare data (preprocess, shuffle, batch)\n",
@@ -468,7 +468,7 @@
             "name": "python",
             "nbconvert_exporter": "python",
             "pygments_lexer": "ipython3",
-            "version": "3.8.17"
+            "version": "3.9.12"
         },
         "orig_nbformat": 4,
         "vscode": {