NNabla trainer

docs/tutorial/library.md

@@ -12,60 +12,96 @@ if os.path.exists('examples/mlruns'):
     shutil.rmtree('examples/mlruns')
 ```
 
-## PyTorch vs TensorFlow
+## Base Parameter File
 
-In this section we compare two libraries and show that using different libraries on the same problem is straightforward.
+Before examples, we write two base or *template* parameter files which are extened by other parameter files.
+
+#File A base parameter YAML file for various libraries (data.yml) {%=/examples/data.yml%}
+
+#File A base parameter YAML file for various libraries (base.yml) {%=/examples/base.yml%} {#base#}
+
+In `base.yml`, the first line "`extends: data`" means that the file extends (or includes, in this case) `data.yml`.
+
+## Neural Network Libraries
+
+In this section we compare three libraries ([TensorFlow](https://www.tensorflow.org/), [NNabla](https://nnabla.org/), and [PyTorch](https://pytorch.org/)), and show that using different libraries on the same problem is straightforward.
+
+```python
+import tensorflow
+import nnabla
+import torch
+print(tensorflow.__version__)
+print(nnabla.__version__)
+print(torch.__version__)
+```
 
 First define models:
 
-#File A Model definition in PyTorch (rectangle/torch.py) {%=/examples/rectangle/torch.py%}
+#File A Model definition in TensorFlow (rectangle/tensorflow.py) {%=/examples/rectangle/tensorflow.py%}
 
-#File A Model definition in TensorFlow (rectangle/tf.py) {%=/examples/rectangle/tf.py%}
+#File A Model definition in NNabla (rectangle/nnabla.py) {%=/examples/rectangle/nnabla.py%}
+
+#File A Model definition in PyTorch (rectangle/torch.py) {%=/examples/rectangle/torch.py%}
 
 For simplicity, the TensorFlow model is defined by using the `keras.Sequential()`, so that we call the `create_model()` function to get the model.
 
 Next, write parameter YAML files:
 
-#File A parameter YAML for PyTorch (torch.yml) {%=/examples/torch.yml%}
+#File A parameter YAML file for TensorFlow (tensorflow.yml) {%=/examples/tensorflow.yml%}
 
-#File A parameter YAML for TensorFlow (tf.yml) {%=/examples/tf.yml%}
+#File A parameter YAML file for NNabla (nnabla.yml) {%=/examples/nnabla.yml%}
 
-These YAML files have a very similar structure. The first difference comes from that, in PyTorch, an optimizer needs model parameters at the time of instantiation and a scheduler needs an optimizer too, while, in TensorFlow, an optimizer and scheduler can be instantiated without other related instances.  The second difference is `loss` functions. The Pytorch YAML file writes the fullname, while the TensorFlow one writes an abbreviation.
+#File A parameter YAML fine for PyTorch (torch2.yml) {%=/examples/torch2.yml%}
 
+These YAML files are very similar. The only difference is that, in PyTorch, an optimizer needs model parameters at the time of instantiation.
 
 !!! note
     The `model` for TensorFlow is a function. A new `call` key is used. (But you can use `class`, too, or `call` for a class, vice versa, because both a class and function are *callable*.)
 
-Next, create two runs.
+Next, create three runs.
 
 ```python
 import ivory
 
 client = ivory.create_client("examples")
-run_torch = client.create_run('torch')
-run_tf = client.create_run('tf')
+run_tf = client.create_run('tensorflow')
+run_nn = client.create_run('nnabla')
+run_torch = client.create_run('torch2')
 ```
 
 For comparison, equalize initial parameters.
 
 ```python
 import torch
 
-for w_tf, w_torch in zip(run_tf.model.weights, run_torch.model.parameters()):
+# These three lines are only needed for this example.
+run, trainer = run_nn, run_nn.trainer
+run.model.build(trainer.loss, run.datasets.train, trainer.batch_size)
+run.optimizer.set_parameters(run.model.parameters())
+
+ws_tf = run_tf.model.weights
+ws_nn = run_nn.model.parameters().values()
+ws_torch = run_torch.model.parameters()
+for w_tf, w_nn, w_torch in zip(ws_tf, ws_nn, ws_torch):
+    w_nn.data.data = w_tf.numpy()
     w_torch.data = torch.tensor(w_tf.numpy().T)
 ```
 
 Then, start the runs.
 
 ```python
-run_torch.start()
+run_tf.start('both')  # Slower due to usage of GPU for a simple network.
+```
+
+```python
+run_nn.start('both')
 ```
 
 ```python
-run_tf.start()
+run_torch.start('both')
 ```
 
-Visualize the results:
+Metrics during training are almost same. Visualize the results:
 
 ```python
 import matplotlib.pyplot as plt
@@ -79,35 +115,58 @@ def plot(run):
     plt.xlabel('Target values')
     plt.ylabel('Predicted values')
 
-for run in [run_tf, run_torch]:
+for run in [run_tf, run_nn, run_torch]:
     plot(run)
 ```
 
 Actual outputs are like below:
 
 ```python
-x = run_tf.datasets.test[:10][1]
+x = run_tf.datasets.val[:5][1]
 run_tf.model.predict(x)
 ```
 
 ```python
+x = run_nn.datasets.val[:5][1]
+run_nn.model(x)
+```
+
+```python
+x = run_torch.datasets.val[:5][1]
 run_torch.model(torch.tensor(x))
 ```
 
+You can *ensemble* these resutls, although this is meaningless in this example.
+
+```python
+from ivory.callbacks.results import concatenate
+
+results = concatenate(list(run.results for run in [run_tf, run_nn, run_torch]))
+index = results.val.index.argsort()
+results.val.output[index[:15]]
+```
+
+```python
+reduced_results = results.mean()
+reduced_results.val.output[:5]
+```
+
 ## Scikit-learn
 
 Ivory can optimize various scikit-learn's estimators. Here are som examples.
 
-### RandomForestRegressor
-
-#File A parameter YAML for RandomForestRegressor (rfr.yml) {%=/examples/rfr.yml%}
+#File A base parameter YAML file for various estimators (data2.yml) {%=/examples/data2.yml%}
 
 The `dataset` has a `transform` argument. This function reshapes the target array to match the shape for scikit-learn estimators (1D from 2D).
 
 {{ import rectangle.data }}
 
 #Code rectangle.data.transform() {{ rectangle.data.transform # inspect }}
 
+### RandomForestRegressor
+
+#File A parameter YAML file for RandomForestRegressor (rfr.yml) {%=/examples/rfr.yml%}
+
 There are nothing different to start a run.
 
 ```python
@@ -123,7 +182,7 @@ plot(run)
 
 ### Ridge
 
-#File A parameter YAML for Ridge (ridge.yml) {%=/examples/ridge.yml%}
+#File A parameter YAML file for Ridge (ridge.yml) {%=/examples/ridge.yml%}
 
 Because the Ridge estimator has no `criterion` attribute, you have to specify metrics if you need. A `mse` key has empty (`None`) value. In this case, the default function (`sklearn.metrics.mean_squared_error()`) is chosen. On the other hand, `mse_2`'s value is a custom funtion's name:
 
@@ -149,7 +208,7 @@ For LightGBM, Ivory implements two estimators:
 * `ivory.lightgbm.estimator.Regressor`
 * `ivory.lightgbm.estimator.Classifier`
 
-#File A parameter YAML for LightGBM (lgb.yml) {%=/examples/lgb.yml%}
+#File A parameter YAML file for LightGBM (lgb.yml) {%=/examples/lgb.yml%}
 
 ```python
 run = client.create_run('lgb')

examples/base.yml

@@ -0,0 +1,15 @@
+extends: data
+model:
+  hidden_sizes: [20, 30]
+optimizer:
+  lr: 1e-3
+results:
+metrics:
+monitor:
+  metric: val_loss
+trainer:
+  loss: mse
+  batch_size: 5
+  epochs: 5
+  shuffle: false
+  verbose: 2

examples/data.yml

@@ -0,0 +1,6 @@
+datasets:
+  data:
+    class: rectangle.data.Data
+    n_splits: 4
+  dataset:
+  fold: 0

examples/data2.yml

@@ -0,0 +1,4 @@
+extends: data
+datasets:
+  dataset:
+    transform: rectangle.data.transform

examples/lgb.yml

@@ -1,10 +1,4 @@
-datasets:
-  data:
-    class: rectangle.data.Data
-    n_splits: 4
-  dataset:
-    transform: rectangle.data.transform
-  fold: 0
+extends: data2
 estimator:
   class: ivory.lightgbm.estimator.Regressor
   boosting_type: gbdt

examples/nnabla.yml

@@ -1,25 +1,6 @@
 library: nnabla
-datasets:
-  data:
-    class: rectangle.data.Data
-    n_splits: 4
-  dataset:
-  fold: 0
+extends: base
 model:
   class: rectangle.nnabla.Model
-  hidden_sizes: [20, 30]
 optimizer:
   class: nnabla.solvers.Sgd
-  lr: 1e-3
-results:
-metrics:
-monitor:
-  metric: val_loss
-early_stopping:
-  patience: 10
-trainer:
-  loss: mse
-  batch_size: 10
-  epochs: 10
-  shuffle: true
-  verbose: 2

examples/rfr.yml

@@ -1,11 +1,5 @@
 library: sklearn
-datasets:
-  data:
-    class: rectangle.data.Data
-    n_splits: 4
-  dataset:
-    transform: rectangle.data.transform
-  fold: 0
+extends: data2
 estimator:
   model: sklearn.ensemble.RandomForestRegressor
   n_estimators: 5

examples/ridge.yml

@@ -1,11 +1,5 @@
 library: sklearn
-datasets:
-  data:
-    class: rectangle.data.Data
-    n_splits: 4
-  dataset:
-    transform: rectangle.data.transform
-  fold: 0
+extends: data2
 estimator:
   model: sklearn.linear_model.Ridge
 results:

examples/tensorflow.yml

@@ -0,0 +1,6 @@
+library: tensorflow
+extends: base
+model:
+  call: rectangle.tensorflow.create_model
+optimizer:
+  class: tensorflow.keras.optimizers.SGD

examples/torch2.yml

@@ -0,0 +1,7 @@
+library: torch
+extends: base
+model:
+  class: rectangle.torch.Model
+optimizer:
+  class: torch.optim.SGD
+  _: $.model.parameters()

ivory/__init__.py

@@ -1,4 +1,4 @@
-__version__ = "0.3.0"
+__version__ = "0.3.1"
 
 from ivory.core.client import create_client
 

ivory/callbacks/results.py

@@ -1,5 +1,5 @@
 """A container to store training, validation and test results. """
-from typing import Callable, Dict, Iterable, List, Optional
+from typing import Callable, Dict, Iterable, Optional
 
 import numpy as np
 import pandas as pd
@@ -42,16 +42,16 @@ def result_dict(self):
         dict = ivory.core.collections.Dict()
         return dict(index=self.index, output=self.output, target=self.target)
 
-    def set(self, **kwargs):
-        results = {}
-        for key, value in kwargs.items():
-            dict = ivory.core.collections.Dict()
-            if len(value) == 3:
-                dict(index=value[0], output=value[1], target=value[2])
-            else:
-                dict(index=value[0], output=value[1], target=None)
-            results[key] = dict
-        super().set(**results)
+    # def set(self, **kwargs):
+    #     results = {}
+    #     for key, value in kwargs.items():
+    #         dict = ivory.core.collections.Dict()
+    #         if len(value) == 3:
+    #             dict(index=value[0], output=value[1], target=value[2])
+    #         else:
+    #             dict(index=value[0], output=value[1], target=None)
+    #         results[key] = dict
+    #     super().set(**results)
 
     def mean(self):
         results = Results()
@@ -98,11 +98,11 @@ def result_dict(self):
         return super().result_dict()
 
 
-def stack(x: List[np.ndarray]) -> np.ndarray:
-    if x[0].ndim == 1:
-        return np.hstack(x)
-    else:
-        return np.vstack(x)
+# def stack(x: List[np.ndarray]) -> np.ndarray:
+#     if x[0].ndim == 1:
+#         return np.hstack(x)
+#     else:
+#         return np.vstack(x)
 
 
 def concatenate(

ivory/core/data.py

@@ -24,9 +24,7 @@
     Use a `'def'` key for `dataset` instead of `'class'`.
     See [Tutorial](/tutorial/data)
 """
-
-
-from dataclasses import InitVar, dataclass
+from dataclasses import dataclass
 from typing import Callable, Optional, Tuple
 
 import numpy as np