Skip to content

Commit

Permalink
Ravin Kohli: [FIX] Additional metrics during training (#316)
Browse files Browse the repository at this point in the history
  • Loading branch information
Github Actions committed Nov 12, 2021
1 parent 1f28082 commit 8f70383
Show file tree
Hide file tree
Showing 23 changed files with 175 additions and 177 deletions.
Binary file modified development/_downloads/bc82bea3a5dd7bdba60b65220891d9e5/examples_python.zip
View file
Binary file not shown.
Binary file modified development/_downloads/fb625db3c50d423b1b7881136ffdeec8/examples_jupyter.zip
View file
Binary file not shown.
Binary file modified development/_images/sphx_glr_example_visualization_001.png
View file
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified development/_images/sphx_glr_example_visualization_thumb.png
View file
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
20 changes: 7 additions & 13 deletions development/_sources/examples/20_basics/example_image_classification.rst.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -85,19 +85,13 @@ Image Classification
Pipeline Random Config:
________________________________________
Configuration:
image_augmenter:GaussianBlur:sigma_min, Value: 0.03894046691345188
image_augmenter:GaussianBlur:sigma_offset, Value: 1.3348068045107708
image_augmenter:GaussianBlur:use_augmenter, Value: True
image_augmenter:GaussianNoise:use_augmenter, Value: False
image_augmenter:RandomAffine:rotate, Value: 334
image_augmenter:RandomAffine:scale_offset, Value: 0.37914615264461815
image_augmenter:RandomAffine:shear, Value: 1
image_augmenter:RandomAffine:translate_percent_offset, Value: 0.13915163483786533
image_augmenter:RandomAffine:use_augmenter, Value: True
image_augmenter:RandomCutout:p, Value: 0.9948220651231314
image_augmenter:RandomCutout:use_augmenter, Value: True
image_augmenter:GaussianBlur:use_augmenter, Value: False
image_augmenter:GaussianNoise:sigma_offset, Value: 0.43287833896879524
image_augmenter:GaussianNoise:use_augmenter, Value: True
image_augmenter:RandomAffine:use_augmenter, Value: False
image_augmenter:RandomCutout:use_augmenter, Value: False
image_augmenter:Resize:use_augmenter, Value: False
image_augmenter:ZeroPadAndCrop:percent, Value: 0.43053137159256827
image_augmenter:ZeroPadAndCrop:percent, Value: 0.4972822105141404
normalizer:__choice__, Value: 'ImageNormalizer'

Fitting the pipeline...
Expand Down Expand Up @@ -177,7 +171,7 @@ Image Classification
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 0 minutes 6.162 seconds)
**Total running time of the script:** ( 0 minutes 5.855 seconds)


.. _sphx_glr_download_examples_20_basics_example_image_classification.py:
Expand Down
34 changes: 18 additions & 16 deletions development/_sources/examples/20_basics/example_tabular_classification.rst.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ Search for an ensemble of machine learning algorithms
.. code-block:: none
<autoPyTorch.api.tabular_classification.TabularClassificationTask object at 0x7fe0faf6af40>
<autoPyTorch.api.tabular_classification.TabularClassificationTask object at 0x7ff1d130deb0>
Expand Down Expand Up @@ -162,7 +162,7 @@ Print the final ensemble performance

.. code-block:: none
<smac.runhistory.runhistory.RunHistory object at 0x7fe06e273730> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
<smac.runhistory.runhistory.RunHistory object at 0x7ff1d130d790> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 64
encoder:__choice__, Value: 'OneHotEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
Expand Down Expand Up @@ -194,7 +194,7 @@ Print the final ensemble performance
scaler:__choice__, Value: 'StandardScaler'
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0014061927795410156, budget=0), TrajEntry(train_perf=0.16374269005847952, incumbent_id=1, incumbent=Configuration:
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.001264810562133789, budget=0), TrajEntry(train_perf=0.16374269005847952, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 64
encoder:__choice__, Value: 'OneHotEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
Expand Down Expand Up @@ -226,7 +226,7 @@ Print the final ensemble performance
scaler:__choice__, Value: 'StandardScaler'
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=1, ta_time_used=3.995302677154541, wallclock_time=5.0308003425598145, budget=5.555555555555555), TrajEntry(train_perf=0.14035087719298245, incumbent_id=2, incumbent=Configuration:
, ta_runs=1, ta_time_used=3.4787635803222656, wallclock_time=4.510373592376709, budget=5.555555555555555), TrajEntry(train_perf=0.14035087719298245, incumbent_id=2, incumbent=Configuration:
data_loader:batch_size, Value: 266
encoder:__choice__, Value: 'OneHotEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
Expand Down Expand Up @@ -270,28 +270,30 @@ Print the final ensemble performance
trainer:MixUpTrainer:alpha, Value: 0.3593787058162444
trainer:MixUpTrainer:weighted_loss, Value: False
trainer:__choice__, Value: 'MixUpTrainer'
, ta_runs=16, ta_time_used=140.4636766910553, wallclock_time=196.30338406562805, budget=50.0)]
{'accuracy': 0.8728323699421965}
, ta_runs=16, ta_time_used=129.56639432907104, wallclock_time=178.3990023136139, budget=50.0)]
{'accuracy': 0.8786127167630058}
| | Preprocessing | Estimator | Weight |
|---:|:------------------------------------------------------------------|:-------------------------------------------------------------------|---------:|
| 0 | None | CBLearner | 0.4 |
| 1 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.1 |
| 2 | SimpleImputer,OneHotEncoder,MinMaxScaler,NoFeaturePreprocessing | embedding,ShapedResNetBackbone,FullyConnectedHead,nn.Sequential | 0.08 |
| 3 | SimpleImputer,OneHotEncoder,NoScaler,KernelPCA | no embedding,ShapedResNetBackbone,FullyConnectedHead,nn.Sequential | 0.08 |
| 4 | None | RFLearner | 0.08 |
| 5 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.08 |
| 6 | None | ETLearner | 0.06 |
| 0 | SimpleImputer,NoEncoder,MinMaxScaler,PolynomialFeatures | no embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.22 |
| 1 | SimpleImputer,OneHotEncoder,NoScaler,KernelPCA | no embedding,ShapedResNetBackbone,FullyConnectedHead,nn.Sequential | 0.14 |
| 2 | None | CBLearner | 0.14 |
| 3 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.14 |
| 4 | SimpleImputer,OneHotEncoder,MinMaxScaler,NoFeaturePreprocessing | embedding,ShapedResNetBackbone,FullyConnectedHead,nn.Sequential | 0.08 |
| 5 | None | ETLearner | 0.06 |
| 6 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
| 7 | SimpleImputer,OneHotEncoder,MinMaxScaler,PowerTransformer | embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
| 8 | None | SVMLearner | 0.04 |
| 9 | None | KNNLearner | 0.04 |
| 8 | None | RFLearner | 0.04 |
| 9 | None | SVMLearner | 0.04 |
| 10 | None | KNNLearner | 0.02 |
| 11 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 5 minutes 22.454 seconds)
**Total running time of the script:** ( 5 minutes 30.492 seconds)


.. _sphx_glr_download_examples_20_basics_example_tabular_classification.py:
Expand Down
10 changes: 5 additions & 5 deletions development/_sources/examples/20_basics/example_tabular_regression.rst.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ Search for an ensemble of machine learning algorithms
.. code-block:: none
<autoPyTorch.api.tabular_regression.TabularRegressionTask object at 0x7fe106ad64f0>
<autoPyTorch.api.tabular_regression.TabularRegressionTask object at 0x7ff273a71a90>
Expand Down Expand Up @@ -157,7 +157,7 @@ Print the final ensemble performance

.. code-block:: none
<smac.runhistory.runhistory.RunHistory object at 0x7fe110cfc7f0> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
<smac.runhistory.runhistory.RunHistory object at 0x7ff273a71e80> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 64
encoder:__choice__, Value: 'OneHotEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
Expand Down Expand Up @@ -188,7 +188,7 @@ Print the final ensemble performance
optimizer:__choice__, Value: 'AdamOptimizer'
scaler:__choice__, Value: 'StandardScaler'
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0013153553009033203, budget=0), TrajEntry(train_perf=0.3195903909944855, incumbent_id=1, incumbent=Configuration:
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.001180887222290039, budget=0), TrajEntry(train_perf=0.3195903909944855, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 64
encoder:__choice__, Value: 'OneHotEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
Expand Down Expand Up @@ -219,7 +219,7 @@ Print the final ensemble performance
optimizer:__choice__, Value: 'AdamOptimizer'
scaler:__choice__, Value: 'StandardScaler'
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=1, ta_time_used=2.4860830307006836, wallclock_time=3.518846035003662, budget=5.555555555555555)]
, ta_runs=1, ta_time_used=2.12186336517334, wallclock_time=3.1505396366119385, budget=5.555555555555555)]
{'r2': 0.943135905978891}
| | Preprocessing | Estimator | Weight |
|---:|:------------------------------------------------------------------|:----------------------------------------------------------------|---------:|
Expand All @@ -235,7 +235,7 @@ Print the final ensemble performance
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 5 minutes 41.542 seconds)
**Total running time of the script:** ( 5 minutes 33.018 seconds)


.. _sphx_glr_download_examples_20_basics_example_tabular_regression.py:
Expand Down
8 changes: 4 additions & 4 deletions development/_sources/examples/20_basics/sg_execution_times.rst.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@

Computation times
=================
**11:10.158** total execution time for **examples_20_basics** files:
**11:09.365** total execution time for **examples_20_basics** files:

+--------------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_examples_20_basics_example_tabular_regression.py` (``example_tabular_regression.py``) | 05:41.542 | 0.0 MB |
| :ref:`sphx_glr_examples_20_basics_example_tabular_regression.py` (``example_tabular_regression.py``) | 05:33.018 | 0.0 MB |
+--------------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_examples_20_basics_example_tabular_classification.py` (``example_tabular_classification.py``) | 05:22.454 | 0.0 MB |
| :ref:`sphx_glr_examples_20_basics_example_tabular_classification.py` (``example_tabular_classification.py``) | 05:30.492 | 0.0 MB |
+--------------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_examples_20_basics_example_image_classification.py` (``example_image_classification.py``) | 00:06.162 | 0.0 MB |
| :ref:`sphx_glr_examples_20_basics_example_image_classification.py` (``example_image_classification.py``) | 00:05.855 | 0.0 MB |
+--------------------------------------------------------------------------------------------------------------+-----------+--------+
14 changes: 7 additions & 7 deletions ...opment/_sources/examples/40_advanced/example_custom_configuration_space.rst.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ the search. Currently, there are two changes that can be made to the space:-

.. code-block:: none
<smac.runhistory.runhistory.RunHistory object at 0x7fe06d866f10> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
<smac.runhistory.runhistory.RunHistory object at 0x7ff1cfc528e0> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 32
encoder:__choice__, Value: 'OneHotEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
Expand Down Expand Up @@ -80,7 +80,7 @@ the search. Currently, there are two changes that can be made to the space:-
scaler:__choice__, Value: 'StandardScaler'
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0013649463653564453, budget=0), TrajEntry(train_perf=0.15204678362573099, incumbent_id=1, incumbent=Configuration:
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.001199960708618164, budget=0), TrajEntry(train_perf=0.15204678362573099, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 32
encoder:__choice__, Value: 'OneHotEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
Expand Down Expand Up @@ -114,7 +114,7 @@ the search. Currently, there are two changes that can be made to the space:-
scaler:__choice__, Value: 'StandardScaler'
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=1, ta_time_used=4.133220195770264, wallclock_time=5.185335159301758, budget=5.555555555555555)]
, ta_runs=1, ta_time_used=3.7214386463165283, wallclock_time=4.764085292816162, budget=5.555555555555555)]
{'accuracy': 0.8670520231213873}
| | Preprocessing | Estimator | Weight |
|---:|:------------------------------------------------------------------|:----------------------------------------------------------|---------:|
Expand All @@ -128,7 +128,7 @@ the search. Currently, there are two changes that can be made to the space:-
| 7 | None | ETLearner | 0.08 |
| 8 | None | SVMLearner | 0.06 |
| 9 | SimpleImputer,OneHotEncoder,StandardScaler,KitchenSink | embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
<smac.runhistory.runhistory.RunHistory object at 0x7fe06d995f10> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
<smac.runhistory.runhistory.RunHistory object at 0x7ff1d0475070> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 32
encoder:__choice__, Value: 'NoEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
Expand Down Expand Up @@ -160,7 +160,7 @@ the search. Currently, there are two changes that can be made to the space:-
scaler:__choice__, Value: 'StandardScaler'
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0012934207916259766, budget=0), TrajEntry(train_perf=0.16959064327485385, incumbent_id=1, incumbent=Configuration:
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.001115560531616211, budget=0), TrajEntry(train_perf=0.16959064327485385, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 32
encoder:__choice__, Value: 'NoEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
Expand Down Expand Up @@ -192,7 +192,7 @@ the search. Currently, there are two changes that can be made to the space:-
scaler:__choice__, Value: 'StandardScaler'
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=1, ta_time_used=2.6655709743499756, wallclock_time=3.714816093444824, budget=5.555555555555555)]
, ta_runs=1, ta_time_used=2.4923224449157715, wallclock_time=3.534742832183838, budget=5.555555555555555)]
{'accuracy': 0.861271676300578}
| | Preprocessing | Estimator | Weight |
|---:|:--------------------------------------------------------------|:-------------------------------------------------------------------|---------:|
Expand Down Expand Up @@ -337,7 +337,7 @@ the search. Currently, there are two changes that can be made to the space:-
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 5 minutes 43.500 seconds)
**Total running time of the script:** ( 5 minutes 36.129 seconds)


.. _sphx_glr_download_examples_40_advanced_example_custom_configuration_space.py:
Expand Down

0 comments on commit 8f70383

Please sign in to comment.