Skip to content

@BenjaminBossan BenjaminBossan released this Mar 23, 2021

This one is a smaller release, but we have some bigger additions waiting for the next one.

First we added support for Sacred to help you better organize your experiments. The CLI helper now also works with non-skorch estimators, as long as they are sklearn compatible. Some issues related to learning rate scheduling have been solved.

A big topic this time was also working on performance. First of all, we added a performance section to the docs. Furthermore, we facilitated switching off callbacks completely if performance is absolutely critical. Finally, we improved the speed of some internals (history logging). In sum, that means that skorch should be much faster for small network architectures.

We are grateful to the contributors, new and recurring:

  • Fariz Rahman
  • Han Bao
  • Scott Sievert
  • supetronix
  • Timo Kaufmann
Assets 2
  • v0.9.0
  • 92ae54b
  • Compare
    Choose a tag to compare
    Search for a tag
  • v0.9.0
  • 92ae54b
  • Compare
    Choose a tag to compare
    Search for a tag

@BenjaminBossan BenjaminBossan released this Aug 30, 2020

This release of skorch contains a few minor improvements and some nice additions. As always, we fixed a few bugs and improved the documentation. Our learning rate scheduler now optionally logs learning rate changes to the history; moreover, it now allows the user to choose whether an update step should be made after each batch or each epoch.

If you always longed for a metric that would just use whatever is defined by your criterion, look no further than loss_scoring. Also, skorch now allows you to easily change the kind of nonlinearity to apply to the module's output when predict and predict_proba are called, by passing the predict_nonlinearity argument.

Besides these changes, we improved the customization potential of skorch. First of all, the criterion is now set to train or valid, depending on the phase -- this is useful if the criterion should act differently during training and validation. Next we made it easier to add custom modules, optimizers, and criteria to your neural net; this should facilitate implementing architectures like GANs. Consult the docs for more on this. Conveniently, net.save_params can now persist arbitrary attributes, including those custom modules.
As always, these improvements wouldn't have been possible without the community. Please keep asking questions, raising issues, and proposing new features. We are especially grateful to those community members, old and new, who contributed via PRs:

Aaron Berk
guybuk
kqf
Michał Słapek
Scott Sievert
Yann Dubois
Zhao Meng

Here is the full list of all changes:

Added

  • Added the event_name argument for LRScheduler for optional recording of LR changes inside net.history. NOTE: Supported only in Pytorch>=1.4
  • Make it easier to add custom modules or optimizers to a neural net class by automatically registering them where necessary and by making them available to set_params
  • Added the step_every argument for LRScheduler to set whether the scheduler step should be taken on every epoch or on every batch.
  • Added the scoring module with loss_scoring function, which computes the net's loss (using get_loss) on provided input data.
  • Added a parameter predict_nonlinearity to NeuralNet which allows users to control the nonlinearity to be applied to the module output when calling predict and predict_proba (#637, #661)
  • Added the possibility to save the criterion with save_params and with checkpoint callbacks
  • Added the possibility to save custom modules with save_params and with checkpoint callbacks

Changed

  • Removed support for schedulers with a batch_step() method in LRScheduler.
  • Raise FutureWarning in CVSplit when random_state is not used. Will raise an exception in a future (#620)
  • The behavior of method net.get_params changed to make it more consistent with sklearn: it will no longer return "learned" attributes like module_; therefore, functions like sklearn.base.clone, when called with a fitted net, will no longer return a fitted net but instead an uninitialized net; if you want a copy of a fitted net, use copy.deepcopy instead;net.get_params is used under the hood by many sklearn functions and classes, such as GridSearchCV, whose behavior may thus be affected by the change. (#521, #527)
  • Raise FutureWarning when using CyclicLR scheduler, because the default behavior has changed from taking a step every batch to taking a step every epoch. (#626)
  • Set train/validation on criterion if it's a PyTorch module (#621)
  • Don't pass y=None to NeuralNet.train_split to enable the direct use of split functions without positional y in their signatures. This is useful when working with unsupervised data (#605).
  • to_numpy is now able to unpack dicts and lists/tuples (#657, #658)
  • When using CrossEntropyLoss, softmax is now automatically applied to the output when calling predict or predict_proba

Fixed

  • Fixed a bug where CyclicLR scheduler would update during both training and validation rather than just during training.
  • Fixed a bug introduced by moving the optimizer.zero_grad() call outside of the train step function, making it incompatible with LBFGS and other optimizers that call the train step several times per batch (#636)
  • Fixed pickling of the ProgressBar callback (#656)
Assets 2
  • v0.8.0
  • 7a84568
  • Compare
    Choose a tag to compare
    Search for a tag
  • v0.8.0
  • 7a84568
  • Compare
    Choose a tag to compare
    Search for a tag

@BenjaminBossan BenjaminBossan released this Apr 12, 2020

This release contains improvements on the callback side of things. Thanks to new contributors, skorch now integrates with neptune through NeptuneLogger and Weights & Biases through WandbLogger. We also added PassthroughScoring, which automatically creates epoch level scores based on computed batch level scores.

If you want skorch not to meddle with moving modules and data to certain devices, you can now pass device=None and thus have full control. And if you would like to pass pandas DataFrames as input data but were unhappy with how skorch currently handles them, take a look at DataFrameTransformer. Moreover, we cleaned up duplicate code in the fit loop, which should make it easier for users to make their own changes to it. Finally, we improved skorch compatibility with sklearn 0.22 and added minor performance improvements.

As always, we're very thankful for everyone who opened issues and asked questions on diverse channels; all forms of feedback and questions are welcome. We're also very grateful for all contributors, some old but many new:

Alexander Kolb
Benjamin Ajayi-Obe
Boris Dayma
Jakub Czakon
Riccardo Di Maio
Thomas Fan
Yann Dubois

Here is a list of all the changes and their corresponding ticket numbers in detail:

Added

  • Added NeptuneLogger callback for logging experiment metadata to neptune.ai (#586)
  • Add DataFrameTransformer, an sklearn compatible transformer that helps working with pandas DataFrames by transforming the DataFrame into a representation that works well with neural networks (#507)
  • Added WandbLogger callback for logging to Weights & Biases (#607)
  • Added None option to device which leaves the device(s) unmodified (#600)
  • Add PassthroughScoring, a scoring callback that just calculates the average score of a metric determined at batch level and then writes it to the epoch level (#595)

Changed

  • When using caching in scoring callbacks, no longer uselessly iterate over the data; this can save time if iteration is slow (#552, #557)
  • Cleaned up duplicate code in the fit_loop (#564)

Fixed

  • Make skorch compatible with sklearn 0.22 (#571, #573, #575)
  • Fixed a bug that could occur when a new "settable" (via set_params) attribute was added to NeuralNet whose name starts the same as an existing attribute's name (#590)
Assets 2

@ottonemo ottonemo released this Nov 29, 2019

Version 0.7.0

Notable additions are TensorBoard support through a callback and several improvements to the NeuralNetClassifier and NeuralNetBinaryClassifier to make them more compatible with sklearn metrics and packages by adding support for class inference among other things. We are actively pursuing some bigger topics which did not fit in this release such as scoring caching improvements (#557), a DataFrameTransformer (#507) and improvements to the training loop layout (#564) which we hope to bring to the next release.

WARNING: In a future release, the behavior of method net.get_params will change to make it more consistent with sklearn: it will no longer return "learned" attributes like module_. Therefore, functions like sklearn.base.clone, when called with a fitted net, will no longer return a fitted net but instead an uninitialized net. If you want a copy of a fitted net, use copy.deepcopy instead. Note that net.get_params is used under the hood by many sklearn functions and classes, such as GridSearchCV, whose behavior may thus be affected by the change. (#521, #527)

We had an influx of new contributors and users whom we thank for their support by adding pull requests and filing issues! Most notably, thanks to the individual contributors that made this release possible:

  • Alexander Kolb
  • Janaki Sheth
  • Joshy Cyriac
  • Matthias Gazzari
  • Sergey Alexandrov
  • Thomas Fan
  • Zhao Meng

Here is a list of all the changes and their coresponding ticket numbers in detail:

Added

  • More careful check for wrong parameter names being passed to NeuralNet (#500)
  • More helpful error messages when trying to predict using an uninitialized model
  • Add TensorBoard callback for automatic logging to tensorboard
  • Make NeuralNetBinaryClassifier work with sklearn.calibration.CalibratedClassifierCV
  • Improve NeuralNetBinaryClassifier compatibility with certain sklearn metrics (#515)
  • NeuralNetBinaryClassifier automatically squeezes module output if necessary (#515)
  • NeuralNetClassifier now has a classes_ attribute after fit is called, which is inferred from y by default (#465, #486)
  • NeuralNet.load_params with a checkpoint now initializes when needed (#497)

Changed

  • Improve numerical stability when using NLLLoss in NeuralNetClassifer (#491)
  • Refactor code to make gradient accumulation easier to implement (#506)
  • NeuralNetBinaryClassifier.predict_proba now returns a 2-dim array; to access the "old" y_proba, take y_proba[:, 1] (#515)
  • net.history is now a property that accesses net.history_, which stores the History object (#527)
  • Remove deprecated skorch.callbacks.CyclicLR, use torch.optim.lr_scheduler.CyclicLR instead

Future Changes

  • WARNING: In a future release, the behavior of method net.get_params will change to make it more consistent with sklearn: it will no longer return "learned" attributes like module_. Therefore, functions like sklearn.base.clone, when called with a fitted net, will no longer return a fitted net but instead an uninitialized net. If you want a copy of a fitted net, use copy.deepcopy instead. Note that net.get_params is used under the hood by many sklearn functions and classes, such as GridSearchCV, whose behavior may thus be affected by the change. (#521, #527)

Fixed

  • Fixed a bug that caused LoadInitState not to work with TrainEndCheckpoint (#528)
  • Fixed NeuralNetBinaryClassifier wrongly squeezing the batch dimension when using batch_size = 1 (#558)
Assets 2

@ottonemo ottonemo released this Jun 19, 2019

[0.6.0] - 2019-07-19

This release introduces convenience features such as SliceDataset which makes using torch datasets (e.g. from torchvision) easier in combination with sklearn features such as GridSearchCV. There was also some work to make the transition from CUDA trained models to CPU smoother and learning rate schedulers were upgraded to use torch builtin functionality.

Here's the full list of changes:

Added

  • Adds FAQ entry regarding the initialization behavior of NeuralNet when passed instantiated models. (#409)
  • Added CUDA pickle test including an artifact that supports testing on CUDA-less CI machines
  • Adds train_batch_count and valid_batch_count to history in training loop. (#445)
  • Adds score method for NeuralNetClassifier, NeuralNetBinaryClassifier, and NeuralNetRegressor (#469)
  • Wrapper class for torch Datasets to make them work with some sklearn features (e.g. grid search). (#443)

Changed

  • Repository moved to https://github.com/skorch-dev/skorch/, please change your git remotes
  • Treat cuda dependent attributes as prefix to cover values set using set_params since
    previously "criterion_" would not match net.criterion__weight as set by
    net.set_params(criterion__weight=w)
  • skorch pickle format changed in order to improve CUDA compatibility, if you have pickled models, please re-pickle them to be able to load them in the future
  • net.criterion_ and its parameters are now moved to target device when using criteria that inherit from torch.nn.Module. Previously the user had to make sure that parameters such as class weight are on the compute device
  • skorch now assumes PyTorch >= 1.1.0. This mainly affects learning rate schedulers, whose inner workings have been changed with version 1.1.0. This update will also invalidate pickled skorch models after a change introduced in PyTorch optimizers.

Fixed

  • Include requirements in MANIFEST.in
  • Add criterion_ to NeuralNet.cuda_dependent_attributes_ to avoid issues with criterion
    weight tensors from, e.g., NLLLoss (#426)
  • TrainEndCheckpoint can be cloned by sklearn.base.clone. (#459)

Thanks to all the contributors:

  • Bram Vanroy
  • Damien Lancry
  • Ethan Rosenthal
  • Sergey Alexandrov
  • Thomas Fan
  • Zayd Hammoudeh
Assets 2
Dec 17, 2018
Satisfy pypi version linter

@ottonemo ottonemo released this Dec 13, 2018

Version 0.5.0

Added

  • Basic usage notebook now runs on Google Colab
  • Advanced usage notebook now runs on Google Colab
  • MNIST with scikit-learn and skorch now runs on Google Colab
  • Better user-facing messages when module or optimizer are re-initialized
  • Added an experimental API (net._register_virtual_param) to register "virtual"
    parameters on the network with custom setter functions. (#369)
  • Setting parameters lr, momentum, optimizer__lr, etc. no longer resets
    the optmizer. As of now you can do net.set_params(lr=0.03) or
    net.set_params(optimizer__param_group__0__momentum=0.86) without triggering
    a re-initialization of the optimizer (#369)
  • Support for scipy sparse CSR matrices as input (as, e.g., returned by sklearn's
    CountVectorizer); note that they are cast to dense matrices during batching
  • Helper functions to build command line interfaces with almost no
    boilerplate, example that shows usage

Changed

  • Reduce overhead of BatchScoring when using train_loss_score or valid_loss_score by skipping superfluous inference step (#381)
  • The on_grad_computed callback function will yield an iterable for named_parameters only when it is used to reduce the run-time overhead of the call (#379)
  • Default fn_prefix in TrainEndCheckpoint is now train_end_ (#391)
  • Issues a warning when Checkpoints's monitor parameter is set to monitor and the history contains <monitor>_best. (#399)

Fixed

  • Re-initialize optimizer when set_params is called with lr argument (#372)
  • Copying a SliceDict now returns a SliceDict instead of a dict (#388)
  • Calling == on SliceDicts now works as expected when values are numpy arrays and torch tensors
Assets 2

@ottonemo ottonemo released this Oct 24, 2018

Organisational

From now on we will organize a change log and document every change directly. If
you are a contributor we encourage you to document your changes directly in the
change log when submitting a PR to reduce friction when preparing new releases.

Added

  • Support for PyTorch 0.4.1
  • There is no need to explicitly name callbacks anymore (names are assigned automatically, name conflicts are resolved).
  • You can now access the training data in the on_grad_computed event
  • There is a new image segmentation example
  • Easily create toy network instances for quick experiments using skorch.toy.make_classifier and friends
  • New ParamMapper callback to modify/freeze/unfreeze parameters at certain point in time during training:
>>> from sklearn.callbacks import Freezer, Unfreezer
>>> net = Net(module, callbacks=[Freezer('layer*.weight'), Unfreezer('layer*.weight', at=10)])
  • Refactored EpochScoring for easier sub-classing
  • Checkpoint callback now supports saving the optimizer, this avoids problems with stateful
    optimizers such as Adam or RMSprop (#360)
  • Added LoadInitState callback for easy continued training from checkpoints (#360)
  • NeuralNetwork.load_params now supports loading from Checkpoint instances
  • Added documentation for saving and loading highlighting the new features

Changed

  • The ProgressBar callback now determines the batches per epoch automatically by default (batches_per_epoch=auto)
  • The on_grad_computed event now has access to the current training data batch

Deprecated

  • Deprecated filtered_optimizer in favor of Freezer callback (#346)
  • NeuralNet.load_params and NeuralNet.save_params deprecate f parameter for the sake
    of f_optimizer, f_params and f_history (#360)

Removed

  • skorch.net.NeuralNetClassifier and skorch.net.NeuralNetRegressor are removed.
    Use from skorch import NeuralNetClassifier or skorch.NeuralNetClassifier instead.

Fixed

  • uses_placeholder_y should not require existence of y field (#311)
  • LR scheduler creates batch_idx on first run (#314)
  • Use OrderedDict for callbacks to fix python 3.5 compatibility issues (#331)
  • Make to_tensor work correctly with PackedSequence (#335)
  • Rewrite History to not use any recursion to avoid memory leaks during exceptions (#312)
  • Use flaky in some neural network tests to hide platform differences
  • Fixes ReduceLROnPlateau when mode == max (#363)
  • Fix disconnected weights between net and optimizer after copying the net with copy.deepcopy (#318)
  • Fix a bug that interfered with loading CUDA models when the model was a CUDA tensor but
    the net was configured to use the CPU (#354, #358)

Contributors

Again we'd like to thank all the contributors for their awesome work.
Thank you

  • Andrew Spott
  • Dave Hirschfeld
  • Scott Sievert
  • Sergey Alexandrov
  • Thomas Fan
Assets 2

@ottonemo ottonemo released this Jul 26, 2018

Features

API changes

  • train_step is now split in train_step and train_step_single in order to support LBFGS, where train_step_single takes the role of your typical training inner-loop when writing PyTorch models
  • device parameter on skorch.dataset.Dataset is now deprecated
  • Checkpoint parameter target is deprecated in favor of f_params

Contributors

A big thanks to our contributors who helped making this release possible:

  • Andrew Spott
  • Scott Sievert
  • Sergey Alexandrov
  • Thomas Fan
  • Tomasz Pietruszka
Assets 2

@ottonemo ottonemo released this May 4, 2018

Features

  • PyTorch 0.4 support
  • Add GradNormClipping callback
  • Add generic learning rate scheduler callback
  • Add CyclicLR learning rate scheduler
  • Add WarmRestartLR learning rate scheduler
  • Scoring callbacks now re-use predictions, accelerating training
  • fit() and inference methods (e.g., predict()) now support torch.util.data.Dataset as input as long as (X, y) pairs are returned
  • forward and forward_iter now allow you to specify on which device to store intermediate predictions
  • Support for setting optimizer param groups using wildcards (e.g., {'layer*.bias': {'lr': 0}})
  • Computed gradients can now be processed by callbacks using on_grad_computed
  • Support for fit_params parameter which gets passed directly to the module
  • Add skorch.helper.SliceDict so that you can use dict as X with sklearn's GridSearchCV, etc.
  • Add Dockerfile

API changes

  • Deprecated use_cuda parameter in favor of device parameter
  • skorch.utils.to_var is gone in favor of skorch.utils.to_tensor
  • training_step and validation_step now return a dict with the loss and the module's prediction
  • predict and predict_proba now handle multiple outputs by assuming the first output to be the prediction
  • NeuralNetClassifier now only takes log of prediction if the criterion is set to NLLLoss

Examples

  • RNN sentiment classification

Communication

Contributors

A big thanks to our contributors who helped making this release possible:

  • Felipe Ribeiro
  • Grzegorz Rygielski
  • Juri Paern
  • Thomas Fan
Assets 2