Skip to content

Latest commit

 

History

History
470 lines (427 loc) · 30 KB

CHANGELOG.md

File metadata and controls

470 lines (427 loc) · 30 KB
Raw

0.6.0-alpha-1 (16/01/2023) New inference api with multiple inputs support

API changes:

  • Changed InferenceModel interface:
    • Added type parameter representing inference result. #515
    • Replaced classification function with general purpose prediction methods for single and multiple inputs. Extensions org.jetbrains.kotlinx.dl.impl.inference.imagerecognition.PredictionKt.predictLabel and org.jetbrains.kotlinx.dl.impl.inference.imagerecognition.PredictionKt.predictProbabilities were added for classification. #515
    • Introduced InferenceModel#resultConverter to process inference results. #515
    • Method reshape was removed, in favor of prediction methods receiving input as FloatData containing input shape information. #513
    • Parameters were removed from the copy function. #503
  • Replaced SavedModel#predict with org.jetbrains.kotlinx.dl.dataset.InferenceModelExtensionsKt.predict extension. #515
  • Added predictionFunction parameter to the org.jetbrains.kotlinx.dl.dataset.InferenceModelExtensionsKt.predict and org.jetbrains.kotlinx.dl.dataset.InferenceModelExtensionsKt.evaluate extension functions. #515
  • Shape information was added to the dataset classes. #513
    • Changed return type of the Dataset.getX function to FloatData.
    • Added TensorShape parameter to DataBatch and OnHeapDataset constructors.
    • Removed OnHeapDataset.Companion#createTrainAndTestDatasets, OnHeapDataset.Companion#create(String, String, int, Function1<String,float[][]>, Function2<String,Integer,float[]>), OnHeapDataset.Companion#create(Function0<float[][]>, Function0<float[]>) functions.
  • No-top models were moved to the separate model types (TFModels.CVnoTop and ONNXModels.CVnoTop). #511
  • Changed high-level model classes so they do not implement InferenceModel. #509
  • Converted SavingFormat to a class and added isKerasFullyCompatible parameter to SavingFormat.JsonConfigCustomVariables. #501
  • Add basic multiple input support to OnnxInferenceModel. #417

0.5.2 (22/05/2023) Bugfix + upgrade ONNX and Kotlin versions

API changes:

  • Made DataBatch constructor public to allow implementing custom datasets #533
  • Made properties public #538

Bug fixes:

  • Upgraded the ONNX Runtime version from 0.12 to 0.14 #551
  • Upgraded the Kotlin version from 1.7.20 to 1.8.21 #547
  • Created all necessary parent directories when saving models #534
  • Fixed dimension order in the image shape#542
  • Fixed transpose shape test #554
  • Ported fix #531 to the 0.5.2 #531

0.5.1 (14/01/2023) Bugfix update

Bug fixes:

  • Fixed preprocessing in SSDObjectDetectionModel, MultiPoseDetectionModel, removed explicit shapes from Fan2d106 and UltraFace models #496
  • Skip image copy creation in Resize operation for the output of the same size as input #502
  • Fixed SGD secondary constructor ignoring clipGradient parameter #504
  • Fixed memory leak ensuring used TensorFlow tensors get closed #507
  • Fix HardShrink activation #505
  • Fixed a problem with downloading models for Android #516
  • Minor documentation fixes

0.5.0 (14/12/2022) Inference on Android with ONNX Runtime

Features:

  • Added Android inference support
    • Built Android artifacts for "impl", "onnx" and "visualization" modules #422
    • Added Android-specific models to the model zoo
      • Classification #438:
        • EfficientNet4Lite
        • MobilenetV1
      • Object Detection:
        • SSDMobileNetV1 #440
        • EfficientDetLite0 #443
      • Pose Detection #442:
        • MoveNetSinglePoseLighting
        • MoveNetSinglePoseThunder
      • Face Detection #461:
        • UltraFace320
        • UltraFace640
      • Face Alignment #441:
        • Fan2d106
    • Implemented preprocessing operations working on Android Bitmap #416 #478:
      • Resize
      • Rotate
      • Crop
      • ConvertToFloatArray
    • Added utility functions to convert ImageProxy to Bitmap #458
    • Added NNAPI execution provider #420
    • Added api to create OnnxInferenceModel from the ByteArray representation #415
    • Introduced a gradle task to download model hub models before the build #444
    • Added utility functions to draw detection results on Android Canvas #450
  • Implemented new preprocessing API #425
    • Introduced an Operation interface to represent a preprocessing operation for any input and output
    • Added PreprocessingPipeline class to combine operations together in a type-safe manner
    • Re-implemented old operations with the new API
    • Added convenience functions such as pipeline to start a new preprocessing pipeline, call to invoke operations defined elsewhere, onResult to access intermediate preprocessing results
    • Converted ModelType#preprocessInput function to Operation #429
    • Converted common preprocessing functions for models trained on ImageNet to Operation #429
  • Added new ONNX features
    • Added execution providers support (CPU, CUDA, NNAPI) and convenient extensions for inference with them #386
    • Introduced OnnxInferenceModel#predictRaw function which allows custom OrtSession.Result processing and extension functions to extract common data types from the result #465
    • Added validation of input shape #385
  • Added Imagenet enum to represent different Imagenet dataset labels and added support for zero indexed COCO labels #438 #446
  • Implemented unified summary printing for Tensorflow and ONNX models #368
  • Added FlatShape interface to allow manipulating the detected shapes in a unified way #480
  • Introduced DataLoader interface for loading and preprocessing data for dataset implementations #424
  • Improved swing visualization utilities #379 #388
  • Simplified Layer interface to leave only build function to be implemented and remove explicit output shape computation #408

Breaking changes:

  • Refactored module structure and packages #412 #469
    • Extracted "tensorflow" module for learning and inference with Tensorflow backend
    • Extracted "impl" module for implementation classes and utilities
    • Moved preprocessing operation implementations to the "impl" module
    • Removed dependency of "api" module on the "dataset" module
    • Changed packages for "api", "impl", "dataset" and "onnx" so that they match the corresponding module name
  • Preprocessing classes such as Preprocessing, ImagePreprocessing, ImagePreprocessor, ImageSaver, ImageShape, TensorPreprocessing, Preprocessor got removed in favor of the new preprocessing API #425
  • Removed Sharpen preprocessor since the ModelType#preprocessor field was introduced, which can be used in the preprocessing pipeline using the call function #429

Bugfixes:

  • Fix loading of jpeg files not supported by standard java ImageIO #384
  • Updated ONNX Runtime version to enable inference on M1 chips #361
  • Fixed channel ordering in for image recognition models #400
  • Avoid warnings from loadWeightsForFrozenLayers function for layers without parameters #382

New documentation and examples:

0.4.0 (01/06/2022) Pose Detection, EfficientDet for Object Detection and EfficientNet for Image Recognition

Features:

  • Added the PoseNet model family to the ONNX Model Hub. #269
    • MoveNetSinglePoseLighting
    • MoveNetMultiPoseLighting
    • MoveNetSinglePoseThunder
  • Added the EfficientDet model family to the ONNX Model Hub. #304
    • EfficientDetD0
    • EfficientDetD1
    • EfficientDetD2
    • EfficientDetD3
    • EfficientDetD4
    • EfficientDetD5
    • EfficientDetD6
  • Added SSD-Mobilenet-v1 model to the ONNX Model Hub. #296
  • Added EfficientNet model family to the ONNX Model Hub. #264
    • EfficientNetB0 ( + noTop model)
    • EfficientNetB1 ( + noTop model)
    • EfficientNetB2 ( + noTop model)
    • EfficientNetB3 ( + noTop model)
    • EfficientNetB4 ( + noTop model)
    • EfficientNetB5 ( + noTop model)
    • EfficientNetB6 ( + noTop model)
    • EfficientNetB7 ( + noTop model)
  • Added NoTop models to the TensorFlow Model Hub. #281
    • VGG'16
    • VGG'19
    • ResNet50
    • ResNet101
    • ResNet152
    • ResNet50V2
    • ResNet101V2
    • ResNet152V2
    • MobileNet
    • MobileNetV2
    • NasNetMobile
    • NasNetLarge
    • DenseNet121
    • DenseNet169
    • DenseNet201
    • Xception
    • Inception
  • Added new Dot layer and Conv1DTranspose, Conv2DTranspose, Conv3DTranspose layers. #144 #124
  • Added new activation functions: SparsemaxActivation and SoftShrinkActivation. #171 #170
  • Added new Padding, CenterCrop, Convert, Grayscale image preprocessors and Normalizing tensor preprocessor. #203 #201 #202 #204

Examples and tutorials:

API changes:

  • Introduced new DSL for creating Sequential and Functional models. #133
  • Added support for multiple Callbacks in fit(), evaluate(), predict() instead of compile(). #270
  • Added support for multiple metrics. #298
  • Added support for the model reset. #271
  • Replaced Long parameters with Integer ones in convolutional, average pool and max pool layers. #273
  • Moved loading section out of image preprocessing. #322
  • Remove obsolete CustomPreprocessor interface. #257
  • Supported exporting tensor data to BufferedImage #293

Internal API changes:

  • Introduced new abstraction for layer parameters -- KVariable. #324
  • Moved some of the Layer functionality to the new interfaces ParametrizedLayer and TrainableLayer. #217

Bug fixes:

  • Added support for correct loading of isTrainable status from Keras. #153
  • Generalized Reshape layer to higher dimensions. #249
  • Fixed incorrect bounding box coordinates in ObjectDetection and FaceDetection examples. #279
  • Fixed toString methods for layer classes. #301
  • Set all the optimizers to have useLocking = True #305
  • Fixed a bug with silently skipped layers in topological sort. #314
  • Fixed loadModelLayersFromConfiguration recursively calling itself. #319
  • Fixed GraphTrainableModel#internalPredict for multi-dimensional predictions. #327
  • Fixed Orthogonal initializer. #348
  • Fixed initialization of the variables in GraphTrainableModel. #355
  • Add model output type checks for OnnxInferenceModel. #356
  • Fixed IndexOutOfBoundsException in the Dot layer. #357
  • Fixed import and export issues:
    • Fixed layers import and export and added tests. #329 #341 #360
    • Fixed exporting Ones, Zeros, Orthogonal and ParametrizedTruncatedNormal initializers. #331
  • Updated log4j version.
  • Fixed the group of examples with ToyResNet.

0.3.0 (28/09/2021) ONNX for inference and transfer learning and ONNX Model Hub

Features:

Bugs:

API breaking changes:

Infrastructure:

  • Loaded the weights and JSON configurations of the newly added ModelHub models to S3 storage
  • Moved ImageDSL and Dataset API to the separate 'dataset' module
  • Added a new 'visualization' module with the basic support for painting on Swing and in Jupyter Notebook with lets-plot
  • Transformed the project from the single-module project to the multi-module project

Docs:

  • Created website with API Documentation from KDoc via Dokka
  • Added support for the multiple version API Documentation from KDoc via Dokka
  • Updated all existing tutorials
  • Updated the Readme.md
  • Updated the existing KDocs
  • Added a new tutorial about ONNX models usage
  • Added a new tutorial about Transfer Learning with ONNX ResNet no-top model and TensorFlow

Examples:

  • Added an example of SSDObjectDetectionModel usage and visualisation of the detected objects on the Swing panel
  • Added an example of Fan2D106 (face alignment) model and landmarks visualisation on the Swing panel
  • Added an example where the prepared ONNX model without top layers is fine-tuned via training of top layers implemented with TensorFlow-based layers
  • Added a lot of examples for the newly added to the ModelHub models (ONNX-based and TensorFlow-based)
  • Added an example with the model SoundNet trained on Free Spoken Digits Dataset to classify the audio
  • Updated 'visualization' examples with the new Batik and lets-plot support

Tests:

  • Added tests for ModelLoading
  • Added tests for InputLayer
  • Added tests for all newly added layers

0.2.0 (17/05/2021) Functional API, Model Zoo and Image Preprocessing DSL

Features:

  • Added support for Functional API
  • Added BatchNorm layer for inference
  • Added GlobalAveragePooling2D layer
  • Added 7 Merge layers (Add, Average, Concatenate, Maximum, Minimum, Multiply, Subtract)
  • Added Activation layer
  • Added ReLU layer
  • Added DepthwiseConv2D layer
  • Added SeparableConv2D layer
  • Added Reshape layer
  • Added Cropping2D layer
  • Added ZeroPadding2D layer
  • Added NoGradients interface to indicate layers whose weights cannot be updated during training due to the lack of gradients in TensorFlow
  • Added Model Zoo with the following models:
    • VGG'16
    • VGG'19
    • ResNet50
    • ResNet101
    • ResNet152
    • ResNet50V2
    • ResNet101V2
    • ResNet152V2
    • MobileNet
    • MobileNetV2
  • Added ImageNet related preprocessing for each of the ModelZoo supported models: available in ModelZoo object and as a sharpen stage in the image preprocessing DSL
  • Added model descriptions for models from ModelZoo (excluding MobileNet) designed with the Functional API in org.jetbrains.kotlinx.dl.api.core.model package
  • Added two implementations of the Dataset class: OnFlyImageDataset and OnHeapDataset
  • Added topological sort for layers as nodes in the DAG model representation
  • Added shuffle function support for both Dataset implementations
  • Added the Kotlin-idiomatic DSL for image preprocessing with the following operations:
    • Load
    • Crop
    • Resize
    • Rotate
    • Rescale
    • Sharpen
    • Save
  • Implemented label generation on the fly from the names of image folders
  • Implemented summary method for the Functional API
  • Added embedded datasets support (MNIST, FashionMNIST, Cifar'10, Cats & Dogs)

Bugs:

  • Fixed a bug with BGR and RGB preprocessing in examples
  • Fixed missed useBias field in convolutional layers

Internals improvements:

  • Refactored: both Sequential and Functional models now inherit the GraphTrainableModel class
  • Completed the Klaxon migration from 5.0.1 to 5.5
  • Removed useless labels and data transformations before sending to Tensor.create(...)

Infrastructure:

  • Loaded the weights and JSON configurations of ModelZoo models to S3 storage
  • Added a TeamCity build for the examples
  • Loaded embedded datasets to S3 storage
  • Removed dependencies from jcenter
  • Moved an artifact to the Maven Central Repository
  • Changed the groupId and artifactId
  • Reduced the size of the downloaded api artifact from 65 MB to 650 KB by cleaning up resources and migrating the model and datasets to the S3 storage

Docs:

  • Updated all the tutorials
  • Updated the Readme.md

Examples:

  • Renamed all the example's packages
  • Regrouped examples between packages
  • Added examples for training all ResNet models from scratch on the Cats & Dogs dataset
  • Tuned hyper-parameters in all examples with VGG-like architecture to achieve convergence
  • Added examples for the Image Preprocessing DSL
  • Added examples for all available ModelZoo models, including additional training on the subset of the Cats & Dogs dataset
  • Added ToyResNet examples (trained on the FashionMnist dataset)

Tests:

  • Converted all examples to integration tests by refactoring main functions

0.1.1 (01/02/2021)

Features:

Bugs:

Internals improvements:

  • Removed unnecessary copying of FloatArray to off-heap memory in internalPredict method

Docs:

  • Added "Release check list" for release managers
  • Updated Readme.md with new chapters about logging and fatJar issue
  • Minor updates in "Tutorials"

0.1.0 (09/12/2020)

Features:

  • Added @JvmStatic for companion methods

Examples:

  • Provided support for VGG'16 and VGG'19 weights and models in examples
  • Added links for loading all models and weights used in examples
  • Moved direct file paths to property file
  • Removed duplicated resources
  • Transfer Learning examples are merged and improved
  • Added description for all examples

Docs:

  • Minor updates in "Transfer Learning Tutorial"

0.0.14 (20/11/2020)

Features:

  • ReductionType support for loss function: SUM and SUM_OVER_BATCH_SIZE
  • Added new loss functions: LogCosh, BinaryCrossEntropy, SquaredHinge

Tests:

  • Added tests for all loss functions (in Eager Mode)
  • Added tests for new scenarios of Keras weights loading

Docs:

  • "Transfer Learning Tutorial" added
  • Code of conduct and ChangeLog documents added