Skip to content
A library containing both highly optimized building blocks and an execution engine for data pre-processing in deep learning applications
C++ Python Cuda C CMake Shell
Branch: master
Clone or download
jantonguirao ImageDecoder libtiff implementation (types.ANY_DATA, YCbCr, ImageDims…
… to TensorShape) (#1280)

Signed-off-by: Joaquin Anton <janton@nvidia.com>
Latest commit 0ea0e31 Sep 21, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Add JIRA Task placeholder in PR template (#1090) Jul 22, 2019
cmake ImageDecoder libtiff implementation (#1264) Sep 20, 2019
conda Conda Recipe for DALI (#1156) Aug 29, 2019
dali ImageDecoder libtiff implementation (types.ANY_DATA, YCbCr, ImageDims… Sep 21, 2019
dali_tf_plugin Improve logic in DALI TF plugin installation (support conda installat… Sep 20, 2019
docker ImageDecoder libtiff implementation (#1264) Sep 20, 2019
docs ImageDecoder libtiff implementation (#1264) Sep 20, 2019
include/dali/core ImageDecoder libtiff implementation (types.ANY_DATA, YCbCr, ImageDims… Sep 21, 2019
platforms DALI for aarch64-QNX platform (#522) Aug 29, 2019
qa Fix DALI_Extra clone (#1276) Sep 19, 2019
third_party Expose tensor as dlpack (#1154) Aug 26, 2019
tools ImageDecoder libtiff implementation (#1264) Sep 20, 2019
.clang-format Remove unnecessary copies by using const ref or move (#655) May 30, 2019
.gitignore DALI for aarch64-linux platform (#856) May 22, 2019
.gitmodules Expose tensor as dlpack (#1154) Aug 26, 2019
Acknowledgements.txt ImageDecoder libtiff implementation (#1264) Sep 20, 2019
CMakeLists.txt ImageDecoder libtiff implementation (#1264) Sep 20, 2019
CONTRIBUTING.md Add a STYLE_GUIDE for DALI, adjust Kernel example (#1167) Aug 19, 2019
COPYRIGHT Initial code drop Jun 18, 2018
DALI_EXTRA_VERSION Improve heuristic for variable frame rate detection (#1242) Sep 13, 2019
Docker_run_cuda Add build script and runner docker file (#236) Jan 28, 2019
Dockerfile ImageDecoder libtiff implementation (#1264) Sep 20, 2019
Dockerfile.build.aarch64-linux ImageDecoder libtiff implementation (#1264) Sep 20, 2019
Dockerfile.build.aarch64-qnx ImageDecoder libtiff implementation (#1264) Sep 20, 2019
Dockerfile.deps ImageDecoder libtiff implementation (#1264) Sep 20, 2019
Doxyfile CPU warp, with tests. (#1159) Aug 29, 2019
LICENSE Initial code drop Jun 18, 2018
README.rst Simplified README with links to NVIDIA docs (#927) Jun 27, 2019
STYLE_GUIDE.md Add a STYLE_GUIDE for DALI, adjust Kernel example (#1167) Aug 19, 2019
VERSION Version bump to 0.15 (#1239) Sep 6, 2019

README.rst

License Documentation

NVIDIA DALI

Deep learning applications require complex, multi-stage pre-processing data pipelines. Such data pipelines involve compute-intensive operations that are carried out on the CPU. For example, tasks such as: load data from disk, decode, crop, random resize, color and spatial augmentations and format conversions, are mainly carried out on the CPUs, limiting the performance and scalability of training and inference.

In addition, the deep learning frameworks have multiple data pre-processing implementations, resulting in challenges such as portability of training and inference workflows, and code maintainability.

NVIDIA Data Loading Library (DALI) is a collection of highly optimized building blocks, and an execution engine, to accelerate the pre-processing of the input data for deep learning applications. DALI provides both the performance and the flexibility for accelerating different data pipelines as a single library. This single library can then be easily integrated into different deep learning training and inference applications.

Highlights

  • Full data pipeline--accelerated from reading the disk to getting ready for training and inference.
  • Flexibility through configurable graphs and custom operators.
  • Support for image classification and segmentation workloads.
  • Ease of integration through direct framework plugins and open source bindings.
  • Portable training workflows with multiple input formats: JPEG, PNG (fallback to CPU), TIFF (fallback to CPU), BMP (fallback to CPU), raw formats, LMDB, RecordIO, TFRecord.
  • Extensible for user-specific needs through open source license.

Table of Contents


Installing or building DALI

DALI is preinstalled in the NVIDIA GPU Cloud TensorFlow, PyTorch, and MXNet containers in versions 18.07 and later.

Latest DALI release as well as weekly and nightly developer builds can be installed using pip. Detailed instructions can be found in the documentation installation guide. Build instructions for DALI can be found in compilation guide.


Examples and tutorials

Introduction to DALI can be found at Getting started docs page.

The docs/examples directory contains a few examples (in the form of Jupyter notebooks) highlighting different features of DALI and how to use DALI to interface with deep learning frameworks.

Also note:

  • Documentation for the latest stable release is available here, and
  • Nightly version of the documentation that stays in sync with the master branch is available here.

Additional resources

  • GPU Technology Conference 2018; Fast data pipeline for deep learning training, T. Gale, S. Layton and P. Trędak: slides, recording.
  • GPU Technology Conference 2019; Fast AI data pre-preprocessing with DALI; Janusz Lisiecki, Michał Zientkiewicz: slides, recording.
  • GPU Technology Conference 2019; Integration of DALI with TensorRT on Xavier; Josh Park and Anurag Dixit: slides, recording.
  • Developer page.
  • Blog post.

Contributing to DALI

We welcome contributions to DALI. To contribute to DALI and make pull requests, follow the guidelines outlined in the Contributing document.

If you are looking for a task good for the start please check one from external contribution welcome label.

Reporting problems, asking questions

We appreciate feedback, questions or bug reports. When you need help with the code, follow the process outlined in the Stack Overflow https://stackoverflow.com/help/mcve document. Ensure that the posted examples are:

  • minimal: Use as little code as possible that still produces the same problem.
  • complete: Provide all parts needed to reproduce the problem. Check if you can strip external dependency and still show the problem. The less time we spend on reproducing the problems, the more time we can dedicate to the fixes.
  • verifiable: Test the code you are about to provide, to make sure that it reproduces the problem. Remove all other problems that are not related to your request.

Contributors

DALI is being built with major contributions from Trevor Gale, Przemek Tredak, Simon Layton, Andrei Ivanov, Serge Panev.

You can’t perform that action at this time.