Repository navigation
v20.0.0-rc.1
Pre-releaseHighlights
First release candidate of NCore 20: beta.6 with two fixes for reading remote archives (see below). The APIs are frozen for 20.0, and only fixes are planned until the final release. Install with pip install --pre nvidia-ncore or nvidia-ncore==20.0.0rc1.
Fixed since beta.6
- Remote archives opened via multi-storage-client (
msc://) requiremulti-storage-client>=1.4.0(not a declared dependency), and raise aRuntimeErrornaming the installed and required versions otherwise. NCore 20 reads remote archives via fsspec byte-range requests, which multi-storage-client before 1.3.0 ignores: opening an archive downloaded all of it (e.g. read timeouts on the first dataset read). 1.3.0 needs an additional metadata request for the suffix range that loads the index, 1.4.0 reads it with a single request. - Records of remote archives are read as
byteswhatever buffer type the filesystem returns for range requests (e.g. multi-storage-client with its Rust client enabled returns a buffer object that isn'tbytes). Records held in memory with such a buffer type were taken for record offsets: reading the consolidated metadata of an archive opened asynchronously raisedTypeError: read() takes 3 positional arguments but N were given.bytesfrom the filesystem are passed through without a copy, other buffers are copied once.
NCore 20 keeps the data format: data written by NCore 19.x is read by NCore 20, and data written by NCore 20 in zarr format v2 (the default) is read by NCore 19.x. Major features compared to NCore 19.x:
- zarr-python 3 support (zarr-python 2 remains supported), and component stores in zarr format v3 (opt-in).
- Faster reads: 1.7x to 5.5x faster frame reads of local
.itararchives via non-blocking and range-read APIs.- Remote storage / faster opening:
.itarstores are opened with a single request, with exact range reads for synchronous reads. - Asynchronous APIs:
*_asyncreads of all component readers and of the compat API, and asynchronous opening of sequences (SequenceLoaderV4.open_async), next to the new synchronousSequenceLoaderV4.open. - Storage write / read APIs: reads into preallocated buffers, arrays written by selection or from chunks encoded on worker threads, and hierarchies opened from consolidated metadata when stored.
- Remote storage / faster opening:
- Public packages:
ncore.data,ncore.data.v4,ncore.data.stores,ncore.geometryandncore.sensorsnow export every type their APIs take and return;ncore.implis fully internal. - Camera models: ray generation and projections return valid flags, the OpenCV pinhole projection validity is derived from the radial distortion, and FTheta cameras support new tangential distortion coefficients.
- Faster motion compensation: ~1.5x faster exact method (default), and an interpolated method (
MotionCompensator.Parameters.Method.INTERPOLATED).
For upgrading from NCore 19.x, see the Upgrading from NCore 19.x guide.
Thanks to Dezhen Lu (@0z5a), Nick Lucas (@nlucas-nvidia) and Sanchit Garg (@sanchitgarg) for their contributions to this release!
Changes since v19.8.0
➕ Added
- (data_converter) default zarr format 2 (0d9137a) - @janickm
- BREAKING (data) sync loader opening, default motion compensators and V4 component API consistency (68391b9) - @janickm
- BREAKING (sensors) factory-only model construction, uniform validity returns and camera model conventions (fbf9400) - @janickm
- BREAKING (data) uniform write checks, release_index_tail without consolidated, and Group / Array API consistency (1347a09) - @janickm
- BREAKING keyword-only optional parameters and public types of the storage, loader and sensor APIs (9a5a639) - @janickm
- (data) export BloscCompression and EncodedArray from ncore.data.stores (bb30302) - @janickm
- (data) open hierarchies from consolidated metadata if stored (f224965) - @janickm
- (data) create arrays from chunks encoded on other threads (8fc4f14) - @janickm
- (data) create fill-valued arrays and write selections of them (d4114e8) - @janickm
- (data) read arrays into preallocated output buffers (9dc561a) - @janickm
- BREAKING (sensors) derive the OpenCV pinhole projection validity from its radial distortion (ffad25e) - @janickm
- (sensors) add FTheta tangential distortion coefficients (b7a158d) - @janickm
- public storage and geometry packages, ray validity helpers (bcd3e49) - @janickm
- BREAKING (sensors) return validity flags from structured lidar ray generation (05e6e86) - @janickm
- BREAKING (sensors) return validity flags from camera ray generation (e1b44a8) - @janickm
- (data) open sequences asynchronously (dfe3729) - @janickm
- (data) asynchronous reads of SequenceLoaderV4 (d49ca46) - @janickm
- (data) asynchronous compat protocols (d97b7ce) - @janickm
- (data) range reads of remote archives for synchronous reads (1880cb6) - @janickm
- (data) asynchronous component reader API (f67caa3) - @janickm
- BREAKING (data) zarr format v3 stores (c5fce26) - @janickm
- BREAKING (data) support zarr-python 3 (2cee604) - @janickm
- BREAKING (data) typed node API for component groups (ff2eeb0) - @janickm
🪲 Fixed
- (data) read remote archive records as bytes for any buffer type of the filesystem (de0d61e) - @janickm
- (data) reject multi-storage-client < 1.4.0 for remote archives (7f5f200) - @janickm
- (geometry) exclude samples outside the interval from HalfClosedInterval.cover_range (0d1d0d9) - @0z5a
- BREAKING (sensors,geometry) raise ValueError instead of AssertionError in user-facing validation (0a1c2a7) - @janickm
- BREAKING (data) raise ValueError instead of AssertionError in V4 writer validation (bc1b114) - @janickm
- (sensors) keep projection valid indices 1-D for single valid points (b47c5a0) - @nlucas-nvidia
- (sensors) apply the converged Newton step in the bracketed fisheye inverse (16d2f65) - @janickm
- (sensors) bracket the OpenCV fisheye inverse to the monotonic branch (71a0602) - @janickm
- (data) read single poses with their stored dtype (1de0192) - @janickm
- type narrowing in ncore_export_ply and the argoverse2 converter test (348262a) - @janickm
- (kitti) keep lidar point timestamps integer (5bf8f9f) - @janickm
⚡ Performance
- (geometry) add an interpolated motion compensation method (84d965b) - @janickm
- (sensors) skip zero tangential and thin-prism terms in OpenCV pinhole (2419967) - @janickm
- (sensors) skip identity linear term in FTheta camera model (1c0c1bf) - @janickm
- (data) release index tail reads of opened archives (10c2609) - @janickm
- (data) open remote indexed tar archives with a single request (f57784c) - @janickm
- (data) direct chunk reads with zarr-python 3 (979882a) - @janickm
- (data) node cache for component readers (0d6c43d) - @janickm
- (data) cache decoded label types and schemas in CameraLabelDescriptor.from_dict (1813046) - @janickm
- (data) load label and point-cloud reader data on first use (fe6eaf2) - @janickm
🔄 Changed
- (tools) import the public packages instead of ncore.impl (7007cc4) - @janickm
- BREAKING (geometry) typed pose interpolator and a result type for motion decompensation (4efd97b) - @janickm
- BREAKING (data) move is_consolidated_metadata_key to Store (5175459) - @janickm
- (sensors) construct sensor return types through their defining class (e7c26c3) - @janickm
- (data) restructure IndexedTarArchive (fd9a4f0) - @janickm
- BREAKING (data) implement IndexedTarArchive independently of zarr-python (cd3078f) - @janickm
📚 Documentation
- upgrading guide for NCore 20, ray validity, motion compensation and storage API (ec6937b) - @janickm
- remove retired skill provenance (e0e7609) - @sanchitgarg
⚙️ CI
- release notes tool in Python, commit check in build-and-test (069aee8) - @janickm
- separate the changes since the previous pre-release in release notes (3c48a54) - @janickm
- publish the documentation built by build-and-test (ed2d016) - @janickm
- refresh the Bazel disk cache on every push (09a3665) - @janickm
- remove merge queue triggers (3f52089) - @janickm
- write release notes into the release commit body (ccc650a) - @janickm
- release from main and release branches when the version changes (e8aba41) - @janickm
🏗️ Build
- (wheel) declare numcodecs and fsspec, pin zarr per Python version (b596c6e) - @janickm
- pinned cocogitto via rules_multitool (e5ca2c9) - @janickm
- don't compress the repacked patched torch wheel (dc91c12) - @janickm
- add python 3.12 / 3.13 test environments (5b5fc02) - @janickm
🎨 Style
🔧 Chore
- map Dezhen Lu (0z5a) to his GitHub username in release notes (0f40da0) - @janickm
- map Nick Lucas to his GitHub username in release notes (0a4d576) - @janickm
🧪 Tests
- (sensors) check camera ray valid flags in the unprojection tests (22db77d) - @janickm
- (sensors) share the lidar model between tests of a parameterization (c490c63) - @janickm
- run tests with a single OpenMP thread (2d4263f) - @janickm
- make tests independent of scipy / numpy versions (f2d59b0) - @janickm
New since v20.0.0-beta.6
🪲 Fixed
- (data) read remote archive records as bytes for any buffer type of the filesystem (de0d61e) - @janickm
- (data) reject multi-storage-client < 1.4.0 for remote archives (7f5f200) - @janickm
Changes since v20.0.0-beta.6: v20.0.0-beta.6...v20.0.0-rc.1
Full diff: v19.8.0...v20.0.0-rc.1