v2.0.0-alpha.3
Pre-release
Pre-release
·
226 commits
to main
since this release
What's Changed
- better contributing docs build instructions by @ianhi in #1347
- add serialization and deserialization property tests for types in config file by @ebarylko in #1343
- New method Repository.manifest_files to get the snap manifests by @paraseba in #1353
- [DONT MERGE]: Icechunk 2.0 by @paraseba in #1154
- Bump the rust-dependencies group across 1 directory with 3 updates by @dependabot[bot] in #1354
- Satisfy clippy by @TomNicholas in #1361
- Support
gs://prefix in virtual chunk containers by @TomNicholas in #1363 - Add some missing steps for first-time rust contributors by @TomNicholas in #1362
- Ian/v1 ci main by @ianhi in #1359
- Fix typo by @paraseba in #1370
- Expose
RepositoryConfig.merge()method in python by @TomNicholas in #1372 - add version policy by @ianhi in #1346
- Ignore mypy complaint about
dask.__version__by @TomNicholas in #1376 - Allow support branch in release workflow by @TomNicholas in #1380
- fix version generation for actual release by @ianhi in #1381
- Improve release guide by @TomNicholas in #1378
- New HTTP Storage instance by @paraseba in #1379
- Add note about conda-forge to release guide by @TomNicholas in #1383
- Bump the rust-dependencies group with 3 updates by @dependabot[bot] in #1387
- Reduce GHA log volume by @paraseba in #1388
- Introduce
redirect_storageby @paraseba in #1386 - builds: fix RTD explicitly add patchelf + update macos runners by @ianhi in #1390
- Remove out-of date GCS retry warning by @dcherian in #1393
- fix errors in bug report by @ianhi in #1401
- use rustls-tls instead of native-tls -> openssl by @ianhi in #1402
- Speed up
ops_logby storing up to 100 changes per repo file by @paraseba in #1395 - Fix
Repository.total_chunks_storagefor V1 repos. by @paraseba in #1407 - Upgrade to Rust 1.91 by @paraseba in #1410
- Allow creation of V1 repos by @paraseba in #1414
- We can commit to IC1 repos now by @paraseba in #1415
- More write operations on V1 repos by @paraseba in #1417
Session.flushon V1 repos by @paraseba in #1418- GC and expiration for V1 repos by @paraseba in #1422
- docs: fix command for editable mode by @evaporei in #1426
- simplify upstream dependecies to avoid resolution errors by @ianhi in #1429
- fix typing errors by @ianhi in #1433
- A bunch of testing for IC1 repos by @paraseba in #1423
- Bump the rust-dependencies group with 5 updates by @dependabot[bot] in #1437
- Bump actions/checkout from 5 to 6 in the actions group by @dependabot[bot] in #1436
- Update FAQ with spec stability and implementations info by @TomNicholas in #1430
- repository: add
list_prefix to manifest_files() method by @evaporei in #1428 - Update dependencies by @paraseba in #1451
- Port requester pays from support branch by @paraseba in #1454
- mypy tests by @evaporei in #1443
- Get/Set/Update repository level metadata by @paraseba in #1457
- DOC: update overview page with more concrete examples by @ianhi in #1391
- [docs] Update
zarr.config.gettozarr.config.setinperformance.mdby @norlandrhagen in #1459 - Specify Python version for mypy check in upstream by @ianhi in #1461
- Document proxy support by @TomNicholas in #1452
- Clarifications to version policy by @TomNicholas in #1470
- Port: Fix fetching of virtual refs with spaces and other non-url char… by @paraseba in #1472
- Port: More fixes for virtual chunk url escapes (#1474) by @paraseba in #1477
- Fix spec version for new snapshots on V1 repos. by @paraseba in #1482
- Fix icechunk dependency URL in bug report template by @TomNicholas in #1473
- Bump the rust-dependencies group with 3 updates by @dependabot[bot] in #1486
- Made get_client and get_path_str public (#1487) by @d0rianb in #1489
- Bump the actions group with 3 updates by @dependabot[bot] in #1485
- Add configurable max_arrays_to_scan to ManifestPreloadConfig (#1464) by @aladinor in #1491
- Fix anonymous parameter not passed to S3Options in storage functions by @aladinor in #1493
- Make spec_version a property instead of a function by @mpiannucci in #1495
- Add basic http/https docs by @mpiannucci in #1496
- Support creating and pickling conflict errors by @mpiannucci in #1497
- Bump the rust-dependencies group with 4 updates by @dependabot[bot] in #1500
- Bump scientific-python/upload-nightly-action from 0.6.2 to 0.6.3 in the actions group by @dependabot[bot] in #1499
- Minor bug in merge conflict report by @paraseba in #1504
- Upgrade object_store dependency to get proper retries on 429 by @paraseba in #1509
- Allow reindex_array on V1 repos by @dcherian in #1514
- Bump the rust-dependencies group with 5 updates by @dependabot[bot] in #1528
- Extend storage stats calculation to include virtual and inline chunks by @TomNicholas in #1483
- Update libraries by @paraseba in #1535
- Port: Bump up railguards for flatbuffer file size. (#1530) by @paraseba in #1536
- Have default contributing instructions do less intensive rebuilds by @ianhi in #1539
- add repo spec version to zarr stateful tests by @ianhi in #1540
- Add amend to stateful_repo_ops by @ianhi in #1537
- Bump the rust-dependencies group with 4 updates by @dependabot[bot] in #1543
- Clarify supported file formats for virtual references by @TomNicholas in #1546
- Add missing check for GcsCredentials.Anonymous by @li-em in #1556
- Identify chunks as virtual/native by @li-em in #1547
- Add roundtrip serialization property tests for types in change sets file by @ebarylko in #1525
- Add flatbuffer schematics by @dcherian in #1570
- Retry on 408 by @dcherian in #1572
- Add warning for large number of chunks in a single commit by @li-em in #1568
- Remove unused Cargo lock file by @li-em in #1575
- Bump the rust-dependencies group across 1 directory with 8 updates by @dependabot[bot] in #1573
- Smaller property test by @paraseba in #1582
- Add snapshot id to NewCommitUpdate repo change by @paraseba in #1581
- Manifests inform the nodes they contain by @paraseba in #1584
- Allow to create manifests with known id by @paraseba in #1585
- Obtain the repo info object used to calculate ops log by @paraseba in #1587
- Ability to write repo info object to a different path by @paraseba in #1588
- Fix clearing vccs bug (IC2) by @TomNicholas in #1592
- Fix dep issues by @mpiannucci in #1604
- Support empty commits by @mpiannucci in #1603
- Bump min zarr to 3.1.0 by @dcherian in #1599
- Clean up stateful tests. by @dcherian in #1512
- FEAT: expose session type as an enum by @ianhi in #1602
- Improve empty commit error message to suggest allow_empty by @ianhi in #1611
- Fix session mode test: use allow_empty and fix mypy error by @ianhi in #1612
- Don't panic when writing manifest fails. by @dcherian in #1609
- TEST: Add stateful move by @ianhi in #1606
- Update dask + forking error message by @dcherian in #1608
- Move test deps to dev-deps by @mpiannucci in #1614
- FEAT: allow_empty for amend by @ianhi in #1613
- Add inline rust documentation by @ianhi in #1579
- FIX: correctly identify initial snapshots by @ianhi in #1615
- DOC: update release doc with more description of the options by @ianhi in #1616
- feat: add
Repository.fetch_spec_versionto Python API by @lindseynield in #1618 - Bump the rust-dependencies group across 1 directory with 8 updates by @dependabot[bot] in #1620
- Bump time from 0.3.44 to 0.3.47 by @dependabot[bot] in #1622
- Sync zarr upstream by @mpiannucci in #1624
- Introduce new Manifest::from_sorted_vec by @paraseba in #1629
- Bump the rust-dependencies group with 2 updates by @dependabot[bot] in #1633
- Update contributor docs by @jbusecke in #1640
- docs: add missing dev setup steps to icechunk-python README by @samantha-earthmover in #1646
- fix: link to Icechunk Python documentation by @pfackeldey in #1650
- xfail
tests/test_zarr/test_store/test_icechunk_store.py::TestIcechunkStore::test_get_manyby @jbusecke in #1643 - Fix redirect test failure (only happening on macos) by @jbusecke in #1654
- Port stalled stream retry to IC2 by @dcherian in #1664
- Bump the rust-dependencies group across 1 directory with 10 updates by @dependabot[bot] in #1668
- Add configurable retries for
update_repo_infoby @mpiannucci in #1659 - Delete FIXMEs and re-enable test by @paraseba in #1669
- Migrate tests from MinIO to RustFS by @paraseba in #1673
- Add stubtest to CI to catch stub/runtime mismatches by @dcherian in #1675
- Log config changes in ops log by @mpiannucci in #1676
wasmbuild of icechunk core by @mpiannucci in #1628- configurable num updates for repo info file by @dcherian in #1479
- fix: make WASI futures Send in compat boxing by @mpiannucci in #1685
- Add
inspect_repo_infoby @dcherian in #1686 - fail with nice error when trying amend the very first commit by @ianhi in #1538
- Optimize fetch of repo object by @li-em in #1638
- Tweak build configs for Rust by @li-em in #1625
- Make the repo info cache in AssetManager optional by @li-em in #1697
- Protect repo object updates from time skew by @mpiannucci in #1695
- Bump the rust-dependencies group with 4 updates by @dependabot[bot] in #1703
- Fix S3 client ignoring user specified settings by @dcherian in #1700
- Give initial snapshot an empty transcation log by @ianhi in #1623
- Update flake by @paraseba in #1704
- Move repo config into repo object by @mpiannucci in #1694
- Amend checks when moves are involved by @li-em in #1706
- Add pytest: exists, fetch_spec_version + settings by @dcherian in #1705
- Two version icechunk stateful tests by @ianhi in #1681
- Alternative way to do manifest split without caching the splits by @paraseba in #1498
- Initialize spec_version when copying a ModelStore by @li-em in #1712
- fix bugs with upgrading the repo in stateful tests by @ianhi in #1714
- Track rebase attempts in snapshot metadata by @mpiannucci in #1713
- require explicit opt in or out on dry run for repo upgrade by @ianhi in #1716
- Set parent of GC-ed snapshots to
INITIAL_SNAPSHOT_IDwhen applicable by @dcherian in #1708 - GC: Make sure "non pointed but newer" snapshots are accessible. by @dcherian in #1717
- Fix migration of deleted tags by @dcherian in #1718
- Upgrade stateful repo ops testing significantly by @dcherian in #1687
- More GC/expiration invariants in stateful test by @dcherian in #1723
- Set up pixi for development by @li-em in #1674
- Add mini-design doc for IC2 GC consistency by @paraseba in #1725
- Remove flakiness in stateful repo test by @dcherian in #1732
- Resolve the issue of concurrent changes during GC by @paraseba in #1729
- Skip logging heavy objects by @dcherian in #1733
- Bump the rust-dependencies group with 2 updates by @dependabot[bot] in #1736
- Bump the actions group with 2 updates by @dependabot[bot] in #1735
- ListBucket permissions testing by @li-em in #1711
- Add rountrip serialization prop tests for remaining types by @ebarylko in #1734
- Fix logo display in README by @li-em in #1745
- Generate fake ops log during migration by @mpiannucci in #1731
- Audit testing for IC1 & IC2 by @dcherian in #1739
- Add criterion microbenchmark suite by @dcherian in #1747
test_storage: concurrent function application by @dcherian in #1751- Add feature flags by @paraseba in #1755
- Optimize manifest writes by @dcherian in #1753
- Parameterize rust tests over V1, V2 by @dcherian in #1743
- Design doc for more IC2 format changes by @paraseba in #1730
- Refactor UpdateType and improve repr by @li-em in #1750
- Add NodeSnapshot cache on Snapshot by @dcherian in #1756
- Simplify RustFS initialization by @li-em in #1762
- Make sure migration errors on duplicate migration by @mpiannucci in #1757
- FEAT/FIX: more consistent move behavior + options on reindex by @ianhi in #1621
- More optimizations by @dcherian in #1764
- Retry on 429, 499 also. by @dcherian in #1768
- FIX: don't override all storage defaults when user passes a config by @ianhi in #1699
- Allow relative virtual chunks by @paraseba in #1765
- FIX: allow delete + recreate in rebase by @ianhi in #1742
- New inspect_manifest functionality by @paraseba in #1770
- make sure contrib docs link to the latest version by @ianhi in #1772
- Fix justfile by @paraseba in #1774
- CI: Fixes for mypy stubs by @li-em in #1777
- AI/Contributing policy by @ianhi in #1766
- CI: Avoid dep on typing-extensions, ignore type check still in draft status by @li-em in #1779
- tweak Pixi config to avoid long CI runtimes by @li-em in #1763
- Bump ruff to newest version and resulting changes by @jbusecke in #1727
- Optimize
get_existing_node: avoid cloningVec<ManifestRef>by @dcherian in #1775 - Fix snapshot node cache. by @dcherian in #1784
- [fix] Update repo info cache when changes made by @paraseba in #1787
- [fix] ancestry was using two different versions of the repo info by @paraseba in #1788
- Bump the rust-dependencies group with 6 updates by @dependabot[bot] in #1786
- [fix] Inconsistencies in Repository's diff and rearrange_session by @paraseba in #1791
- Compress virtual chunk locations. by @paraseba in #1776
- Retry on Connection Reset errors while streaming bytes. by @dcherian in #1795
- Minimum supported Rust version policy by @li-em in #1793
- Docs: Update VirtualChunkContainer examples by @tylanderson in #1645
- Warm snapshot cache to speed up migration by @mpiannucci in #1778
- Don't use snapshot flush time as repo update time by @paraseba in #1798
- Fix docs warning for readthedocs by @li-em in #1802
- bump all versions to 2.0.0-alpha3 by @ianhi in #1796
New Contributors
- @ebarylko made their first contribution in #1343
- @evaporei made their first contribution in #1426
- @d0rianb made their first contribution in #1489
- @aladinor made their first contribution in #1491
- @samantha-earthmover made their first contribution in #1646
- @pfackeldey made their first contribution in #1650
- @tylanderson made their first contribution in #1645
Full Changelog: v1.1.11...v2.0.0-alpha.3