HISTORY.rst

History

X.Y.Z (YYYY-MM-DD)

• Bump donfig dependency to 0.8.0 due to install error with pip (332)
• Fix katdal import typos (331)
• Add an epoch argument to xds{from,to}_zarr to uniquely identify datasets in a distributed context (330)
• Improve table schema handling (329)
• Identify channel and correlation-like dimensions in non-standard MS columns (329)
• DaskMSStore depends on fsspec >= 2022.7.0 (328)
• Optimise broadcast_arrays in katdal import (326)
• Change dask-ms katdal import to dask-ms import katdal (325)
• Configure dependabot (319)
• Add chunk specification to dask-ms katdal import (318)
• Add a dask-ms katdal import application for exporting SARAO archive data directly to zarr (315)
• Define dask-ms command line applications with click (317)
• Make poetry dev and docs groups optional (316)
• Only test Github Action Push events on master (313)
• Move consolidated metadata into partition subdirectories (312)
• Set _ARRAY_DIMENSIONS attribute on Data Variables (311)
• Use JSON codec for writing zarr strings (310)
• Address warnings (309)

0.2.20 (2024-01-30) ------------------* Update calver dependencies to use inequality, rather than caret dependency specifiers (307) * Update pre-commit to actions/python@v5.0.0 (308) * Update readthedocs python version to 3.9 and poetry to 1.7.1 (303) * Re-enable exceptions in multiprocessing test case (302) * Fix auto-formatted f-strings (301)

0.2.19 (2023-11-13) ------------------* Upgrade to pyarrow 14.0.1 (299) * Use new minio.MinioAdmin client in test suite and update minio server version (298) * Replace black with ruff in pre-commit hooks (297) * Lazily load casacore tables module (294) * Deprecate Python 3.8 support (296) * Temporarily add Pandas as an arrow extra dependency (296)

0.2.18 (2023-09-20) ------------------* Ignore non-existent columns (290) * Add experimental fragments functionality (282, 291) * Add rechunk by size utility funtion (284) * Run CI weekly on Monday @ 2h30 am UTC (288) * Update minio server and client versions (287) * Retain ROWID coordinates during MS conversion (286)

0.2.17 (2023-08-02) ------------------* Change setmaxcachesize to require a read lock, not a write lock (281)

0.2.16 (2023-05-26) ------------------* Update calver versioned software such as dask and xarray to 2023 variants (279) * Remove unused requirements_dev.txt (275) * Support optional CASA columns (270)

0.2.15 (2022-10-19)

• Fix poetry install and cache hit detection on CI (266)
• Support --taql-where for Measurement Sets in dask-ms convert (264)
• xds_from_zarr should always open zarr groups in read mode (262)
• Fail on reads if non-existent or invalid store type found (259, 260)

0.2.14 (2022-10-04)

• Fix for nan chunks/dims breaking writes (255)
• Fix minio deb download URL (257)
• Apply black to the code base (252)
• Manage dask-ms with python poetry (250)
• Don't force conversion to tuple during attribute decoding (248)
• Fail if zarr chunks exceed 2GB (247)
• Add a --exclude argument to dask-ms convert that allows columns to be excluded during conversion (246).
• Make --output a required dask-ms convert argument (245).

0.2.12 (2022-09-09)

• Consolidate metadata by default during xds_to_zarr and always attempt to read consolidated metadata in xds_from_zarr (243).
• Fail if ROWID dimensions don't match associated dataset arrays (239)
• Pin flake8 to >= 4.0.0 and < 5.0.0 (240)

0.2.11 (2022-07-27)

• Improve chunking in xds_to_zarr when rechunk==True. (236)
• Assign custom fsspec storage options from url match in yaml configuration files. (237)

0.2.10 (2022-07-18)

• Minor DaskMSStore changes to improve URL handling. (233)
• Deprecate Python 3.7 support and test on Python 3.10 (232)
• Support loading of storage_options in dask-ms convert (231)
• Reintroduce group columns on output CASA format only (230)
• Stop converting results returned from DaskMSStore into Path objects (229)
• Accept storage_options in dataset read/write methods (228)
• Add __repr__ and __str__ for DaskMSStore (224)

0.2.9 (2022-06-28)

• Recreate Grouping Columns when writing to CASA format in dask-ms convert (222)
• Ignore SOURCE subtable in dask-ms convert (221)
• Fix performance bug in cached_array implementation. (220)
• Use DaskMSStore throughout dask-ms convert (218)
• Upgrade from deprecated visit_Num to visit_Constant (217)
• Ensure url and table consistency in DaskMSStore (216)
• Wait for minio to start with greater precision (215)
• Chunk correctly when reading from parquet. (210)
• Fix minor bugs in zarr and conversion functionality. (208)
• Add xds_to_storage_table. (207)
• Add option to rechunk automatically on writes. (204)
• Raise more informative error. (203)
• Improve tiling. (202)
• Do not create spurious fields in zarr writes. (200)
• Error out when missing datavars should be written. (197, 198)
• Allow non-standard columns to be tiled. (196)

0.2.8 (2022-04-06)

• Fix roundtripping of boolean tensor arrays. (194)
• Ignore unsupported kwargs in xds_from_storage_* functions. (192)
• Improve handling of subtables with variably sized rows in daskms-convert. (191)
• Ensure that xds_from_zarr sorts groups as integers and not strings (188)
• Ensure Natural Ordering for parquet files (183)
• Fix xds_from_zarr and xds_from_parquet chunking behaviour (182)
• Add LazyProxy and LazyProxyMultiton patterns to dask-ms (177)
• Support cloud native storage formats via fsspec (174)

0.2.7 (2022-01-13)

• Fix inclusion of MANIFEST.in files (173)
• Add --group-columns to dask-ms convert for CASA Formats (169)
• Add ComplexArray -> numpy conversion (168)
• Ignore row dimension when fixing column shapes (165)
• Bump pip from 9.0.1 to 19.2 (164)
• Fix zarr coordinate writes (162)
• Deprecate Python 3.6 (161)
• Add IMAGING_WEIGHT_SPECTRUM to default Measurement Schema (160)
• Remove default time ordering from xds_from_ms (156)
• Make zarr writes completely lazy (157)
• Copy partitioning information when writing (155)
• Add a dask-ms convert script for converting between CASA, Zarr and Parquet formats (145)
• Convert code-base to f-strings with flynt (144)
• Consolidate Dataset Types into daskms.dataset (143)
• Correct Dataset persistence issues (140)
• Experimental arrow support (130, 132, 133, 135, 136, 138, 145)
• Experimental zarr support (129, 133, 139, 142, 150, 145)
• Test data fix (128)
• Fix array inlining for writes (126)
• Allow Multi-Layer Inlining (125)
• Support DATA Column Expressions (124, 134, 146, 147, 148, 151)

0.2.6 (2020-10-20)

• Remove table close in ThreadPool for the last time (122)
• Respect the High Level Graph specification better during inline array creation (123)
• Support dictionary writes via putvarcol (119)
• Use getcell instead of getcellslice in sorted orderings (120)
• Update to pytest-flake8 1.0.6 (117)
• Test on Python 3.8 (116)
• Depend on python-casacore 3.3.1 (116)

0.2.5 (2020-05-11)

• Remove deadlock in TableProxy weakref.finalize on Python 3.6 (113)
• Use python-casacore wheels for travis testing, instead of kernsuite packages (115)

0.2.4 (2020-04-24)

• Documentation updates (110)
• Provide better warnings for unusual ROWID graphs during table updates (108)
• Work around casacore getcolslice caching (107)
• Update LICENSE year (105)
• Update license and production status in pypi classifiers (104)
• Use WHERE rather than HAVING clause in group ordering TAQL (98)
• Improve the dask task key names for clearer graph visualization (102)
• Cache and inline row runs in write operations (96)
• Support getcolslice and putcolslice in TableProxy (91)
• Use weakref.finalize to cleanup TableProxy and Executor objects (89)
• Pickle Executor key argument (88)
• Deprecate Python 3.5 support and test on Python 3.7 (87)
• Optionally expose TableProxy objects in dataset read/write methods (85)
• Upgrade to python-casacore 3.2 (84)
• Re-introduce xarray handling in dataset.as_variable (83)
• Explicitly require dask Arrays on write datasets (83)
• Document python-casacore install process (80, 81)

0.2.3 (2019-12-09)

• Remove __future__ import (79)
• Update examples (78)
• Only log aggressively when the log level is DEBUG (76)
• Optimise dask graphs produced by dask-ms such that each data access node no longer has common ancestors but is instead an independent root node. This improves memory usage in case of the predict. (75)
• Read-lock TAQL row reference table by default (74)
• Produce write datasets rather a single concatenated dask array (70, 72)

0.2.2 (2019-10-25)

• Fix spacing in TAQL WHERE queries (68)

0.2.1 (2019-10-23)

• Constrain table object access to Executor. Simplify table locking (66).
• Fix stress test (65)
• Remove keywords from variable attributes (64)

0.2.0 (2019-09-30)

• Fix and test non-standard sub-table creation (60)
• Improve sub-table creation logic (59, 60)
• Support table and column keywords (58, 62)
• Support concurrent access of multiple independent tables (57)
• Fix WEIGHT_SPECTRUM schema dimensions (56)
• Pin python-casacore to 3.0.0 (54)
• Drop python 2 support (51)
• Simplify Table Schemas (50)
• Add Concepts + Tutorial Documentation (48)
• Supporting reading and updating column keywords (48)
• Add OBSERVATION, FEED, POINTING, SOURCE table schemas (48)
• Remove single row squeezing in the group_cols="__row__" case (48)
• Handle multi-dimensional string arrays (48)
• Add preliminary example_ms (48)
• Add Concepts + Tutorial Documentation (48)
• Make xarray an optional dependency (45)
• Rename xarray-ms to dask-ms (43)
• Allow chunking by arbitrary dimensions (41)
• Add a simple Dataset, making xarray an optional dependency. (41, 46, 47, 52)
• Add support for writing new tables from Datasets (41, 53)
• Add support for appending to tables from Datasets (41)