2024.11.1

What's Changed

• DM-47444: Extend Qserv integration test to cover the user table ingest API of the HTTP frontend
• DM-46919: Extend the http frontend of Qserv to support pushing CSV files in the multipart/form-data body
• DM-46111: Refactored and extend the Qserv Documentation Portal: https://qserv.lsst.io/
• DM-47345: Upgrade Qserv to Apache Arrow and Parquet version 18
• DM-47268: Sample catalog data and configuration files referenced in the documentation. The top-level folder doc_datasets/test101 provides a complete set of the input data and configuration files that are needed to ingest sample 3 tables into Qserv. These files are references in the "Ingest Workflow Developer's Guide" document.
• DM-46952: Update submodules log and sphgeom to the latest versions

NOTE: The version number of the Qserv REST API: 39

Full Changelog: 2024.10.1-rc1...2024.11.1-rc1

2024.10.1

What's Changed

• DM-45596: Migrate GHA CI setup of Qserv to install docker-compose as an action
• DM-46119: Install application debug tools lsof, strace and netstat into the run base container
• DM-46430: Garbage collection of the completed table contribution requests to prevent OOM crashes of the long running worker ingest services
• DM-45811: Additional thresholds in the query selectors on the Qserv Web Dashboard to allow selecting large queries exceeding the specified number of chunks, the number of bytes, or the number of rows
• DM-46017: Fixed a bug in the Qserv worker ingest service causing the service to lock up while ingesting large contributions
• DM-45842: Add the AWS SDK for C++ into the Qserv build container
• DM-45230: Qserv Web Dashboard crashes when trying to render the very long queries
• DM-45595: Migrate the Qserv worker-side table ingest service to CPP-HTTPLIB
• DM-45929: Worker-centric view of the ingest contributions on the Qserv Web Dashboard to prevent overloadng the Replication Controller and the Dashboard UI when exploring millions of contributions into a catalog
• DM-46320: Supporting the multipart/form-data format in the Qserv's push-mode ingest protocol as a more convenient alternative for the proprietary binary protocol
• DM-46397: Extend Qserv integration test to use HTTP-based push mode catalog ingest

NOTE: The version number of the Qserv REST API: 38

Full Changelog: 2024.7.1-rc1...2024.10.1-rc1

2024.7.1

What's Changed

• DM-45311: Upgraded the Apache Arrow and Parquet libraries to version 17
• DM-45049: Add support for SSL/TLS in the Qserv Czar frontend: https://confluence.lsstcorp.org/display/DM/HTTP+frontend+of+Qserv
• DM-44780: Implement multipart/form-data parser for Qserv. Added CPP-HTTPLIB to Qserv.
• DM-44641: Extend qhttp to allow streaming read of requests

Full Changelog: 2024.5.1-rc5...2024.7.1-rc1

2024.5.1

What's Changed

• DM-42810: Extend Qserv integration test to support the HTTP-based front-end
• DM-43282: HTTP-based simple ingest service for fully-replicated tables in Qserv
• DM-44281: Add support for base64 encoding for the binary column data in the HTTP frontend of Qserv
• Updated Arrow and Parquet libraries to version 16

Full Changelog: 2024.4.1-rc1...2024.5.1-rc5

2024.4.1

What's Changed

• Upgraded Log4cxx to release v1.1.0
• Added HTTP frontend for submitting user queries: https://confluence.lsstcorp.org/display/DM/HTTP+frontend+of+Qserv
• Improved algorithm for “booting” long-running queries at workers
• Improved monitoring of the worker queries on the Dashboard
• Fixed the state transition bug on the sync HTTP interface

Full Changelog: 2024.3.1-rc2...2024.4.1-rc1

2024.3.1

What's Changed

• Qserv build and run containers now based on almalinux 9
• Dashboard improvements including configuration display, query state, progress monitoring, and partial result file status
• Ingest seatbelt added against TEXT/BLOB columns in director tables
• Fix for mysql connector problem which was causing worker crashes
• arrow and parquet libs updated to version 15
• Per-build documentation auto-upload fixed, and docs updated to latest Rubin style
• Migration of Qserv control plane to HTTP
• Removal of legacy XRootD SSI result streaming code
• Fix for race condition in qhttp module which was causing intermittent unit test failures
• Fix for temporary message table cleanup issue triggered by SQL COMMIT user queries

Full Changelog: 2023.10.1-rc1...2024.3.1-rc2

2023.10.1

What's Changed

• New file-based results delivery configuration option
• Large-result query early termination fixes
• Web Dashboard extensions to allow monitoring of activities at czar and worker nodes
• Full-text index on query metadata table has been removed (was found to be cause of long startup times)
• Fix for director index column match case-sensitivity issue

Full Changelog: 2023.7.1-rc3...2023.10.1-rc1

2023.7.1

What's Changed

• Fixes to folder-creation logic at replication controller launch
• Support for director indices with non-unique primary keys
• Error checking to prevent attempts to create director indices for non-published tables

Full Changelog: 2023.6.1-rc1...2023.7.1-rc3

2023.6.1

What's Changed

• The jemalloc memory allocator was re-enabled
• Fixes for excessive memory use by the administrative web dashboard
• Partitioner now supports parquet file inputs (thanks, @hellebore74!)
• Non-local director index ingest option is no longer supported
• Global query completion/cancellation protocol enhancement

Full Changelog: 2023.5.1-rc1...2023.6.1-rc1

2023.5.1

What's Changed

• Use templatized queries at workers to reduce memory allocation
• Fixes for extreme slowdowns when cancelling broad subchunk queries
• Eliminated false-positive error messages for missing optional ingest config parameters
• Fixed some issues with transaction state counters and colors on web dashboard
• Upgrades of package dependencies (XRootD, lsst/log, and lsst/sphgeom)

Full Changelog: 2023.4.1-rc1...2023.5.1-rc1