- orphan
True
The 12.0 release of the LSST Science Pipelines includes Qserv release 2016_05.
release-v12-0-qserv-major-changes
release-v12-0-qserv-bug-fixes
release-v12-0-qserv-internal-improvements
See also:
release-v12-0-qserv-shared-scans
release-v12-0-qserv-large-results
release-v12-0-qserv-query-cancellation
release-v12-0-qserv-query
release-v12-0-qserv-logging
release-v12-0-qserv-sqlalchemy
release-v12-0-qserv-sql-css
release-v12-0-qserv-multinode-docker
release-v12-0-qserv-database-delete
release-v12-0-qserv-mariadb
release-v12-0-qserv-czar-in-proxy
release-v12-0-qserv-docs
Qserv's shared scan capability was extensively reworked, including a new scheduler and page-locking memory support on the workers. Performance was greatly improved. DM-3755,DM-4677,DM-4697,DM-4807,DM-4943,DM-5313,DM-5514
Qserv previously had an issue where dense and highly distributed queries could cause workers to "fire-hose" the czar, causing it to lock up or fail due to memory and/or CPU exhaustion. Threading and flow control changes were made on the czar and workers to address this. A memory management issue in the mysql proxy LUA code was also addressed. DM-5908,DM-5909,DM-5910,DM-6149
Query cancellation improvements and rework begun in the W16 cycle were completed. Queries in flight are now canceled robustly on both czar and workers when a user types ^C to the mysql client. DM-2699,DM-3562,DM-3564,DM-3946,DM-3945
Qserv now correctly handles queries with "where objectId between", and "where objectId in". DM-2873,DM-2887
Qserv log messages now include user-friendly thread IDs and unique query IDs. This improves consumability of logs for both real users and automated tools. DM-5314,DM-4755,DM-4756
The SQlAlchemy client library makes a few probe queries on connect to assess Unicode support by the engine. Some of these queries were problematic for the czar. This was addressed and SQLAlchemy can now be used as an alternative client for Qserv. DM-4648
Qserv's central shared-state (CSS) meta-data service implementation, formerly based on Zookeeper, was replaced with a more robust and transactional SQL-based implementation. Dependencies on Zookeeper were removed from the build. DM-4003,DM-4138,DM-3192,DM-3574,DM-2733
A multi-node integration test suite was added, which may be run on a single host via Docker. The multi-node integration test is integrated with Travis CI, and now runs automatically on commits to all branches of the LSST Qserv git repo on github. DM-5218,DM-3985,DM-4295,DM-3910,DM-3922,DM-4395
Distributed table and database deletion were implemented. Watcher process (wmgr) does deletion on workers, and state is synchronized via CSS. DM-2522,DM-2622,DM-2624,DM-4206,DM-2625
Qserv and all associated services and libraries were ported from MySQL to MariaDB. Dependencies on mysql and mysqlclient were removed from the build. DM-224,DM-5319,DM-5125,DM-5122,DM-4705,DM-3949,DM-5026
The Qserv czar was previously wrapped with SWIG then hosted within a Python process which communicated with mysqlproxy over an XMLRPC interface implemented in Twisted and LUA. The czar has been reworked so it is now directly wrapped to LUA and brought into the mysqlproxy process. This allowed elimination of the XMLRPC wire protocol and associated code, elimination of several external library dependencies, and removal of all Python involvement from the proxy/czar process. DM-4348,DM-5307
Qserv user and installation documentation (Qserv 2016_05 documentation) was updated/corrected. DM-5754,DM-4105
release-v12-0-qserv-service-timeout
release-v12-0-qserv-testqdisp
release-v12-0-qserv-match-tables
Qserv services would crash in some instances if left running for several days. The cause was tracked down to a missing null handle check in a mysql wrapper library, which was provoked when server connections would timeout. DM-5594
This was tracked down to a problem with the Executive class mocks used by the unit test. These mocks did not handle threading during cancellation correctly. DM-4928
The qserv-data-loader.py script was not invoking the correct partitioner for match tables, and was not passing all required CSS parameters down to the CSS update code. DM-3656
release-v12-0-qserv-stream-logs
release-v12-0-qserv-scons
release-v12-0-qserv-compilers
release-v12-0-qserv-style
release-v12-0-qserv-lib-updates
release-v12-0-qserv-dead-code
release-v12-0-qserv-docker
release-v12-0-qserv-integration-tests
release-v12-0-qserv-futurize
release-v12-0-qserv-worker-config
release-v12-0-qserv-taskmsgfactory2
release-v12-0-qserv-installation-files
Qserv was cut over to using stream based logging macros exclusively, and the boost format style logging macros (considered harmful) were removed from the LSST log package. A redundant logging wrapping layer in qserv was also removed. DM-4616,DM-5204,DM-5202,DM-3037
Overly verbose build output from scons was greatly reduced. Scons files were reworked to treat shared libraries consistently, and some latent incorrect shared lib linkages were corrected. Scons files were also adjusted to avoid unnecessary copying of the source tree into the build tree. DM-3447,DM-2421,DM-4145,DM-3686,DM-3707
Issues were addressed to ensure that qserv builds and passes all unit tests on Linux with gcc 4.8.5 - 5.3.1, and on MacOSX with XCode 7.3.0. Warnings were addressed wherever possible, and the builds are now largely warning free except for some warnings produced by third-party library dependencies. Warnings generated by the Eclipse Neon C++ code analyzer were also addressed wherever possible. DM-3584,DM-3663,DM-3803,DM-3772,DM-3779,DM-3915,DM-4398,DM-4470,DM-4529,DM-4704,DM-5788,DM-6292
Various small systematic changes were made across the Qserv code base for style consistency. Anonymous namespaces were moved to top level of translation units. A single space was added after "if" before the subsequent paren. toString() functions were removed in favor of streaming operators. Non-standard uint type was replaced with unsigned int. DM-4753,DM-3888,DM-2452,DM-3805
Qserv was rolled forward to scisql 0.3.5, mysqlproxy 0.8.5, boost 1.60, and the latest changes from XRootD were incorporated. We also moved from using a forked version of the sphgeom library to following the tip of the official LSST version. DM-4938,DM-4786,DM-5394,DM-2178,DM-4092,DM-2334
Unused worker configuration templates and deprecated czar merging codes were removed. Unused objectId hinting code was removed from the proxy LUA miniParser. DM-4440,DM-2320,DM-3952
Docker container build and deploy scripts continued to be extended, enhanced, and debugged. Scripts are currently based on shmux, and have been used for administration of multiple qserv clusters at both NCSA and IN2P3. DM-3199,DM-6130,DM-4438,DM-5187,DM-5402,DM-4523,DM-5336
Integration tests were added involving blobs and non-box spatial constraints. Additionally, a facility to reset the empty chunk list in the czar was added, which greatly streamlines the integration tests. DM-991,DM-2900,DM-4383
Python admin scripts were run through "futurize -1". One print change was made to runQueries.py. DM-6324
INI file style configuration support was added for the worker, in support of being able to configure shared scans without resorting to environment variables. DM-5209
to TaskMsgFactory. I can't believe we track this kind of nonsense. DM-2060
Directories cfg/ and proxy/ in the qserv install tree were moved under share/ and lib/ for consistency. DM-1355