DM-55404: A low-latency synchronization mechanism on the data management events sent by the Replication/Ingest system to Czar in Qserv#1033
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a low-latency event-notification path from the Replication/Ingest subsystem to the Czar, so Czar-side components (notably CSS cache) can react promptly to data-management events (publish/delete), and centralizes chunk-disposition (“chunk map”) updates into a dedicated service.
Changes:
- Bump the management/replication REST API version to 57 across UI, C++ HTTP meta modules, and the Python admin interface.
- Add
ChunkMapas a replica service (lazy-created viaServiceProvider) and refactor replication task chunk-map updates to use it. - Add
cconfig::EventServiceandcconfig::DataManagementEvent, plus a new Czar/eventendpoint and a Replication-side request (PostEventQservCzarMgtRequest) to post events to Czars.
Reviewed changes
Copilot reviewed 31 out of 31 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| src/www/qserv/js/Common.js | Bumps REST API version used by the web UI. |
| src/replica/services/ServiceProvider.h | Adds chunkMap() accessor and member for the new service. |
| src/replica/services/ServiceProvider.cc | Lazily constructs and returns the ChunkMap service. |
| src/replica/services/CMakeLists.txt | Adds ChunkMap.cc to the replica services build. |
| src/replica/services/ChunkMap.h | New service interface for tracking/updating chunk disposition. |
| src/replica/services/ChunkMap.cc | Implements chunk-map refresh and bulk persistence into QMeta. |
| src/replica/qserv/PostEventQservCzarMgtRequest.h | New request type for posting management events to Czars. |
| src/replica/qserv/PostEventQservCzarMgtRequest.cc | Implements HTTP POST of events to the Czar /event endpoint. |
| src/replica/qserv/CMakeLists.txt | Adds the new request implementation to the build. |
| src/replica/contr/ReplicationTask.h | Removes in-task chunk-map implementation (moved to service). |
| src/replica/contr/ReplicationTask.cc | Switches chunk-map updates to use ServiceProvider::chunkMap(). |
| src/replica/contr/HttpIngestModule.h | Declares helper to notify Czars about data-management events. |
| src/replica/contr/HttpIngestModule.cc | Posts events on publish/delete operations and updates chunk map prior to notifications. |
| src/replica/CMakeLists.txt | Links replica library against cconfig for event types. |
| src/http/MetaModule.cc | Bumps HTTP meta-module version to 57. |
| src/http/ChttpMetaModule.cc | Bumps CHTTP meta-module version to 57. |
| src/czar/HttpSvc.h | Plumbs EventService into the Czar HTTP service creation API. |
| src/czar/HttpSvc.cc | Registers new POST /event handler and passes EventService through. |
| src/czar/HttpManagementModule.h | New Czar module for handling posted management events. |
| src/czar/HttpManagementModule.cc | Implements event ingestion endpoint and posts into EventService. |
| src/czar/Czar.h | Adds EventService member to Czar. |
| src/czar/Czar.cc | Starts EventService and subscribes to invalidate CSS cache on relevant events. |
| src/czar/CMakeLists.txt | Adds HttpManagementModule.cc to the Czar build. |
| src/cconfig/EventService.h | New async event bus abstraction for Czar to handle posted events. |
| src/cconfig/EventService.cc | Implements thread pool and dispatch logic for event notifications. |
| src/cconfig/DataManagementEvent.h | New event type and JSON serialization interface. |
| src/cconfig/DataManagementEvent.cc | Implements event type parsing/serialization and validation. |
| src/cconfig/CzarConfig.h | Adds config knob for event service threads. |
| src/cconfig/CzarConfig.cc | Validates replication.num_event_threads is non-zero. |
| src/cconfig/CMakeLists.txt | Adds new cconfig sources and links boost_system. |
| python/lsst/qserv/admin/replication_interface.py | Bumps Python admin replication API version to 57. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
iagaponenko
force-pushed
the
tickets/DM-55404
branch
from
July 16, 2026 20:35
16face5 to
3c0f908
Compare
iagaponenko
force-pushed
the
tickets/DM-55404
branch
from
July 16, 2026 23:06
3c0f908 to
8005a27
Compare
jgates108
approved these changes
Jul 17, 2026
jgates108
left a comment
jgates108
left a comment
Contributor
There was a problem hiding this comment.
LGTM - I focused mostly on the qserv parts.
Unit tests for the new classes. Subscribe Czar for handling certain data management events which require clearning the empty chunk lists in the CSS cache.
Pushed the chunk map management from the Replication thread into the ServiceProvider component. This will allow accessing/updating the map from various components of the Replication Controller.
iagaponenko
force-pushed
the
tickets/DM-55404
branch
from
July 17, 2026 04:14
8005a27 to
0f1991d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.