DM-43097: Replication-related improvements #48
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andy-slac
force-pushed
the
tickets/DM-43097
branch
6 times, most recently
from
7c07e7a
to
64cdd83
Compare
Replication-related methods now appear in a separate class `ApdbReplica`. Instances of `ApdbReplica` are made from the same ApdbConfig.
Since code version for ApdbReplica classes is now separate from Apdb code version we also need to store and check those numbers. Those new version are stored and checked only if replica tables are enabled (use_insert_id is true).
DiaInsertId structure changed, `id` now has integer type. One DiaInsertId corresponds to multiple `store` calls. This is needed to reduce the number of transfer chunks. One chunk corresponds to a configurable time window (10 minutes by default). Note that this update changes version number for replica classes to 1.0.0. We should not have databases with replication enabled so this should not be an issue. Change types of collections returned from ApdbTableData
andy-slac
force-pushed
the
tickets/DM-43097
branch
3 times, most recently
from
b6705a9
to
5dcec20
Compare
andy-slac
commented
Apr 16, 2024
andy-slac
force-pushed
the
tickets/DM-43097
branch
from
5dcec20
to
f020e14
Compare
InsertId concept is replaced with ReplicaChunk. All names related to InsertId have been changed to a combination of replica/chunk. The only remaining mention of insert_id is the config field `use_insert_id`. I do not want to change it yet, will drop and replace when I migrate to YAML config. Related to that - frozen config also has `use_insert_id` for now.
SQL code is now in `lsst.dax.apdb.sql`, Cassandra is in `...cassandra`.
`felis.simple` is going to be replaced by `felis.datamodel`, but `datamodel` is not exactly the same as `simple` was. I copied `felis.simple` module here and updated it to use `datamodel` classes as an input.
Move parts of ApdbSqlSchema class responsible for conversion of schema model into SQLAlchemy schema into a separate class which can be reused by dax_ppdb.
andy-slac
force-pushed
the
tickets/DM-43097
branch
from
f020e14
to
2653230
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A big refactoring happened as well:
Apdbclass to a newApdbReplicaclass.apdb.sqlandapdb.cassandrapackages.Interfaces used by AP pipeline are unchanged and will work as before.
Replication tools are being implemented in
dax_ppdbpackage.