New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DM-33314: Implement missing methods for Cassandra backend #25
Conversation
Packing columns into blobs did not help with performance, and storage size grows too. Not useful, getting rid of it to simplify things.
# prepare it because it's not reusable. | ||
statement = cassandra.query.SimpleStatement(full_query) | ||
statements.append((statement, params)) | ||
_LOG.debug("getDiaObjects: #queries: %s", len(statements)) | ||
# _LOG.debug("getDiaObjects: queries: %s", queries) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove commented line?
values = (ssObjectId, apdb_part, apdb_time_part, diaSourceId) | ||
queries.add(self._prep_statement(query), values) | ||
|
||
# _LOG.debug("query: %s", query) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove?
if not clust_columns: | ||
raise ValueError(f"Table {table_name} configuration is missing primary index") | ||
# if not clust_columns: | ||
# raise ValueError(f"Table {table_name} configuration is missing primary index") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
uncomment? remove lines?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I forgot to clean that up after testing, will remove it completely.
LGTM! |
The implementation uses ssObjectId for partitioning key, there are no other natural keys and size is too large for one partition. This may be reconsidered once we learn more about how SSObjects will be queried.
Efficient search by diaSourceId needs another table which is partitioned by that column. There may be other ways to implement association, will need to think more about it.
Remove non_prepared_statements option, always use prepared.
7cde542
to
7a03f92
Compare
APDB API added few methods to support replication to/from PPDB on a previous ticket. This update adds an implementation of these new methods to Cassandra backend. Cassandra code was updated to use execution profiles, as the previous model gets deprecated. Also a continuing refactoring of Cassandra implementation in a couple of commits. It makes sense to look at the final diff, some pieces were changed more than once.