HPCC-16965 Performance warnings from Cassandra workunit code #9542
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix "Aggregation query used without partition key" warning when counting total number of workunits. Execute counts independently and asynchronously on all partitions instead. Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
|
https://track.hpccsystems.com/browse/HPCC-16965 |
|
@ghalliday Please review |
ghalliday
reviewed
Feb 1, 2017
plugins/cassandra/cassandrawu.cpp
Outdated
| @@ -3670,39 +3712,31 @@ class CCasssandraWorkUnitFactory : public CWorkUnitFactory, implements ICassandr | |||
| unsigned validateRepository(bool fix) | |||
| { | |||
| unsigned errCount = 0; | |||
| // MORE - if the batch gets too big you may need to flush it occasionally | |||
| CassandraBatch batch(fix ? cass_batch_new(CASS_BATCH_TYPE_LOGGED) : NULL); | |||
| CIArrayOf<CassandraStatement> secondaryBatch; | |||
|
@richardkchapman all looks good to me. One comment about an unused variable. |
Change partitioning so that all tables use same partitioning column as far as possible. Updates to the parent and all children can thus be done efficiently in a single unlogged batch. Changes to secondary tables used for searching are moved into a separate batch, implemented as independent async Cassandra calls for best performance. Change validateRepository code to use much smaller and more appropriate batches Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Deleting and then adding some exceptions within a single workunit commit would end up losing the newly added ones - this is because of how Cassandra batches work when rows are both deleted and updated within the same batch. I doubt this would cause any issues in practice outside of test suite code. Also, removing certain information from a workunit - specifically the file associations but there may be other instances - would result in this information not being properly committed. We need to bind NULL to any columns that we want cleared in an update. Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
richardkchapman
force-pushed
the
cassandra-warn4
branch
from
afc336a
to
dab7202
Compare
|
@ghalliday I have removed the unused variable and repushed |
|
Automated Smoketest
Install hpccsystems-platform-community_6.3.0-trunk0.el7.x86_64.rpm Unittest result:
HPCC Stop: OK |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.