simplify creating repair run and segments #101

michaelsembwever · 2017-05-12T07:08:54Z

Repair runs and their segments are one unit of work in concept and the persistence layer should be designed accordingly.

Previous they were separated because the concern of sequence generation for IDs were exposed in the code. This is now encapsulated within storage implementations.

This work allows the CassandraStorage to implement segments as clustering keys within the repair_run table.

ref:

Cassandra performance improvements #94

Repair runs and their segments are one unit of work in concept and the persistence layer should be designed accordingly. Previous they were separated because the concern of sequence generation for IDs were exposed in the code. This is now encapsulated within storage implementations. This work allows the CassandraStorage to implement segments as clustering keys within the repair_run table. ref: - #94 - #101

tlpsonarqube · 2017-05-13T05:00:38Z

SonarQube analysis reported 1 issue

1 minor

Watch the comments in this conversation to review them.

adejanovski · 2017-05-24T16:16:43Z

@michaelsembwever : although I fail to understand why, the 001_Initialize_db.cql file is different in this branch but the change doesn't appear in the files changes in the PR...

The problem is the same here than in another PR : the CQL files that exist in master shouldn't be modified and only new incremental changes should be performed in subsequent files so that we don't break the schema migration logic.
Maybe we could merge this PR with the one that merges the repair_run and repair_segment tables ? This way we would have a single table change instead of 2.

What do you think ?

adejanovski

Schema changes need to be added in new CQL files with a higher version. This will ensure that we don't break the schema migration pattern.

michaelsembwever · 2017-05-24T22:55:31Z

The problem is the same here than in another PR : the CQL files that exist in master shouldn't be modified and only new incremental changes should be performed in subsequent files so that we don't break the schema migration logic.
Maybe we could merge this PR with the one that merges the repair_run and repair_segment tables ? This way we would have a single table change instead of 2.

Let's do that. I'll take the migration to a new table approach as well, as you suggest in the other PR.

Repair runs and their segments are one unit of work in concept and the persistence layer should be designed accordingly. Previous they were separated because the concern of sequence generation for IDs were exposed in the code. This is now encapsulated within storage implementations. This work allows the CassandraStorage to implement segments as clustering keys within the repair_run table. ref: - #94 - #101

ref: - #99 - #94

Makes the schema changes in a separate migration step, so that data in the repair_unit and repair_schedule tables can be migrated over. ref: #99 (comment)

Repair runs and their segments are one unit of work in concept and the persistence layer should be designed accordingly. Previous they were separated because the concern of sequence generation for IDs were exposed in the code. This is now encapsulated within storage implementations. This work allows the CassandraStorage to implement segments as clustering keys within the repair_run table. ref: - #94 - #101

tlpsonarqube · 2017-05-29T14:52:43Z

src/main/java/com/spotify/reaper/resources/CommonTools.java

-    RepairRun repairRun = storeNewRepairRun(context, cluster, repairUnit, cause, owner, nodes.keySet().size(),
-                                            repairParallelism, intensity);
-    checkNotNull(repairRun, "failed preparing repair run");
+    segments = repairUnit.getIncrementalRepair() ? nodes.keySet().size() : tokenSegments.size();


Introduce a new variable instead of reusing the parameter "segments".

Repair runs and their segments are one unit of work in concept and the persistence layer should be designed accordingly. Previous they were separated because the concern of sequence generation for IDs were exposed in the code. This is now encapsulated within storage implementations. This work allows the CassandraStorage to implement segments as clustering keys within the repair_run table. ref: - #94 - #101

* Cassandra performance: Replace sequence ids with time-based UUIDs Makes the schema changes in a separate migration step, so that data in the repair_unit and repair_schedule tables can be migrated over. ref: - #99 - #94 - #99 (comment) * Simplify the creation of repair runs and their segments. Repair runs and their segments are one unit of work in concept and the persistence layer should be designed accordingly. Previous they were separated because the concern of sequence generation for IDs were exposed in the code. This is now encapsulated within storage implementations. This work allows the CassandraStorage to implement segments as clustering keys within the repair_run table. ref: - #94 - #101 * In CassandraStorage implement segments as clustering keys within the repair_run table. Change required in IStorage so to identify a segment both by runId and segmentId. ref: - #94 - #102 * Fix number of parallel repair computation Downgrade to Dropwizard 1.0.7 and Guava 19.0 to fix dependency issues Make repair manager schedule cycle configurable (was 30s hardcoded) ref: #108 * In CassandraStorage replace the table scan on `repair_run` with a async break-down of per cluster run-throughs of known run IDs. ref: #105

Repair runs and their segments are one unit of work in concept and the persistence layer should be designed accordingly. Previous they were separated because the concern of sequence generation for IDs were exposed in the code. This is now encapsulated within storage implementations. This work allows the CassandraStorage to implement segments as clustering keys within the repair_run table. ref: - #94 - #101

* Cassandra performance: Replace sequence ids with time-based UUIDs Makes the schema changes in a separate migration step, so that data in the repair_unit and repair_schedule tables can be migrated over. ref: - #99 - #94 - #99 (comment) * Simplify the creation of repair runs and their segments. Repair runs and their segments are one unit of work in concept and the persistence layer should be designed accordingly. Previous they were separated because the concern of sequence generation for IDs were exposed in the code. This is now encapsulated within storage implementations. This work allows the CassandraStorage to implement segments as clustering keys within the repair_run table. ref: - #94 - #101 * In CassandraStorage implement segments as clustering keys within the repair_run table. Change required in IStorage so to identify a segment both by runId and segmentId. ref: - #94 - #102 * Fix number of parallel repair computation Downgrade to Dropwizard 1.0.7 and Guava 19.0 to fix dependency issues Make repair manager schedule cycle configurable (was 30s hardcoded) ref: #108 * In CassandraStorage replace the table scan on `repair_run` with a async break-down of per cluster run-throughs of known run IDs. ref: #105

michaelsembwever requested a review from adejanovski May 12, 2017 07:08

michaelsembwever force-pushed the mck/simplify-creating-repair-run-and-segments branch from 4084359 to 2a778bd Compare May 13, 2017 04:59

adejanovski requested changes May 24, 2017

View reviewed changes

michaelsembwever force-pushed the mck/replace-sequence-ids-with-time-uuids branch from 58c70c7 to f5291d9 Compare May 28, 2017 05:23

michaelsembwever force-pushed the mck/simplify-creating-repair-run-and-segments branch from 2a778bd to 25a3f9f Compare May 28, 2017 05:24

michaelsembwever force-pushed the mck/simplify-creating-repair-run-and-segments branch from 79eae1c to 25a3f9f Compare May 29, 2017 09:04

michaelsembwever mentioned this pull request May 29, 2017

In CassandraStorage implement segments as clustering keys within the repair_run table #102

Merged

adejanovski force-pushed the mck/replace-sequence-ids-with-time-uuids branch from f5291d9 to 4b430cb Compare May 29, 2017 13:22

adejanovski changed the base branch from mck/replace-sequence-ids-with-time-uuids to mck/cassandra-improvements-94 May 29, 2017 13:55

michaelsembwever and others added 4 commits May 29, 2017 16:29

Cassandra performance: Replace sequence ids with time-based UUIDs

f4792a1

ref: - #99 - #94

SQUASH ME

9745740

Makes the schema changes in a separate migration step, so that data in the repair_unit and repair_schedule tables can be migrated over. ref: #99 (comment)

Fix file names and 002 migration file

33be3fa

adejanovski force-pushed the mck/simplify-creating-repair-run-and-segments branch from 25a3f9f to 33be3fa Compare May 29, 2017 14:51

adejanovski approved these changes May 29, 2017

View reviewed changes

tlpsonarqube reviewed May 29, 2017

View reviewed changes

adejanovski merged commit e303141 into mck/cassandra-improvements-94 May 29, 2017

michaelsembwever deleted the mck/simplify-creating-repair-run-and-segments branch June 1, 2017 12:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

simplify creating repair run and segments #101

simplify creating repair run and segments #101

michaelsembwever commented May 12, 2017

tlpsonarqube commented May 13, 2017

adejanovski commented May 24, 2017

adejanovski left a comment

michaelsembwever commented May 24, 2017

tlpsonarqube May 29, 2017

simplify creating repair run and segments #101

simplify creating repair run and segments #101

Conversation

michaelsembwever commented May 12, 2017

tlpsonarqube commented May 13, 2017

adejanovski commented May 24, 2017

adejanovski left a comment

Choose a reason for hiding this comment

michaelsembwever commented May 24, 2017

tlpsonarqube May 29, 2017

Choose a reason for hiding this comment