Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add force option and warning for overlapping repair schedules #1099

Merged
merged 3 commits into from
Jul 13, 2021

Conversation

emerkle826
Copy link
Collaborator

This PR adds a "force" option to the add repair schedule API to allow for creating overlapping schedules, even though it's not recommended (and broken for earlier versions of Cassandra).

Fixes #1078

@codecov
Copy link

codecov bot commented Jul 1, 2021

Codecov Report

Merging #1099 (a0fa289) into master (b3e30d7) will decrease coverage by 4.25%.
The diff coverage is 49.38%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1099      +/-   ##
==========================================
- Coverage   74.41%   70.16%   -4.26%     
==========================================
  Files         133      133              
  Lines        9724     9786      +62     
  Branches     1004     1018      +14     
==========================================
- Hits         7236     6866     -370     
- Misses       1895     2324     +429     
- Partials      593      596       +3     
Impacted Files Coverage Δ
.../io/cassandrareaper/service/RepairUnitService.java 50.84% <11.62%> (-24.48%) ⬇️
...o/cassandrareaper/resources/RepairRunResource.java 49.55% <83.33%> (-1.51%) ⬇️
...cassandrareaper/service/RepairScheduleService.java 92.10% <90.90%> (+2.81%) ⬆️
...sandrareaper/resources/RepairScheduleResource.java 52.65% <95.00%> (+0.16%) ⬆️
...assandrareaper/service/ClusterRepairScheduler.java 84.44% <100.00%> (ø)
...n/java/io/cassandrareaper/jmx/CompactionProxy.java 37.83% <0.00%> (-35.14%) ⬇️
.../main/java/io/cassandrareaper/ReaperException.java 66.66% <0.00%> (-33.34%) ⬇️
...io/cassandrareaper/jmx/HostConnectionCounters.java 50.00% <0.00%> (-29.42%) ⬇️
...ain/java/io/cassandrareaper/jmx/ClusterFacade.java 68.57% <0.00%> (-15.00%) ⬇️
...r/src/main/java/io/cassandrareaper/AppContext.java 64.28% <0.00%> (-14.29%) ⬇️
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b3e30d7...a0fa289. Read the comment docs.

Copy link
Contributor

@adejanovski adejanovski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work @emerkle826! I love that you went the extra mile and handled the React changes 👏

There are still some combinations that don't work as expected. If I create a full repair schedule on the whole keyspace, and then create an incremental schedule on a specific table of that keyspace, after clicking "Force" I'm getting the following exception in the log:

ERROR  [2021-07-02 10:01:25,562] [dw-206 - POST /repair_schedule?clusterName=3-11-6&keyspace=reaper_db&owner=Alex&scheduleTriggerTime=2021-07-02T07%3A49&scheduleDaysBetween=1&tables=running_reapers&repairParallelism=PARALLEL&incrementalRepair=true&repairThreadCount=1&force=true] i.d.j.e.LoggingExceptionMapper - Error handling a request: b12d341e7cad467c
java.lang.IllegalArgumentException: unit conflicts with existing in 3-11-6:reaper_db
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:135)
	at io.cassandrareaper.service.RepairUnitService.createRepairUnit(RepairUnitService.java:134)
	at io.cassandrareaper.service.RepairUnitService.getOrCreateRepairUnit(RepairUnitService.java:72)
	at io.cassandrareaper.resources.RepairScheduleResource.addRepairSchedule(RepairScheduleResource.java:275)
	at io.cassandrareaper.resources.RepairScheduleResource.addRepairSchedule(RepairScheduleResource.java:215)
	at sun.reflect.GeneratedMethodAccessor48.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)

In the comments I also mention a javascript exception that's triggering when schedule responses that are undefined.

One last issue is that forcing the creation will allow to create multiple fully identical schedules:
Capture d’écran 2021-07-02 à 10 08 42

<p>For Cassandra 4.0 and later, you can force creating this schedule by clicking the "Force" button below.</p>
</Modal.Body>
<Modal.Footer>
<Button variant="secondary" onClick={this._onClose}>Close</Button>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this button be named "Cancel" instead of "Close"? It would make it more obvious that you're canceling the creation.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do

r => this.setState({addRepairResultMsg: r.responseText})
r => this.setState({
addRepairResultMsg: null,
addRepairResultStatus: r.status,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I try to overwrite a schedule with the exact same one, it seems like r is undefined which triggers an exception:
Capture d’écran 2021-07-02 à 09 58 30
We may need a null check here.

Copy link
Collaborator Author

@emerkle826 emerkle826 Jul 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

r.status was something I was playing with in testing, but didn't end up using. I need to just remove it here.
I'm losing my mind. I do want status as I'm checking for 409. I'll fix this.

@jdonenine jdonenine changed the title Add force option and warning for overlapping repair schedules Add force option and warning for overlapping repair schedules [K8SSAND-474] Jul 2, 2021
@emerkle826
Copy link
Collaborator Author

@adejanovski I'd appreciate another look if you have the time

@emerkle826 emerkle826 force-pushed the overlap-warn branch 3 times, most recently from 79a9333 to 9c786d6 Compare July 9, 2021 15:32
Copy link
Contributor

@adejanovski adejanovski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work @emerkle826 !

With 4.0 coming, this is going to be very helpful.

@michaelsembwever michaelsembwever changed the title Add force option and warning for overlapping repair schedules [K8SSAND-474] Add force option and warning for overlapping repair schedules Jul 10, 2021
@michaelsembwever
Copy link
Member

Removed "K8SSAND-474" from title, as it is downstream.

Please keep all Reaper collaboration in the project.

@jdonenine jdonenine merged commit 5a2c27d into thelastpickle:master Jul 13, 2021
@emerkle826 emerkle826 deleted the overlap-warn branch July 13, 2021 15:01
adejanovski pushed a commit that referenced this pull request Aug 30, 2021
* Add force option and warning for overlapping repair schedules

* [fixup] PR review comments

* add tests for force option
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Warn if repairs overlap on tables using different settings
4 participants