Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix concurrent mod exception on shutting down #2507

Merged
merged 4 commits into from Mar 6, 2019

Conversation

@big-andy-coates
Copy link
Member

commented Mar 1, 2019

Description

This PR fixes a potential concurrent modification exception when shutting down the RESTful server. The exception is thrown when shutting down the command runner because the command runner thread is still active and can be using the consumer. Attempting to wakeup/close the consumer results in a concurrent mod exception if it is being used from the command runner thread.

Fixes #2339

Reviewing notes:

  • the command runner thread is now managed by the command runner, so that it can ensure things shutdown in the right order to avoid concurrent mod exceptions.
  • to allow things to shutdown I've changed the CommandQueue.getNewCommands to take a timeout. This allows control to pass back to the command runner's internal thread, allowing it to check if it should shutdown.
  • added a couple of 'early outs' at various points in the processing & retry logic to ensure timely exiting of the thread.

Testing done

Added more unit testing around shutdown.

Reviewer checklist

  • Ensure docs are updated if necessary. (eg. if a user visible feature is being added or changed).
  • Ensure relevant issues are linked (description should include text like "Fixes #")
Fix concurrent mod exception when shutting down rest app due to comma…
…nd topic thread still being active when trying to shutdown the command topic.

Fixes #2339

@big-andy-coates big-andy-coates requested a review from confluentinc/ksql as a code owner Mar 1, 2019

@rodesai

rodesai approved these changes Mar 4, 2019

Copy link
Contributor

left a comment

LGTM, couple of small things inline

@@ -126,13 +138,45 @@ public void processPriorCommands() {
ksqlEngine.getPersistentQueries().forEach(PersistentQueryMetadata::start);
}

void fetchAndRunCommands() {
final List<QueuedCommand> commands = commandStore.getNewCommands(NEW_CMDS_TIMEOUT);
if (commands.isEmpty()) {

This comment has been minimized.

Copy link
@rodesai

rodesai Mar 4, 2019

Contributor

nit: unnecessary check

This comment has been minimized.

Copy link
@big-andy-coates

big-andy-coates Mar 5, 2019

Author Member

Well, it avoids an unnecessary trace message which, given the getNewCommands message has a timeout and doesn't just block indefinitely, has some benefit. Though I admit, not much.

return;
try {
closed = true;
executor.awaitTermination(SHUTDOWN_TIMEOUT_MS, TimeUnit.MILLISECONDS);

This comment has been minimized.

Copy link
@rodesai

rodesai Mar 4, 2019

Contributor

It would be nice to do a wakeup of the consumer here before calling awaitTermination. Also, CommandStore.close has an unnecessary wakeup we can clean up.

This comment has been minimized.

Copy link
@big-andy-coates

big-andy-coates Mar 5, 2019

Author Member

Tis true. Done.

@big-andy-coates big-andy-coates requested a review from confluentinc/ksql Mar 5, 2019

@agavra

agavra approved these changes Mar 5, 2019

Copy link
Contributor

left a comment

LGTM :)

@agavra agavra requested a review from confluentinc/ksql Mar 5, 2019

@rodesai

rodesai approved these changes Mar 5, 2019

Copy link
Contributor

left a comment

LGTM!

big-andy-coates added some commits Mar 5, 2019

Merge from master
Conflicting files
ksql-rest-app/src/main/java/io/confluent/ksql/rest/server/computation/CommandRunner.java

@big-andy-coates big-andy-coates merged commit aa81293 into confluentinc:master Mar 6, 2019

1 check passed

continuous-integration/jenkins/pr-merge This commit looks good
Details

@big-andy-coates big-andy-coates deleted the big-andy-coates:command_topic branch Mar 6, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.