Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix more deadlocks #226

Merged
merged 7 commits into from
May 8, 2018
Merged

Conversation

svanoort
Copy link
Member

@svanoort svanoort commented May 7, 2018

See deadlock file from JENKINS-51132 comment.

This prevents deadlocks by ensuring we do not lock Execution before Run (usually by separating the synchronization scopes).

Lock order should always be Run -> Execution when locks are nested, or locks should be separated (the approach generally used here).

Why aren't there testcases? I have low confidence in our ability produce a testcase that replicates all these deadlocks.

@svanoort svanoort requested review from jglick and abayer May 7, 2018 22:27
Copy link
Member

@abayer abayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just making sure I grok the general point here - the deadlock was a result of the methods themselves being synchronized and so juggling around the synchronization fixes things?

@svanoort
Copy link
Member Author

svanoort commented May 7, 2018

@abayer Yeah, we had methods in CpsFlowExecution that synchronize on first the FlowExecution, then call Run#save(), which synchronizes on the Run --- but in general the WorkflowRun will synchronize on itself and save itself (synchronizing on the Execution to lock it during save). Ergo deadlockdom.

@jglick
Copy link
Member

jglick commented May 8, 2018

I have low confidence in our ability produce a testcase that replicates all these deadlocks.

In general it is pretty fruitless to write automated tests for deadlocks. Possible, just not productive. You are not demonstrating that the code is free of deadlocks, only that the behavior of a particular sequence of events has changed for the better.

@@ -620,47 +620,52 @@ private synchronized String getHeadsAsString() {
* Bypasses {@link #croak(Throwable)} and {@link #onProgramEnd(Outcome)} to guarantee a clean path.
*/
@GuardedBy("this")
synchronized void createPlaceholderNodes(Throwable failureReason) throws Exception {
this.done = true;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreadable diff. better

end.addAction(new ErrorAction(failureReason));
head.setNewHead(end);
} catch (Exception ex) {
throw ex;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uh, what is this try-catch-throw accomplishing? Just delete it!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main reason is to separate out the synchronization components, but also try to ensure the run save happens.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, no. I'm going to blame jet lag and lack of sleep for this. You're right, nonsensical.

}
}
try {
saveOwner();
} catch (Exception ex) {
throw ex;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto, makes little sense

try {
this.execution.saveOwner();
} catch (Exception ex) {
LOGGER.log(Level.WARNING, "Error saving execution for "+this, ex);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Useless toString(). Try

LOGGER.log(Level.WARNING, "Error saving execution for " + execution, ex);

@@ -18,6 +18,7 @@
import org.codehaus.groovy.control.CompilationUnit;
import org.codehaus.groovy.control.CompilerConfiguration;
import org.codehaus.groovy.control.SourceUnit;
import org.jenkinsci.plugins.workflow.flow.FlowExecution;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unused import

@@ -1217,7 +1226,7 @@ public boolean isComplete() {
}

/**
* Record the end of the build.
* Record the end of the build. Note: we should always follow this with a call to saveOwner to persist the result.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@link

@svanoort svanoort merged commit 0318a5e into jenkinsci:master May 8, 2018
@svanoort svanoort deleted the fix-more-deadlocks branch May 8, 2018 02:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants