Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix a rare race condition in treeaction timeout detection #38

Merged
merged 3 commits into from
Jan 26, 2021
Merged

fix a rare race condition in treeaction timeout detection #38

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3173bfb
fix a rare race condition in treeaction timeout detection
Jan 22, 2021
5ac3a39
WORK: adjust for Retry exhausted scenario
Jan 26, 2021
db140f4
WORK: update the comment
Jan 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 29 additions & 16 deletions Forge.TreeWalker/src/TreeWalkerSession.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -795,25 +795,21 @@ internal async Task PerformActionTypeBehavior(TreeNode treeNode, string treeNode
return;
}

break;
// Retries are exhausted. Throw ActionTimeoutException with executeAction exception as innerException.
throw new ActionTimeoutException(
string.Format(
"Action did not complete successfully with retry attempts exhausted. TreeNodeKey: {0}, TreeActionKey: {1}, ActionName: {2}, RetryCount: {3}, RetryPolicy: {4}",
treeNodeKey,
treeActionKey,
treeAction.Action,
retryCount,
retryPolicyType),
innerException);
}

// Break out early if we would hit timeout before next retry.
if (actionTimeout != -1 && stopwatch.ElapsedMilliseconds + waitTime.TotalMilliseconds >= actionTimeout)
{
// If the timeout is hit and the ContinuationOnTimeout flag is set, commit a new ActionResponse.
// with the status set to TimeoutOnAction and return.
if (treeAction.ContinuationOnTimeout)
{
ActionResponse timeoutResponse = new ActionResponse
{
Status = "TimeoutOnAction"
};

await this.CommitActionResponse(treeActionKey, timeoutResponse).ConfigureAwait(false);
return;
}

break;
}

Expand All @@ -823,15 +819,32 @@ internal async Task PerformActionTypeBehavior(TreeNode treeNode, string treeNode
maxRetryCount--;
}

if (actionTimeout != -1 && stopwatch.ElapsedMilliseconds + waitTime.TotalMilliseconds >= actionTimeout)
{
// If the timeout is hit and the ContinuationOnTimeout flag is set, commit a new ActionResponse.
// with the status set to TimeoutOnAction and return.
if (treeAction.ContinuationOnTimeout)
{
ActionResponse timeoutResponse = new ActionResponse
{
Status = "TimeoutOnAction"
};

await this.CommitActionResponse(treeActionKey, timeoutResponse).ConfigureAwait(false);
return;
}
}

// Retries are exhausted. Throw ActionTimeoutException with executeAction exception as innerException.
Copy link
Member

@TravisOnGit TravisOnGit Jan 26, 2021 •
edited by zhengmu
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retries are exhausted [](start = 15, length = 21)

Let's update this message as well since it should be hitting timeouts and not retries exhausted. #Resolved

throw new ActionTimeoutException(
string.Format(
"Action did not complete successfully. TreeNodeKey: {0}, TreeActionKey: {1}, ActionName: {2}, RetryCount: {3}, RetryPolicy: {4}",
"Action did not complete successfully with timeout reached. TreeNodeKey: {0}, TreeActionKey: {1}, ActionName: {2}, RetryCount: {3}, RetryPolicy: {4}, Timeout: {5}",
treeNodeKey,
treeActionKey,
treeAction.Action,
retryCount,
retryPolicyType),
retryPolicyType,
actionTimeout),
innerException);
}

Expand Down