Fix for agent hang when process cannot be killed #4074
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
merlynomsft
changed the title
merlynomsft
commented
Dec 2, 2022
DenisRumyantsev
approved these changes
Dec 7, 2022
There was a problem hiding this comment.
@merlynomsft thank you so much for your work!
I artificially reproduced the issue to test these changes.
The agent is working properly and the issue is resolved.
LiliaSabitova
approved these changes
Dec 7, 2022
kirill-ivlev
approved these changes
Dec 7, 2022
vmapetr
approved these changes
Dec 7, 2022
wojgasior
approved these changes
Dec 7, 2022
|
Please note currently this behavior is opt-in. To do so, set the following environment variable on the agent: VSTSAGENT_CONTINUE_AFTER_CANCEL_PROCESSTREEKILL_ATTEMPT: true |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes an issue where the agent will hang and fail to upload logs if the process executed cannot be killed during cancellation. The pipeline hangs until timeoutInMinutes + cancelTimeoutInMinutes
The log fails to upload and a blank log is shown in the log viewer UX:
Analysis
Killing process during cancellation is 'best effort' and may fail if, for some reason, the process cannot be killed. In that case, the agent waits forever in a loop for the process to exit. Fix is to exit the loop after attempting to kill the process.
Testing
This issue is not easy to reproduce, because it requires the agent to launch a process that cannot be killed
I simulated this condition by modifying ProcessInvoker to skip killing the process
**Knob / Enabling **
The behavior is disabled by default. This change is enabled by setting knob: VSTSAGENT_CONTINUE_AFTER_CANCEL_PROCESSTREEKILL_ATTEMPT
VSO 2009025