Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup artifact handlers hanging node process #1596

Merged
merged 3 commits into from
Dec 7, 2023

Conversation

robherley
Copy link
Contributor

Fixes a bug where an error response causes the actions to hang for ~2 min.

Using why-is-node-running we were able to track down the ZLIB, TCPWRAP, and TLSWRAP handlers clogging things up.

tl;dr

  1. ZIP stream was being created way too early, we encounter an error & throw (reject) and the stream just hangs around.
  2. In our retry http wrapper, we weren't reading the body on the request if it's a non-200 status code. For some reason (more investigation needed) this keeps things going until the keep-alive is met for the request? (or some other 2 minute timeout)

@robherley robherley requested a review from a team as a code owner December 6, 2023 23:46
Comment on lines +71 to +75
const zipUploadStream = await createZipUploadStream(
zipSpecification,
options?.compressionLevel
)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has just been moved down, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, we were initializing this wayyy above in the method before we were even consuming the stream

Comment on lines +84 to +86
debug(`[Response] - ${response.message.statusCode}`)
debug(`Headers: ${JSON.stringify(response.message.headers, null, 2)}`)
debug(`Body: ${body}`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we may need to remove these lines.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually added them to assist in debugging customer workflows. If they run into an issue, they can turn on debug logs and this can potentially help figure out what's wrong


if (this.isSuccessStatusCode(statusCode)) {
return response
return {response, body}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is changing the return type; are there any tests failing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is only used internally by the twirp client. However I do want to revisit this retry logic eventually and add more tests.

@@ -80,11 +80,13 @@ class ArtifactHttpClient implements Rpc {
try {
const response = await operation()
const statusCode = response.message.statusCode
debug(`[Response] ${response.message.statusCode}`)
debug(JSON.stringify(response.message.headers, null, 2))
const body = await response.readBody()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noice! that's it 👍

@jtamsut jtamsut self-requested a review December 7, 2023 00:24
@robherley robherley merged commit 43ccaf0 into main Dec 7, 2023
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants