Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PDE-3251 fix(core): abort logger connection at the end of invocation #562

Merged
merged 4 commits into from
Jul 14, 2022

Conversation

eliangcs
Copy link
Member

@eliangcs eliangcs commented Jul 8, 2022

Fixes the issue where a Lambda invocation can take ~1 minute before a socket hang up error with the log server shows up.

You can find the steps to reproduce the error on Slack.

I’m not sure under what situations the log server would stop responding to an open connection on production. (Maybe the log server sometimes crashes?) But we should at least make the core more resilient and not affected by a hanging connection.

The idea is to limit the wait time for the log server to respond. We use node-fetch to make requests to the log server. And to abort a node-fetch request, we can use AbortController. See node-fetch's README for example. Since AbortController only became natively supported on Node.js 15+, I'm using the node-abort-controller package.

@eliangcs eliangcs requested a review from benzapier July 8, 2022 09:11
@eliangcs eliangcs changed the title PDE-3151 fix(core): abort logger connection at the end of invocation PDE-3251 fix(core): abort logger connection at the end of invocation Jul 8, 2022
@zapzap
Copy link
Collaborator

zapzap commented Jul 8, 2022

I'm no Node expert so forgive me if I'm totally wrong here, but would we want to pass a value for the timeoutToAbort argument so that we don't kill off the connection immediately but give the server at least some time to close it for us?

Edit: this comment was from @robgolding!

@eliangcs
Copy link
Member Author

eliangcs commented Jul 8, 2022

@zapzap (not sure who you are) that makes sense. In the latest commits, apart from fixing a test, I've made the time limit default to 200 ms, so the server has a chance to close the connection for us.

err,
'http options:',
httpOptions
);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm removing httpOptions from the log because there were credentials in the AWS logs.

@eliangcs eliangcs merged commit 968691f into master Jul 14, 2022
@eliangcs eliangcs deleted the PDE-3151-abort-logger branch July 14, 2022 07:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants