Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle suspend/resume error from watch stream read #2136

Merged
merged 2 commits into from
Feb 12, 2021
Merged

Conversation

jakolehm
Copy link
Contributor

Had to copy ReadableWebToNodeStream to here because we cannot catch read errors with the upstream version. We should probably contribute a fix to upstream.

Fixes #2133

Signed-off-by: Jari Kolehmainen <jari.kolehmainen@gmail.com>
@jakolehm jakolehm added bug Something isn't working blocker labels Feb 12, 2021
@jakolehm jakolehm added this to the 4.1.0 milestone Feb 12, 2021
@jakolehm jakolehm requested a review from a team February 12, 2021 08:25
@jakolehm jakolehm marked this pull request as draft February 12, 2021 08:33
Signed-off-by: Jari Kolehmainen <jari.kolehmainen@gmail.com>
@jakolehm jakolehm marked this pull request as ready for review February 12, 2021 08:45
Copy link
Contributor

@nevalla nevalla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jakolehm jakolehm requested review from ixrock and a team February 12, 2021 12:36
@@ -0,0 +1,87 @@
import { Readable } from "readable-stream";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
import { Readable } from "readable-stream";
import { Readable } from "stream";

This package is a mirror of the streams implementations in Node.js.

If you want to guarantee a stable streams base, regardless of what version of Node you, or the users of your libraries are using, use readable-stream only and avoid the "stream" module in Node-core, for background see this blogpost.

Since we bundle app with Node.js v12.x maybe we could use native stream module?
https://nodejs.org/dist/latest-v12.x/docs/api/stream.html#stream_readable_streams

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, I'll do that in a separate PR.

if (abortController.signal.aborted) return;
clearTimeout(timedRetry);
timedRetry = setTimeout(() => { // we did not get any kubernetes errors so let's retry
if (abortController.signal.aborted) return;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this give lines 379-381 above, if the signal has already aborted, shouldn't the watch just stop anyway?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, we don't need this.

Comment on lines +60 to +62
} catch(error) {
this.push(null); // Signal EOF
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the change? Why do we want to signal EOF on an error? Shouldn't we log these errors anyway?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We cannot even catch errors that are thrown from here. If we could I would not have copied this library to our codebase.

All other error cases end up with EOF so it makes sense to push this error to the similar path. I really would like to get this fix into upstream and that means this class itself won't be logging anything.

if (errorReceived) return; // kubernetes errors should be handled in a callback
["end", "close", "error"].forEach((eventName) => {
nodeStream.on(eventName, () => {
if (errorReceived) return; // kubernetes errors should be handled in a callback
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this comment is wrong. Shouldn't it read something like:

// kubernetes error status events should prevent auto retrying of the watch.

@jakolehm
Copy link
Contributor Author

@ixrock @Nokel81 I'll open a separate PR based on suggestions.

@jakolehm jakolehm merged commit 557d96d into master Feb 12, 2021
@jakolehm jakolehm deleted the fix/issue-2133 branch February 12, 2021 14:37
@jakolehm jakolehm mentioned this pull request Feb 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocker bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Kubernetes watch does not always retry after resume from sleep
4 participants