Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Activator: Retry on Get Revision error #1573

Closed
akyyy opened this issue Jul 11, 2018 · 11 comments

Comments

@akyyy
Copy link
Contributor

@akyyy akyyy commented Jul 11, 2018

/area API
/area autoscale

/kind bug

Expected Behavior

Get revision calls have transient errors. #1558. Activator should deal with that.

Actual Behavior

Activator uses Get revision api as one time shot, sometimes it fails with this error in the log:
Unable to get revision: Get https://10.35.240.1:443/apis/serving.knative.dev/v1alpha1/namespaces/default/revisions/configuration-example-00001: unexpected EOF
In this case, a potential reason is master is down.

Steps to Reproduce the Problem

This usually doesn't happen.

@mattmoor

This comment has been minimized.

Copy link
Member

@mattmoor mattmoor commented Nov 8, 2018

We should consider replacing the activator's direct client usage with something informer based. This is only peripherally related to this issue, but is probably how the only real action I'd take on our side to fix this.

The activator and activation in general is at the mercy of the availability of the K8s control plane (I've been meaning to write up an issue on this for a while).

@mattmoor

This comment has been minimized.

Copy link
Member

@mattmoor mattmoor commented Nov 8, 2018

cc @dgerd

@dgerd

This comment has been minimized.

Copy link
Contributor

@dgerd dgerd commented Nov 12, 2018

/assign @dgerd

I am going to take a stab at replacing the client usage with an informer.

@dgerd

This comment has been minimized.

Copy link
Contributor

@dgerd dgerd commented Dec 21, 2018

/unassign

I have not made progress on this. Going to unassign for now.

@vvraskin

This comment has been minimized.

Copy link
Member

@vvraskin vvraskin commented Jan 4, 2019

/assign
I'm going to follow up on this one once the #2653 gets in.

fyi @dgerd

@dgerd

This comment has been minimized.

Copy link
Contributor

@dgerd dgerd commented Jan 4, 2019

Thanks for picking this up. Let me know when you need another look at #2653.

@mattmoor

This comment has been minimized.

Copy link
Member

@mattmoor mattmoor commented Jan 8, 2019

@vvraskin I'm going to put this into 0.4 then

/milestone Serving 0.4

@mattmoor

This comment has been minimized.

Copy link
Member

@mattmoor mattmoor commented Feb 14, 2019

Moving the work left to 0.5 so that we can bake alongside a new e2e test.

@mattmoor mattmoor modified the milestones: Serving 0.4, Serving 0.5 Feb 14, 2019
@yanweiguo

This comment has been minimized.

Copy link
Contributor

@yanweiguo yanweiguo commented Mar 18, 2019

Hi @vvraskin, do you have any update for this one? Thanks.

@mattmoor mattmoor modified the milestones: Serving 0.5, Serving 0.6 Mar 19, 2019
@vvraskin

This comment has been minimized.

Copy link
Member

@vvraskin vvraskin commented Mar 22, 2019

Unfortunately I don't have cycles for this one in the next weeks.
/unassign

@markusthoemmes

This comment has been minimized.

Copy link
Contributor

@markusthoemmes markusthoemmes commented Mar 22, 2019

Gonna pick this up

/assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.