Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkg: kubelet: remote: increase grpc client default size #63977

Merged
merged 1 commit into from May 19, 2018

Conversation

@runcom
Copy link
Member

runcom commented May 17, 2018

Signed-off-by: Antonio Murdaca runcom@redhat.com

What this PR does / why we need it:

when running lots and lots of containers and having tons of images on a given node, we started seeing this in the logs (with docker):

Unable to retrieve pods: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4208374 vs. 4194304)

That's because the grpc client is defaulting to a 4MB response size.
This patch increases the resp size to 8MB to avoid such issue.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Release note:

increase grpc client default response size
@runcom

This comment has been minimized.

Copy link
Member Author

runcom commented May 17, 2018

@runcom

This comment has been minimized.

Copy link
Member Author

runcom commented May 17, 2018

@mrunalp PTAL as well

pkg: kubelet: remote: increase grpc client default size
Signed-off-by: Antonio Murdaca <runcom@redhat.com>

@runcom runcom force-pushed the runcom:increase-grpc-resp-size branch from dd10ce6 to 57a2eec May 17, 2018

@sjenning

This comment has been minimized.

Copy link
Contributor

sjenning commented May 17, 2018

/lgtm

@dims

This comment has been minimized.

Copy link
Member

dims commented May 17, 2018

@runcom also see discussion in #63858 i had a WIP PR too which i can abandon

@resouer
Copy link
Member

resouer left a comment

/lgtm Thanks!

@mrunalp

This comment has been minimized.

Copy link
Contributor

mrunalp commented May 17, 2018

/lgtm

@ceshihao

This comment has been minimized.

Copy link
Contributor

ceshihao commented May 18, 2018

The same concern I commented in dims's PR #63894

I think it is OK as a temporary solution.
However, it can not resolve the problem finally if more and more message (size > 16M), and we have to increase it again.

Perhaps, it can have a final solution, like stream output message, result pagination or something else.

@derekwaynecarr

This comment has been minimized.

Copy link
Member

derekwaynecarr commented May 18, 2018

/test pull-kubernetes-kubemark-e2e-gce
/test pull-kubernetes-e2e-gce

@derekwaynecarr

This comment has been minimized.

Copy link
Member

derekwaynecarr commented May 18, 2018

/assign @derekwaynecarr

@derekwaynecarr

This comment has been minimized.

Copy link
Member

derekwaynecarr commented May 18, 2018

i am fine w/ this as a temporary solution.

/approve

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented May 18, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, mrunalp, resouer, runcom, sjenning

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sjenning

This comment has been minimized.

Copy link
Contributor

sjenning commented May 18, 2018

flake #63427
/retest

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented May 18, 2018

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

1 similar comment
@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented May 19, 2018

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

k8s-github-robot commented May 19, 2018

Automatic merge from submit-queue (batch tested with PRs 60012, 63692, 63977, 63960, 64008). If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-github-robot k8s-github-robot merged commit 4d786a9 into kubernetes:master May 19, 2018

18 checks passed

Submit Queue Queued to run github e2e tests a second time.
Details
cla/linuxfoundation runcom authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gke Skipped
pull-kubernetes-e2e-kops-aws Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped
pull-kubernetes-local-e2e-containerized Job succeeded.
Details
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details

@runcom runcom deleted the runcom:increase-grpc-resp-size branch May 19, 2018

@feiskyer
Copy link
Member

feiskyer left a comment

LGTM

k8s-github-robot pushed a commit that referenced this pull request May 21, 2018

Kubernetes Submit Queue
Merge pull request #63894 from dims/bump-grpc-max-message-size-for-do…
…cker-service

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Bump grpc max message size for docker service

**What this PR does / why we need it**:
When we have a lot of containers, we run into the limit in grpc ( https://github.com/grpc/grpc-go/blob/master/clientconn.go#L118 )

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #63858

**Special notes for your reviewer**:
In #63977 we fixed the send and receive sizes on the client side. we should fix the docker service too 

**Release note**:

```release-note
NONE
```

k8s-github-robot pushed a commit that referenced this pull request Jun 13, 2018

Kubernetes Submit Queue
Merge pull request #64804 from mcluseau/automated-cherry-pick-of-#63977-
#64672-upstream-release-1.10

Automatic merge from submit-queue.

Automated cherry pick of #63977: pkg: kubelet: remote: increase grpc client default size #64672: pkg: kubelet: remote: increase grpc client default size to

Cherry pick of #63977 #64672 on release-1.10.

#63977: pkg: kubelet: remote: increase grpc client default size
#64672: pkg: kubelet: remote: increase grpc client default size to

```release-note
CRI: increase grpc client default size to 16MB
```

mysunshine92 referenced this pull request in mysunshine92/k8s-study Aug 12, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.