/kubernetes

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Heapster sizing adjustments #22940

Merged
merged 1 commit into from Mar 14, 2016

Conversation

Reviewers
No reviews
Assignees

@gmarek gmarek

Labels
Projects
None yet
Milestone
  v1.2
@mwielgus
Contributor

mwielgus commented Mar 14, 2016

Do not merge yet.

@mwielgus mwielgus added sig/autoscaling do-not-merge labels Mar 14, 2016

@mwielgus mwielgus assigned piosz Mar 14, 2016

@mwielgus mwielgus added this to the v1.2 milestone Mar 14, 2016

@googlebot googlebot added the cla: yes label Mar 14, 2016

@k8s-merge-robot k8s-merge-robot added the size/M label Mar 14, 2016

@k8s-merge-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

k8s-merge-robot Mar 14, 2016

Contributor

Labelling this PR as size/M
Contributor

k8s-merge-robot commented Mar 14, 2016

Labelling this PR as size/M

@piosz piosz assigned gmarek and unassigned piosz Mar 14, 2016

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Mar 14, 2016

GCE e2e build/test passed for commit 07d5e67.

k8s-bot commented Mar 14, 2016

GCE e2e build/test passed for commit 07d5e67.
Show outdated Hide outdated cluster/addons/cluster-monitoring/google/heapster-controller.yaml
@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Mar 14, 2016

GCE e2e build/test passed for commit 798f0bd.

k8s-bot commented Mar 14, 2016

GCE e2e build/test passed for commit 798f0bd.

@mwielgus mwielgus referenced this pull request Mar 14, 2016

Closed

Eventer has the same memory requirements as Metrics Heapster #22937

@davidopp

This comment has been minimized.

Show comment
Hide comment
@davidopp

davidopp Mar 14, 2016

Member

please ping this thread when you think it is ready to merge
Member

davidopp commented Mar 14, 2016

please ping this thread when you think it is ready to merge
cluster/addons/cluster-monitoring/googleinfluxdb/heapster-controller-combined.yaml
{% set num_nodes = pillar.get('num_nodes', -1) -%}
{% if num_nodes >= 0 -%}
{% set heapster_memory = (200 + num_nodes * 3)|string + "Mi" -%}
{% set metrics_memory = (200 + num_nodes * 4)|string + "Mi" -%}

This comment has been minimized.

Show comment
Hide comment
@vishh

vishh Mar 14, 2016

Member

Are we increasing the memory requirement for heapster pod as a whole?
Copy URL
@vishh

vishh Mar 14, 2016

Member

Are we increasing the memory requirement for heapster pod as a whole?

This comment has been minimized.

Show comment
Hide comment
@mwielgus

mwielgus Mar 14, 2016

Contributor

Yes. Just in case.
For 1000 node cluster we are able to operate without any sink at 2-2.3gb (with 30k pause and 2k system pods). GKE http output creation consumes +400MB. That brings us to 2.75. If couple events occur at once: scraping, gke scraping and pod relist then the temporary memory consumption can be bigger. So to be on the VERY safe side I decided to go with 4 multpilier.

As we did a lot of noise (=events) during the tests it became apparent that Eventer should also have more memory. List that happens inside of watch consumes LOTS of memory (we will try to get rid of it just after 1.2).
Copy URL
@mwielgus

mwielgus Mar 14, 2016

Contributor

Yes. Just in case.
For 1000 node cluster we are able to operate without any sink at 2-2.3gb (with 30k pause and 2k system pods). GKE http output creation consumes +400MB. That brings us to 2.75. If couple events occur at once: scraping, gke scraping and pod relist then the temporary memory consumption can be bigger. So to be on the VERY safe side I decided to go with 4 multpilier.

As we did a lot of noise (=events) during the tests it became apparent that Eventer should also have more memory. List that happens inside of watch consumes LOTS of memory (we will try to get rid of it just after 1.2).

@mwielgus mwielgus removed the do-not-merge label Mar 14, 2016

@mwielgus

This comment has been minimized.

Show comment
Hide comment
@mwielgus

mwielgus Mar 14, 2016

Contributor

Also bumping the image to beta2.
Contributor

mwielgus commented Mar 14, 2016

Also bumping the image to beta2.
@mwielgus
Heapster sizing adjustments
Loading status checks…
6123df9
@k8s-teamcity-mesosphere

This comment has been minimized.

Show comment
Hide comment
@k8s-teamcity-mesosphere

k8s-teamcity-mesosphere Mar 14, 2016

TeamCity OSS :: Kubernetes Mesos :: 4 - Smoke Tests Build 19062 outcome was SUCCESS
Summary: Tests passed: 1, ignored: 262 Build time: 00:05:55

k8s-teamcity-mesosphere commented on 6123df9 Mar 14, 2016

TeamCity OSS :: Kubernetes Mesos :: 4 - Smoke Tests Build 19062 outcome was SUCCESS
Summary: Tests passed: 1, ignored: 262 Build time: 00:05:55
@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Mar 14, 2016

GCE e2e build/test passed for commit f5f6a80.

k8s-bot commented Mar 14, 2016

GCE e2e build/test passed for commit f5f6a80.
@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Mar 14, 2016

GCE e2e build/test passed for commit 6123df9.

k8s-bot commented Mar 14, 2016

GCE e2e build/test passed for commit 6123df9.

@davidopp davidopp added lgtm priority/critical-urgent cherrypick-candidate labels Mar 14, 2016

@davidopp

This comment has been minimized.

Show comment
Hide comment
@davidopp

davidopp Mar 14, 2016

Member

LGTM
Member

davidopp commented Mar 14, 2016

LGTM
@k8s-merge-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

k8s-merge-robot Mar 14, 2016

Contributor

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]
Contributor

k8s-merge-robot commented Mar 14, 2016

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]
cluster/addons/cluster-monitoring/googleinfluxdb/heapster-controller-combined.yaml
{% set num_nodes = pillar.get('num_nodes', -1) -%}
{% if num_nodes >= 0 -%}
{% set heapster_memory = (200 + num_nodes * 3)|string + "Mi" -%}
{% set metrics_memory = (200 + num_nodes * 4)|string + "Mi" -%}
{% set eventer_memory = (200000 + num_nodes * 500)|string + "Ki" -%}

This comment has been minimized.

Show comment
Hide comment
@piosz

piosz Mar 14, 2016

Member

/s/200000/200*1024
Copy URL
@piosz

piosz Mar 14, 2016

Member

/s/200000/200*1024
@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Mar 14, 2016

GCE e2e build/test passed for commit 6123df9.

k8s-bot commented Mar 14, 2016

GCE e2e build/test passed for commit 6123df9.
@k8s-merge-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

k8s-merge-robot Mar 14, 2016

Contributor

Automatic merge from submit-queue
Contributor

k8s-merge-robot commented Mar 14, 2016

Automatic merge from submit-queue

k8s-merge-robot added a commit that referenced this pull request Mar 14, 2016

@k8s-merge-robot
Merge pull request #22940 from mwielgus/heapster-adjustments 
Auto commit by PR queue bot
f899a40

@k8s-merge-robot k8s-merge-robot merged commit f899a40 into kubernetes:master Mar 14, 2016
4 of 5 checks passed

4 of 5 checks passed

Submit Queue Github CI tests are not green.
Details
Jenkins GCE e2e 263 tests run, 107 skipped, 0 failed.
Details
Jenkins unit/integration 5808 tests run, 18 skipped, 0 failed.
Details
cla/google All necessary CLAs are signed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@bgrant0607

This comment has been minimized.

Show comment
Hide comment
@bgrant0607

bgrant0607 Mar 15, 2016

Member

PR description still says "do not merge yet".
Member

bgrant0607 commented Mar 15, 2016

PR description still says "do not merge yet".
@bgrant0607

This comment has been minimized.

Show comment
Hide comment
@bgrant0607

bgrant0607 Mar 15, 2016

Member

Not good for release-note generation
Member

bgrant0607 commented Mar 15, 2016

Not good for release-note generation

@bgrant0607 bgrant0607 added the cherry-pick-approved label Mar 15, 2016

@bgrant0607

This comment has been minimized.

Show comment
Hide comment
@bgrant0607

bgrant0607 Mar 15, 2016

Member

And please ensure all PRs have a description or at least a reference to an issue.
Member

bgrant0607 commented Mar 15, 2016

And please ensure all PRs have a description or at least a reference to an issue.

@andyzheng0831 andyzheng0831 referenced this pull request Mar 15, 2016

Merged

Trusty: Fix heapster breakage caused by #22940 #23005

eparis added a commit to eparis/kubernetes that referenced this pull request Mar 15, 2016

@k8s-merge-robot @eparis
Merge pull request #22940 from mwielgus/heapster-adjustments 
Auto commit by PR queue bot
(cherry picked from commit f899a40)
be2ad9e

@eparis eparis referenced this pull request Mar 15, 2016

Merged

Release 1.2 cherry pick batch #23006

@k8s-cherrypick-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-cherrypick-bot

k8s-cherrypick-bot Mar 15, 2016

Commit be2ad9e found in the "release-1.2" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this s an error find help to get your PR picked.

k8s-cherrypick-bot commented Mar 15, 2016

Commit be2ad9e found in the "release-1.2" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this s an error find help to get your PR picked.

@k8s-cherrypick-bot k8s-cherrypick-bot removed the cherrypick-candidate label Mar 15, 2016

@mwielgus

This comment has been minimized.

Show comment
Hide comment
@mwielgus

mwielgus Mar 15, 2016

Contributor

@bgrant0607 Please merge it to 1.2.
Contributor

mwielgus commented Mar 15, 2016

@bgrant0607 Please merge it to 1.2.

@eparis eparis referenced this pull request Mar 15, 2016

Merged

Release 1.2 cherrypick batch #23024

@piosz

This comment has been minimized.

Show comment
Hide comment
@piosz

piosz Mar 16, 2016

Member

@mwielgus this is already merged to 1.2 branch.
Member

piosz commented Mar 16, 2016

@mwielgus this is already merged to 1.2 branch.

vulpecula pushed a commit to vulpecula/kubernetes that referenced this pull request Apr 18, 2016

Andy Zheng Danka Niedzialkowska
Trusty: Fix heapster breakage caused by #22940
32e3444

shyamjvs pushed a commit to shyamjvs/kubernetes that referenced this pull request Dec 1, 2016

@k8s-merge-robot @eparis
Merge pull request #22940 from mwielgus/heapster-adjustments 
Auto commit by PR queue bot
(cherry picked from commit f899a40)
799fd34

shouhong pushed a commit to shouhong/kubernetes that referenced this pull request Feb 14, 2017

@k8s-merge-robot @eparis
Merge pull request #22940 from mwielgus/heapster-adjustments 
Auto commit by PR queue bot
(cherry picked from commit f899a40)
e679323
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment