Bug 1821788:libvirt: Bump bootstrap memory to 5G for ppc64le #3396

Prashanth684 · 2020-04-02T13:31:09Z

On ppc64le there were OOM kills being observed during the bootstrap process
because of insufficient memory and bumping the memory seemed to solve the problem.
The libvirt defaults for the master and worker memory are 7G and 5G respectively,
so setting the boostrap default to 5G.

abhinavdahiya · 2020-04-02T14:12:27Z

Do you have examples or error logs for these?

Prashanth684 · 2020-04-02T15:01:29Z

Do you have examples or error logs for these?

journalctl on the bootstrap:

Apr 02 14:50:36 test1-24bm8-bootstrap kernel: hyperkube invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=-999
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: hyperkube cpuset=/ mems_allowed=0
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: CPU: 1 PID: 1888 Comm: hyperkube Not tainted 4.18.0-147.5.1.el8_1.ppc64le #1
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: Call Trace:
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: [c0000000556a7680] [c000000000d1e21c] dump_stack+0xb0/0xf4 (unreliable)
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: [c0000000556a76c0] [c00000000039e6c0] dump_header+0x80/0x390
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: [c0000000556a7780] [c00000000039eb0c] oom_kill_process+0x13c/0x210
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: [c0000000556a77c0] [c00000000039fce0] out_of_memory+0x200/0x7f0
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: [c0000000556a7860] [c0000000003af03c] __alloc_pages_nodemask+0x10ac/0x1150
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: [c0000000556a7a50] [c00000000045c858] alloc_pages_current+0xb8/0x220
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: [c0000000556a7a90] [c000000000399fac] filemap_fault+0x62c/0x990
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: [c0000000556a7b10] [c008000001860598] __xfs_filemap_fault+0xf0/0x330 [xfs]
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: [c0000000556a7b70] [c00000000040c294] __do_fault+0x44/0x1c0
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: [c0000000556a7bb0] [c0000000004149d8] do_fault+0x238/0x950
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: [c0000000556a7c00] [c000000000417e64] __handle_mm_fault+0x344/0xd70
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: [c0000000556a7ce0] [c0000000004189d0] handle_mm_fault+0x140/0x340
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: [c0000000556a7d20] [c00000000007aa54] __do_page_fault+0x244/0xc20
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: [c0000000556a7df0] [c00000000007b468] do_page_fault+0x38/0xd0
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: [c0000000556a7e30] [c00000000000a704] handle_page_fault+0x18/0x38
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: Mem-Info:
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: active_anon:23667 inactive_anon:134 isolated_anon:0
                                               active_file:8 inactive_file:0 isolated_file:0
                                               unevictable:0 dirty:0 writeback:0 unstable:0
                                               slab_reclaimable:1242 slab_unreclaimable:4611
                                               mapped:23 shmem:329 pagetables:62 bounce:0
                                               free:1542 free_pcp:0 free_cma:0
Apr 02 14:50:36 test1-24bm8-bootstrap kernel: Node 0 active_anon:1514688kB inactive_anon:8576kB active_file:512kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:1472kB dirty:0kB writeback:0kB shmem:21056kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no

clnperez · 2020-04-02T15:58:59Z

Thanks @Prashanth684 !

abhinavdahiya · 2020-04-02T16:11:48Z

#3396 (comment)

Thanks for that!

next do we know exactly which process(es) are involved in reaching OOMKilled state.

my hesitation in bumping this comes from the fact, that we control exactly what processes run on bootstrap-host... and we should be able to fix or fine tune to fit in required size.
also can you explain the reason for 2 -> 5 GB change. since libvirt runs on user machines, this kind of jump is very difficult.

Prashanth684 · 2020-04-02T17:43:48Z

#3396 (comment)

Thanks for that!

next do we know exactly which process(es) are involved in reaching OOMKilled state.

This happens after etcd is up and the cluster-bootstrap starts. Around the time that the hyperkube process gets killed this is the ps output:

[core@test1-ldtcw-bootstrap ~]$ ps aux --sort=-%mem | awk 'NR<=10{print $0}'
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        6014 59.5 47.2 1586752 974784 ?      Ssl  17:31   3:01 hyperkube kube-apiserver --openshift-config=/etc/kubernetes/config/kube-apiserver-config.yaml --logtostderr=false --alsologtostderr --log-file=/var/log/bootstrap-control-plane/kube-apiserver.log
root        1836  2.8  3.1 1303552 64704 ?       Ssl  17:23   0:21 /usr/bin/crio --enable-metrics=true --metrics-port=9537
root        6828  5.8  2.8 793216 58368 ?        Ssl  17:35   0:03 hyperkube kube-controller-manager --openshift-config=/etc/kubernetes/config/kube-controller-manager-config.yaml --kubeconfig=/etc/kubernetes/secrets/kubeconfig --logtostderr=false --alsologtostderr --log-file=/var/log/bootstrap-control-plane/kube-controller-manager.log
root        5470  0.6  2.0 916672 42048 ?        Sl   17:31   0:02 podman run --quiet --net=host --rm --volume /var/opt/openshift:/assets:z --volume /etc/kubernetes:/etc/kubernetes:z quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75ca476849edee259b340ce8711b2d78a7115b5289fd5651b68559cd6f50d43d start --tear-down-early=false --asset-dir=/assets --required-pods=openshift-kube-apiserver/kube-apiserver,openshift-kube-scheduler/openshift-kube-scheduler,openshift-kube-controller-manager/kube-controller-manager,openshift-cluster-version/cluster-version-operator
root        1867  1.8  1.8 1475392 38208 ?       Ssl  17:23   0:13 /usr/bin/hyperkube kubelet --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --runtime-request-timeout=10m --pod-manifest-path=/etc/kubernetes/manifests --minimum-container-ttl-duration=6m0s --cluster-domain=cluster.local --cgroup-driver=systemd --serialize-image-pulls=false --v=2 --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec
rpcuser     1594  0.0  1.3  39616 27584 ?        Ss   17:23   0:00 /usr/sbin/rpc.statd
root        6766  3.3  1.2 735680 25152 ?        Ssl  17:35   0:01 hyperkube kube-scheduler --kubeconfig=/etc/kubernetes/secrets/kubeconfig --leader-elect=true --cert-dir=/var/run/kubernetes --port=0 --authentication-kubeconfig=/etc/kubernetes/secrets/kubeconfig --authorization-kubeconfig=/etc/kubernetes/secrets/kubeconfig --logtostderr=false --alsologtostderr --log-file=/var/log/bootstrap-control-plane/kube-scheduler.log
root        6937  1.9  1.1 142208 23168 ?        Ssl  17:35   0:00 /usr/bin/cluster-version-operator start --release-image=registry.svc.ci.openshift.org/ocp-ppc64le/release-ppc64le@sha256:46b598ee4a6405d187610160c6874bea93bee36161f466b29c44d374f1745992 --enable-auto-update=false --enable-default-cluster-version=false --v=4 --kubeconfig=/etc/kubernetes/kubeconfig
core        1680  0.0  1.0  63232 22656 ?        S    17:23   0:00 /usr/bin/podman

my hesitation in bumping this comes from the fact, that we control exactly what processes run on bootstrap-host... and we should be able to fix or fine tune to fit in required size.

also can you explain the reason for 2 -> 5 GB change. since libvirt runs on user machines, this kind of jump is very difficult.

Talking to @zeenix , the defaults for the master is 7G and worker node is 5G and the suggestion was to change the bootstrap memory to match that to keep it on the lower side.

I completely understand the concern and also given that this issue only happens on ppc64le, I have asked the IBM team to see if there are any parameters they can tune to improve the performance. In the interim is there at least an option to configure the memory for bootstrap through some env variable or such or should this PR address that rather than making this change?

abhinavdahiya · 2020-04-02T17:47:42Z

@smarterclayton @deads2k the kube-apiserver on the bootstrap host is comsuming a lot of memory..

any suggestions as to how to make sure we can fit everything in 2 Gigs? maybe modify the cache sizes on the bootstrap kube-apiserver??

Prashanth684 · 2020-04-03T19:58:59Z

@abhinavdahiya Instead of hardcoding this increase across the board, would it be better to have a "terraform overrides" asset which would read terraform overrides files and apply it when the cluster is created. I played around with something like that here: https://gist.github.com/Prashanth684/c52737c522b379edb5cf154d859315e4

We could even make this specific to libvirt if there are concerns that users would muck around with terraform . This would allow us to just drop in a file in the installation directory inside a tf folder which contains something like:

module "bootstrap" {
  bootstrap_memory = "4096"
}

Thoughts?

Prashanth684 · 2020-04-06T14:37:01Z

Updated PR - have a bootstrap variable and bump the memory specifically for ppc64le based on the control plane arch on suggestion from @crawford

data/data/libvirt/bootstrap/main.tf

data/data/libvirt/bootstrap/variables.tf

pkg/tfvars/libvirt/libvirt.go

On ppc64le there were OOM kills being observed during the bootstrap process because of insufficient memory and bumping the memory seemed to solve the problem. The libvirt defaults for the master and worker memory are 7G and 5G respectively, so setting the boostrap default to 5G for ppc64le. ppc64le uses 64K pages rather than the default 4K page size and thus requires more memory.

crawford · 2020-04-07T00:20:27Z

High-level approach looks good to me.

Prashanth684 · 2020-04-07T15:48:42Z

/retitle Bug 1820219:libvirt: Bump bootstrap memory to 5G for ppc64le

openshift-ci-robot · 2020-04-07T15:49:08Z

@Prashanth684: This pull request references Bugzilla bug 1820219, which is invalid:

expected the bug to target the "4.5.0" release, but it targets "4.4.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1820219:libvirt: Bump bootstrap memory to 5G for ppc64le

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Prashanth684 · 2020-04-07T15:55:19Z

/retitle Bug 1821788:libvirt: Bump bootstrap memory to 5G for ppc64le

openshift-ci-robot · 2020-04-07T15:59:57Z

@Prashanth684: This pull request references Bugzilla bug 1821788, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.5.0) matches configured target release for branch (4.5.0)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1821788:libvirt: Bump bootstrap memory to 5G for ppc64le

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Prashanth684 · 2020-04-07T19:57:45Z

Tested on x86 and ppc64le. On ppc64le this is how the terraform libvirt variables file looks:

{
  "libvirt_uri": "qemu+tcp://192.168.122.1/system",
  "os_image": "/root/.cache/openshift-installer/image_cache/fb6c0f80e08c56bb158157e1823ec677",
  "libvirt_network_if": "tt0",
  "libvirt_master_ips": [
    "192.168.126.11",
    "192.168.126.12",
    "192.168.126.13"
  ],
  "libvirt_bootstrap_ip": "192.168.126.10",
  "libvirt_master_memory": "7168",
  "libvirt_master_vcpu": "4",
  "libvirt_bootstrap_memory": 5120
}

And this is a snippet of the tfstate file which has the profiles for the machines:

{
  "module": "module.bootstrap",
  ...
  "fw_cfg_name": "opt/com.coreos/config",
  "graphics": [],
  "id": "aa78614b-0087-440f-83ea-1fdf8e08d7ed",
  "initrd": "",
  "kernel": "",
  "machine": "pseries",
  "memory": 5120,
  ...
}

And on x86:

{
  "libvirt_uri": "qemu+tcp://192.168.122.1/system",
  "os_image": "/root/.cache/openshift-installer/image_cache/7478321162862330985a4516fe6798c7",
  "libvirt_network_if": "tt0",
  "libvirt_master_ips": [
    "192.168.126.11",
    "192.168.126.12",
    "192.168.126.13"
  ],
  "libvirt_bootstrap_ip": "192.168.126.10",
  "libvirt_master_memory": "7168",
  "libvirt_master_vcpu": "4"
}

and the tfstate file:

{
  "module": "module.bootstrap",
  ...   
  "fw_cfg_name": "opt/com.coreos/config",
  "graphics": [],
  "id": "e937e701-689f-40f8-80f8-ee63f2095c25",
  "initrd": "",
  "kernel": "",
  "machine": "pc",
  "memory": 2048,
  ...
}

abhinavdahiya · 2020-04-08T01:24:01Z

/lgtm
/approve

openshift-ci-robot · 2020-04-08T01:24:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [abhinavdahiya]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-04-08T01:41:32Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-04-08T02:07:34Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-04-08T02:20:32Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-04-08T02:33:32Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-04-08T02:40:54Z

@Prashanth684: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-scaleup-rhel7	`c57c680`	link	`/test e2e-aws-scaleup-rhel7`
ci/prow/e2e-libvirt	`c57c680`	link	`/test e2e-libvirt`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2020-04-08T02:59:32Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-04-08T03:05:13Z

@Prashanth684: All pull requests linked via external trackers have merged: openshift/installer#3396. Bugzilla bug 1821788 has been moved to the MODIFIED state.

In response to this:

Bug 1821788:libvirt: Bump bootstrap memory to 5G for ppc64le

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Prashanth684 · 2020-04-08T13:20:18Z

/cherry-pick release-4.4

openshift-cherrypick-robot · 2020-04-08T13:20:38Z

@Prashanth684: new pull request created: #3426

In response to this:

/cherry-pick release-4.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot requested review from fabianofranz and patrickdillon April 2, 2020 13:31

Prashanth684 force-pushed the libvirt-bootstrap branch from a541903 to 3d834a1 Compare April 2, 2020 13:34

Prashanth684 force-pushed the libvirt-bootstrap branch from 3d834a1 to f94d653 Compare April 6, 2020 14:34

Prashanth684 force-pushed the libvirt-bootstrap branch from f94d653 to 7494900 Compare April 6, 2020 14:55

abhinavdahiya reviewed Apr 6, 2020

View reviewed changes

data/data/libvirt/bootstrap/main.tf Outdated Show resolved Hide resolved

abhinavdahiya reviewed Apr 6, 2020

View reviewed changes

data/data/libvirt/bootstrap/variables.tf Outdated Show resolved Hide resolved

abhinavdahiya reviewed Apr 6, 2020

View reviewed changes

pkg/tfvars/libvirt/libvirt.go Outdated Show resolved Hide resolved

Prashanth684 force-pushed the libvirt-bootstrap branch from 7494900 to c57c680 Compare April 6, 2020 18:56

openshift-ci-robot changed the title ~~libvirt: Bump bootstrap memory to 5G~~ Bug 1820219:libvirt: Bump bootstrap memory to 5G for ppc64le Apr 7, 2020

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Apr 7, 2020

openshift-ci-robot changed the title ~~Bug 1820219:libvirt: Bump bootstrap memory to 5G for ppc64le~~ Bug 1821788:libvirt: Bump bootstrap memory to 5G for ppc64le Apr 7, 2020

openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Apr 7, 2020

openshift-ci-robot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Apr 7, 2020

openshift-ci-robot assigned abhinavdahiya Apr 8, 2020

openshift-ci-robot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Apr 8, 2020

openshift-merge-robot merged commit 32dd0f3 into openshift:master Apr 8, 2020

openshift-cherrypick-robot mentioned this pull request Apr 8, 2020

Bug 1820219: libvirt: Bump bootstrap memory to 5G for ppc64le #3426

Merged

Prashanth684 deleted the libvirt-bootstrap branch February 12, 2021 14:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug 1821788:libvirt: Bump bootstrap memory to 5G for ppc64le #3396

Bug 1821788:libvirt: Bump bootstrap memory to 5G for ppc64le #3396

Prashanth684 commented Apr 2, 2020 •

edited

abhinavdahiya commented Apr 2, 2020

Prashanth684 commented Apr 2, 2020

clnperez commented Apr 2, 2020

abhinavdahiya commented Apr 2, 2020

Prashanth684 commented Apr 2, 2020

abhinavdahiya commented Apr 2, 2020

Prashanth684 commented Apr 3, 2020

Prashanth684 commented Apr 6, 2020

crawford commented Apr 7, 2020

Prashanth684 commented Apr 7, 2020

openshift-ci-robot commented Apr 7, 2020

Prashanth684 commented Apr 7, 2020

openshift-ci-robot commented Apr 7, 2020

Prashanth684 commented Apr 7, 2020

abhinavdahiya commented Apr 8, 2020

openshift-ci-robot commented Apr 8, 2020

openshift-bot commented Apr 8, 2020

openshift-bot commented Apr 8, 2020

openshift-bot commented Apr 8, 2020

openshift-bot commented Apr 8, 2020

openshift-ci-robot commented Apr 8, 2020 •

edited

openshift-bot commented Apr 8, 2020

openshift-ci-robot commented Apr 8, 2020

Prashanth684 commented Apr 8, 2020

openshift-cherrypick-robot commented Apr 8, 2020

Bug 1821788:libvirt: Bump bootstrap memory to 5G for ppc64le #3396

Bug 1821788:libvirt: Bump bootstrap memory to 5G for ppc64le #3396

Conversation

Prashanth684 commented Apr 2, 2020 • edited

abhinavdahiya commented Apr 2, 2020

Prashanth684 commented Apr 2, 2020

clnperez commented Apr 2, 2020

abhinavdahiya commented Apr 2, 2020

Prashanth684 commented Apr 2, 2020

abhinavdahiya commented Apr 2, 2020

Prashanth684 commented Apr 3, 2020

Prashanth684 commented Apr 6, 2020

crawford commented Apr 7, 2020

Prashanth684 commented Apr 7, 2020

openshift-ci-robot commented Apr 7, 2020

Prashanth684 commented Apr 7, 2020

openshift-ci-robot commented Apr 7, 2020

Prashanth684 commented Apr 7, 2020

abhinavdahiya commented Apr 8, 2020

openshift-ci-robot commented Apr 8, 2020

openshift-bot commented Apr 8, 2020

openshift-bot commented Apr 8, 2020

openshift-bot commented Apr 8, 2020

openshift-bot commented Apr 8, 2020

openshift-ci-robot commented Apr 8, 2020 • edited

openshift-bot commented Apr 8, 2020

openshift-ci-robot commented Apr 8, 2020

Prashanth684 commented Apr 8, 2020

openshift-cherrypick-robot commented Apr 8, 2020

Prashanth684 commented Apr 2, 2020 •

edited

openshift-ci-robot commented Apr 8, 2020 •

edited