Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nautilus: common/config: respect POD_MEMORY_REQUEST *and* POD_MEMORY_LIMIT env vars #29562

Merged
merged 4 commits into from Aug 15, 2019

Conversation

@liewegas
Copy link
Member

commented Aug 8, 2019

liewegas and others added 4 commits Aug 6, 2019
common/config: do no include multiple 'default' values
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 712409c)
common/config: let diff show non-build defaults
Notably, this includes things from set_val_default(), which may be set
at runtime.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 77b634a)
common: add comment about pod memory requests/limits
Written in private correspondence by Travis Nielson.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 94bcc45)
common/config: respect POD_MEMORY_REQUEST *and* POD_MEMORY_LIMIT env …
…vars

If a kubernetes pod spec specifies a limit of X, then the pod gets both
the limits.memory and requests.memory resource fields set, and rook passes
those as POD_MEMORY_LIMIT and POD_MEMORY_REQUEST environment variables.

This is a problem if only the limit is set, because we will end up
setting our osd_memory_target (and, in the future, other *_memory_targets)
to the hard limit, and the daemon will inevitably reach that threshold
and get killed.

Fix this by also looking at the POD_MEMORY_LIMIT value, and applying the
ratio (default: .8) to it, and setting our actual target to the min of
that and the POD_MEMORY_REQUEST.

Also, set the "default" target to ratio*limit, so that it will apply in
general when no request is specified.

When both request and limit are 10M, we then see

        "osd_memory_target": {
            "default": "800000000000",
            "env": "800000000000",
            "final": "800000000000"
        },

In a more "normal" situation where limit is 10M and request is 5M, we get

        "osd_memory_target": {
            "default": "800000000000",
            "env": "500000000000",
            "final": "500000000000"
        },

If only limit is specified (to 10M), we get

        "osd_memory_target": {
            "default": "800000000000",
            "final": "800000000000"
        },

Fixes: https://tracker.ceph.com/issues/41037
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 25f28e6)

@liewegas liewegas added this to the nautilus milestone Aug 8, 2019

@liewegas liewegas changed the title common/config: respect POD_MEMORY_REQUEST *and* POD_MEMORY_LIMIT env vars nautilus: common/config: respect POD_MEMORY_REQUEST *and* POD_MEMORY_LIMIT env vars Aug 8, 2019

@yuriw

This comment has been minimized.

Copy link
Contributor

commented Aug 14, 2019

@yuriw yuriw merged commit bc18d44 into ceph:nautilus Aug 15, 2019

4 checks passed

Docs: build check OK - docs built
Details
Signed-off-by all commits in this PR are signed
Details
Unmodified Submodules submodules for project are unmodified
Details
make check make check succeeded
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.