Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ephemeralStorage request to -ctrl- pods #2294

Open
iMartyn opened this Issue Nov 26, 2018 · 6 comments

Comments

@iMartyn
Copy link

iMartyn commented Nov 26, 2018

Description

If a cluster is running low on space and supports ephemeral storage requests, because the -ctrl pods currently don't request any and output a log file, they become candidates for eviction. K8s has some high limits for this by default (15%) and so this can be unexpected.

Example event :

Status:             Failed
Reason:             Evicted
Message:            The node was low on resource: ephemeral-storage. Container maya-volume-exporter was using 52Ki, which exceeds its request of 0. Container pvc-12bf95df-e8b6-11e8-b3bb-0050569f015b-ctrl-con was using 520Ki, which exceeds its request of 0. 

Context

By adding requests of ephemeralStorage to the deployment for the -ctrl pods, even a low amount, OpenEBS storage will continue to work down to a much lower threshold of disk available.

Point to remember

Adding ephemeralStorage to requests/limits makes the deployment incompatible with earlier versions of kubernetes, so OpenEBS would have to verify the version of k8s before adding the requests.

@kmova kmova added this to the 0.8 milestone Nov 26, 2018

@AmitKumarDas

This comment has been minimized.

Copy link
Member

AmitKumarDas commented Nov 27, 2018

This PR openebs/maya#835 exposes installed Kubernetes Version as an option in cast config.
NOTE: Further changes will be required at CASTemplate's default config & RunTask's go templating to solve this issue.

@AmitKumarDas

This comment has been minimized.

Copy link
Member

AmitKumarDas commented Nov 28, 2018

Kubernetes server version value can vary a lot. This introduces a bit of complexity in comparing between two versions. I have updated the PR openebs/maya#835 as WIP to handle version comparison.

Some of valid versions can be:

v1.11.0
v1.11.0+a0ce1bf1
v1.9.7-gke.11
v1.10.3-eks
v1.13.0-alpha.0.3346+94c5953904f4f3

Challenges:

e.g.

CASTemplate.openebs.io "storage-pool-read-default-0.8.0" is invalid: metadata.labels: Invalid value: "1.9+": a valid label must be an empty string or consist of alphanumeric characters, '-', '' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9.]*)?[A-Za-z0-9])?')

@kmova

This comment has been minimized.

Copy link
Member

kmova commented Dec 6, 2018

The version information is available for Run Tasks. The next PR should use the version to set the options accordingly. Will be taken up for 0.8.1

@kmova kmova closed this Dec 6, 2018

@kmova kmova reopened this Dec 7, 2018

kmova added a commit to kmova/maya that referenced this issue Jan 16, 2019

fix(policies): support for resource requests on side-cars
Ref: openebs/openebs#2294

With 0.7.0, there was support for configuring resource
limits on the pool and volume pod side-cars using
AuxillaryResourceLimits. With this PR, added the support
to specify AuxillaryResourceRequests.

This feature is useful in cases where user has to specify
minimum requests like ephimeral storage etc. to avoid
erraneous eviction by K8s as described in the above issue.

Also, some of the resource types are very version dependent
so keeping this as generic and not providing any defaults in
this PR.

In the future, it is possible that OpenEBS Operator could
automatically determine the version and set default values
on the generated default Storage Class.

Signed-off-by: kmova <kiran.mova@openebs.io>

kmova added a commit to kmova/maya that referenced this issue Jan 16, 2019

fix(policies): support for resource requests on side-cars
Ref: openebs/openebs#2294

With 0.7.0, there was support for configuring resource
limits on the pool and volume pod side-cars using
AuxilaryResourceLimits. With this PR, added the support
to specify AuxilaryResourceRequests.

This feature is useful in cases where user has to specify
minimum requests like ephemeral storage etc. to avoid
erroneous eviction by K8s as described in the above issue.

Also, some of the resource types are very version dependent
so keeping this as generic and not providing any defaults in
this PR.

In the future, it is possible that OpenEBS Operator could
automatically determine the version and set default values
on the generated default Storage Class.

Signed-off-by: kmova <kiran.mova@openebs.io>

AmitKumarDas added a commit to openebs/maya that referenced this issue Jan 16, 2019

fix(policies): support for resource requests on side-cars (#889)
Ref: openebs/openebs#2294

With 0.7.0, there was support for configuring resource
limits on the pool and volume pod side-cars using
AuxilaryResourceLimits. With this PR, added the support
to specify AuxilaryResourceRequests.

This feature is useful in cases where user has to specify
minimum requests like ephemeral storage etc. to avoid
erroneous eviction by K8s as described in the above issue.

Also, some of the resource types are very version dependent
so keeping this as generic and not providing any defaults in
this PR.

In the future, it is possible that OpenEBS Operator could
automatically determine the version and set default values
on the generated default Storage Class.

Signed-off-by: kmova <kiran.mova@openebs.io>

kmova added a commit to kmova/maya that referenced this issue Jan 23, 2019

fix(policies): support for resource requests on side-cars (openebs#889)
Ref: openebs/openebs#2294

With 0.7.0, there was support for configuring resource
limits on the pool and volume pod side-cars using
AuxilaryResourceLimits. With this PR, added the support
to specify AuxilaryResourceRequests.

This feature is useful in cases where user has to specify
minimum requests like ephemeral storage etc. to avoid
erroneous eviction by K8s as described in the above issue.

Also, some of the resource types are very version dependent
so keeping this as generic and not providing any defaults in
this PR.

In the future, it is possible that OpenEBS Operator could
automatically determine the version and set default values
on the generated default Storage Class.

Signed-off-by: kmova <kiran.mova@openebs.io>
(cherry picked from commit 9f12a28)

vishnuitta added a commit to openebs/maya that referenced this issue Jan 25, 2019

fix(policies): support for resource requests on side-cars (#889) (#902)
Ref: openebs/openebs#2294

With 0.7.0, there was support for configuring resource
limits on the pool and volume pod side-cars using
AuxilaryResourceLimits. With this PR, added the support
to specify AuxilaryResourceRequests.

This feature is useful in cases where user has to specify
minimum requests like ephemeral storage etc. to avoid
erroneous eviction by K8s as described in the above issue.

Also, some of the resource types are very version dependent
so keeping this as generic and not providing any defaults in
this PR.

In the future, it is possible that OpenEBS Operator could
automatically determine the version and set default values
on the generated default Storage Class.

Signed-off-by: kmova <kiran.mova@openebs.io>
(cherry picked from commit 9f12a28)
@AmitKumarDas

This comment has been minimized.

Copy link
Member

AmitKumarDas commented Jan 28, 2019

/doc needs to be updated
Fixes have gone into 0.8.x & master

@kmova

This comment has been minimized.

Copy link
Member

kmova commented Feb 14, 2019

Ephemeral storage requests can be used only with 1.13 (where the feature has moved to beta) or in earlier versions if the feature-gate flag is enabled. As suggested above, it will be good for OpenEBS to automatically determine the version and update the ephemeral requests for the pods it is generating.

When using the automated setup, we will need to audit the logs of all the containers to make sure they are not flooding and hogging the disk space.

With 0.8.1, the following mechanism can be used to set the Ephemeral storage requests.

(a) For Jiva Volumes, modify the storage class to include the Resource Requests policies as shown below.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-jiva
  annotations:
    openebs.io/cas-type: jiva
    cas.openebs.io/config: |
      - name: TargetResourceRequests
        value: |-
            ephemeral-storage: "100Mi"
      - name: ReplicaResourceRequests
        value: |-
            ephemeral-storage: "50Mi"
      - name: AuxResourceRequests
        value: |-
            ephemeral-storage: "100Mi"

With the above configuration, the Jiva Target container and the Jiva Replica container will be set with ephemeral requests of 100M, where as the side-cars in the Jiva Target container will be set with 50M (taken from the AuxResourceRequests).

(b) For cStor Volumes, modify the storage class and the SPC to include the Resource Requests policies as shown below.

---
apiVersion: openebs.io/v1alpha1
kind: StoragePoolClaim
metadata:
  name: cstor-disk
  annotations:
    cas.openebs.io/config: |
      - name: PoolResourceRequests
        value: |-
            ephemeral-storage: "100Mi"
      - name: AuxResourceRequests
        value: |-
            ephemeral-storage: "50Mi"
spec:
  name: cstor-disk
  type: disk
  maxPools: 3
  poolSpec:
    poolType: striped

The above spec will set the cstor-pool (main) container with 100M ephemeral requests and the side-cars with 50M.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-jiva
  annotations:
    openebs.io/cas-type: cstor
    cas.openebs.io/config: |
      - name: StoragePoolClaim
        # update with the correct pool name here. 
        value: "cstor-disk"
      - name: TargetResourceRequests
        value: |-
            ephemeral-storage: "100Mi"
      - name: AuxResourceRequests
        value: |-
            ephemeral-storage: "100Mi"

The above spec will set the cstor-istgt (main) container with 100M ephemeral requests and the side-cars with 50M.

@kmova

This comment has been minimized.

Copy link
Member

kmova commented Feb 14, 2019

Leaving this issue to be in open-state till we get the automated configuration working via the OpenEBS operator workflow - that will help with launching and managing the other openebs pods. ( cc: @AmitKumarDas )

@kmova kmova modified the milestones: 0.8.1, Backlog Feb 14, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.