Skip to content

make minimum limits boundaries configurable #808

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Feb 3, 2020
Merged

make minimum limits boundaries configurable #808

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
eb24d72
make minimum limits boundaries configurable
FxKu Jan 31, 2020
efb6731
minor fixes
FxKu Jan 31, 2020
a1406ff
fix e2e
FxKu Jan 31, 2020
5544843
it's a dict
FxKu Jan 31, 2020
fe7aa92
enable other tests
FxKu Jan 31, 2020
8611536
rename function that enforces min resource limits
FxKu Feb 3, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions charts/postgres-operator/crds/operatorconfigurations.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,12 @@ spec:
default_memory_request:
type: string
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
min_cpu_limit:
type: string
pattern: '^(\d+m|\d+(\.\d{1,3})?)$'
min_memory_limit:
type: string
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
timeouts:
type: object
properties:
Expand Down
26 changes: 15 additions & 11 deletions charts/postgres-operator/values-crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,13 +115,17 @@ configKubernetes:
# configure resource requests for the Postgres pods
configPostgresPodResources:
# CPU limits for the postgres containers
default_cpu_limit: "3"
# cpu request value for the postgres containers
default_cpu_limit: "1"
# CPU request value for the postgres containers
default_cpu_request: 100m
# memory limits for the postgres containers
default_memory_limit: 1Gi
default_memory_limit: 500Mi
# memory request value for the postgres containers
default_memory_request: 100Mi
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here a sound value should also be like 256MB from our expectations?

# hard CPU minimum required to properly run a Postgres cluster
min_cpu_limit: 250m
# hard memory minimum required to properly run a Postgres cluster
min_memory_limit: 250Mi

# timeouts related to some operator actions
configTimeouts:
Expand Down Expand Up @@ -251,7 +255,7 @@ configScalyr:
# CPU rquest value for the Scalyr sidecar
scalyr_cpu_request: 100m
# Memory limit value for the Scalyr sidecar
scalyr_memory_limit: 1Gi
scalyr_memory_limit: 500Mi
# Memory request value for the Scalyr sidecar
scalyr_memory_request: 50Mi

Expand All @@ -272,13 +276,13 @@ serviceAccount:

priorityClassName: ""

resources: {}
# limits:
# cpu: 100m
# memory: 300Mi
# requests:
# cpu: 100m
# memory: 300Mi
resources:
limits:
cpu: 500m
memory: 500Mi
requests:
cpu: 100m
memory: 250Mi

# Affinity for pod assignment
# Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
Expand Down
24 changes: 14 additions & 10 deletions charts/postgres-operator/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,13 +108,17 @@ configKubernetes:
# configure resource requests for the Postgres pods
configPostgresPodResources:
# CPU limits for the postgres containers
default_cpu_limit: "3"
# cpu request value for the postgres containers
default_cpu_limit: "1"
# CPU request value for the postgres containers
default_cpu_request: 100m
# memory limits for the postgres containers
default_memory_limit: 1Gi
default_memory_limit: 500Mi
# memory request value for the postgres containers
default_memory_request: 100Mi
# hard CPU minimum required to properly run a Postgres cluster
min_cpu_limit: 250m
# hard memory minimum required to properly run a Postgres cluster
min_memory_limit: 250Mi

# timeouts related to some operator actions
configTimeouts:
Expand Down Expand Up @@ -248,13 +252,13 @@ serviceAccount:

priorityClassName: ""

resources: {}
# limits:
# cpu: 100m
# memory: 300Mi
# requests:
# cpu: 100m
# memory: 300Mi
resources:
limits:
cpu: 500m
memory: 500Mi
requests:
cpu: 100m
memory: 250Mi

# Affinity for pod assignment
# Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
Expand Down
14 changes: 11 additions & 3 deletions docs/reference/operator_parameters.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -318,11 +318,19 @@ CRD-based configuration.

* **default_cpu_limit**
CPU limits for the Postgres containers, unless overridden by cluster-specific
settings. The default is `3`.
settings. The default is `1`.

* **default_memory_limit**
memory limits for the Postgres containers, unless overridden by cluster-specific
settings. The default is `1Gi`.
settings. The default is `500Mi`.

* **min_cpu_limit**
hard CPU minimum what we consider to be required to properly run Postgres
clusters with Patroni on Kubernetes. The default is `250m`.

* **min_memory_limit**
hard memory minimum what we consider to be required to properly run Postgres
clusters with Patroni on Kubernetes. The default is `250Mi`.

## Operator timeouts

Expand Down Expand Up @@ -579,4 +587,4 @@ scalyr sidecar. In the CRD-based configuration they are grouped under the
CPU limit value for the Scalyr sidecar. The default is `1`.

* **scalyr_memory_limit**
Memory limit value for the Scalyr sidecar. The default is `1Gi`.
Memory limit value for the Scalyr sidecar. The default is `500Mi`.
10 changes: 5 additions & 5 deletions docs/user.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,11 +232,11 @@ spec:
memory: 300Mi
```

The minimum limit to properly run the `postgresql` resource is `256m` for `cpu`
and `256Mi` for `memory`. If a lower value is set in the manifest the operator
will cancel ADD or UPDATE events on this resource with an error. If no
resources are defined in the manifest the operator will obtain the configured
[default requests](reference/operator_parameters.md#kubernetes-resource-requests).
The minimum limits to properly run the `postgresql` resource are configured to
`250m` for `cpu` and `250Mi` for `memory`. If a lower value is set in the
manifest the operator will raise the limits to the configured minimum values.
If no resources are defined in the manifest they will be obtained from the
configured [default requests](reference/operator_parameters.md#kubernetes-resource-requests).

## Use taints and tolerations for dedicated PostgreSQL nodes

Expand Down
68 changes: 59 additions & 9 deletions e2e/tests/test_e2e.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,57 @@ def setUpClass(cls):
k8s.create_with_kubectl("manifests/minimal-postgres-manifest.yaml")
k8s.wait_for_pod_start('spilo-role=master')

@timeout_decorator.timeout(TEST_TIMEOUT_SEC)
def test_min_resource_limits(self):
'''
Lower resource limits below configured minimum and let operator fix it
'''
k8s = self.k8s
cluster_label = 'version=acid-minimal-cluster'
_, failover_targets = k8s.get_pg_nodes(cluster_label)

# configure minimum boundaries for CPU and memory limits
minCPULimit = '250m'
minMemoryLimit = '250Mi'
patch_min_resource_limits = {
"data": {
"min_cpu_limit": minCPULimit,
"min_memory_limit": minMemoryLimit
}
}
k8s.update_config(patch_min_resource_limits)

# lower resource limits below minimum
pg_patch_resources = {
"spec": {
"resources": {
"requests": {
"cpu": "10m",
"memory": "50Mi"
},
"limits": {
"cpu": "200m",
"memory": "200Mi"
}
}
}
}
k8s.api.custom_objects_api.patch_namespaced_custom_object(
"acid.zalan.do", "v1", "default", "postgresqls", "acid-minimal-cluster", pg_patch_resources)
k8s.wait_for_master_failover(failover_targets)

pods = k8s.api.core_v1.list_namespaced_pod(
'default', label_selector='spilo-role=master,' + cluster_label).items
self.assert_master_is_unique()
masterPod = pods[0]

self.assertEqual(masterPod.spec.containers[0].resources.limits['cpu'], minCPULimit,
"Expected CPU limit {}, found {}"
.format(minCPULimit, masterPod.spec.containers[0].resources.limits['cpu']))
self.assertEqual(masterPod.spec.containers[0].resources.limits['memory'], minMemoryLimit,
"Expected memory limit {}, found {}"
.format(minMemoryLimit, masterPod.spec.containers[0].resources.limits['memory']))

@timeout_decorator.timeout(TEST_TIMEOUT_SEC)
def test_multi_namespace_support(self):
'''
Expand All @@ -76,10 +127,9 @@ def test_multi_namespace_support(self):

@timeout_decorator.timeout(TEST_TIMEOUT_SEC)
def test_scaling(self):
"""
'''
Scale up from 2 to 3 and back to 2 pods by updating the Postgres manifest at runtime.
"""

'''
k8s = self.k8s
labels = "version=acid-minimal-cluster"

Expand All @@ -93,9 +143,9 @@ def test_scaling(self):

@timeout_decorator.timeout(TEST_TIMEOUT_SEC)
def test_taint_based_eviction(self):
"""
'''
Add taint "postgres=:NoExecute" to node with master. This must cause a failover.
"""
'''
k8s = self.k8s
cluster_label = 'version=acid-minimal-cluster'

Expand Down Expand Up @@ -145,15 +195,15 @@ def test_taint_based_eviction(self):

@timeout_decorator.timeout(TEST_TIMEOUT_SEC)
def test_logical_backup_cron_job(self):
"""
'''
Ensure we can (a) create the cron job at user request for a specific PG cluster
(b) update the cluster-wide image for the logical backup pod
(c) delete the job at user request

Limitations:
(a) Does not run the actual batch job because there is no S3 mock to upload backups to
(b) Assumes 'acid-minimal-cluster' exists as defined in setUp
"""
'''

k8s = self.k8s

Expand Down Expand Up @@ -208,10 +258,10 @@ def test_logical_backup_cron_job(self):
"Expected 0 logical backup jobs, found {}".format(len(jobs)))

def assert_master_is_unique(self, namespace='default', version="acid-minimal-cluster"):
"""
'''
Check that there is a single pod in the k8s cluster with the label "spilo-role=master"
To be called manually after operations that affect pods
"""
'''

k8s = self.k8s
labels = 'spilo-role=master,version=' + version
Expand Down
4 changes: 2 additions & 2 deletions manifests/complete-postgres-manifest.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ spec:
cpu: 10m
memory: 100Mi
limits:
cpu: 300m
memory: 300Mi
cpu: 500m
memory: 500Mi
patroni:
initdb:
encoding: "UTF8"
Expand Down
6 changes: 4 additions & 2 deletions manifests/configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ data:
# custom_pod_annotations: "keya:valuea,keyb:valueb"
db_hosted_zone: db.example.com
debug_logging: "true"
# default_cpu_limit: "3"
# default_cpu_limit: "1"
# default_cpu_request: 100m
# default_memory_limit: 1Gi
# default_memory_limit: 500Mi
# default_memory_request: 100Mi
docker_image: registry.opensource.zalan.do/acid/spilo-cdp-12:1.6-p16
# enable_admin_role_for_users: "true"
Expand Down Expand Up @@ -48,6 +48,8 @@ data:
# master_pod_move_timeout: 10m
# max_instances: "-1"
# min_instances: "-1"
# min_cpu_limit: 250m
# min_memory_limit: 250Mi
# node_readiness_label: ""
# oauth_token_secret_name: postgresql-operator
# pam_configuration: |
Expand Down
6 changes: 6 additions & 0 deletions manifests/operatorconfiguration.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,12 @@ spec:
default_memory_request:
type: string
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
min_cpu_limit:
type: string
pattern: '^(\d+m|\d+(\.\d{1,3})?)$'
min_memory_limit:
type: string
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
timeouts:
type: object
properties:
Expand Down
4 changes: 2 additions & 2 deletions manifests/postgres-operator.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ spec:
imagePullPolicy: IfNotPresent
resources:
requests:
cpu: 500m
cpu: 100m
memory: 250Mi
limits:
cpu: 2000m
cpu: 500m
memory: 500Mi
securityContext:
runAsUser: 1000
Expand Down
8 changes: 5 additions & 3 deletions manifests/postgresql-operator-default-configuration.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,12 @@ configuration:
# toleration: {}
# watched_namespace: ""
postgres_pod_resources:
default_cpu_limit: "3"
default_cpu_limit: "1"
default_cpu_request: 100m
default_memory_limit: 1Gi
default_memory_limit: 500Mi
default_memory_request: 100Mi
# min_cpu_limit: 250m
# min_memory_limit: 250Mi
timeouts:
pod_label_wait_timeout: 10m
pod_deletion_wait_timeout: 10m
Expand Down Expand Up @@ -115,6 +117,6 @@ configuration:
scalyr_cpu_limit: "1"
scalyr_cpu_request: 100m
# scalyr_image: ""
scalyr_memory_limit: 1Gi
scalyr_memory_limit: 500Mi
scalyr_memory_request: 50Mi
# scalyr_server_url: ""
8 changes: 8 additions & 0 deletions pkg/apis/acid.zalan.do/v1/crds.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -810,6 +810,14 @@ var OperatorConfigCRDResourceValidation = apiextv1beta1.CustomResourceValidation
Type: "string",
Pattern: "^(\\d+(e\\d+)?|\\d+(\\.\\d+)?(e\\d+)?[EPTGMK]i?)$",
},
"min_cpu_limit": {
Type: "string",
Pattern: "^(\\d+m|\\d+(\\.\\d{1,3})?)$",
},
"min_memory_limit": {
Type: "string",
Pattern: "^(\\d+(e\\d+)?|\\d+(\\.\\d+)?(e\\d+)?[EPTGMK]i?)$",
},
},
},
"timeouts": {
Expand Down
2 changes: 2 additions & 0 deletions pkg/apis/acid.zalan.do/v1/operator_configuration_type.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@ type PostgresPodResourcesDefaults struct {
DefaultMemoryRequest string `json:"default_memory_request,omitempty"`
DefaultCPULimit string `json:"default_cpu_limit,omitempty"`
DefaultMemoryLimit string `json:"default_memory_limit,omitempty"`
MinCPULimit string `json:"min_cpu_limit,omitempty"`
MinMemoryLimit string `json:"min_memory_limit,omitempty"`
}

// OperatorTimeouts defines the timeout of ResourceCheck, PodWait, ReadyWait
Expand Down
Loading