diff --git a/advocacy_docs/kubernetes/cloud_native_postgresql/api_reference.mdx b/advocacy_docs/kubernetes/cloud_native_postgresql/api_reference.mdx index 88c8b29293f..e4656eab0c1 100644 --- a/advocacy_docs/kubernetes/cloud_native_postgresql/api_reference.mdx +++ b/advocacy_docs/kubernetes/cloud_native_postgresql/api_reference.mdx @@ -36,6 +36,7 @@ Below you will find a description of the defined resources: * [ClusterSpec](#clusterspec) * [ClusterStatus](#clusterstatus) * [DataBackupConfiguration](#databackupconfiguration) +* [MonitoringConfiguration](#monitoringconfiguration) * [NodeMaintenanceWindow](#nodemaintenancewindow) * [PostgresConfiguration](#postgresconfiguration) * [RecoveryTarget](#recoverytarget) @@ -212,6 +213,7 @@ ClusterSpec defines the desired state of Cluster | backup | The configuration to be used for backups | *[BackupConfiguration](#backupconfiguration) | false | | nodeMaintenanceWindow | Define a maintenance window for the Kubernetes nodes | *[NodeMaintenanceWindow](#nodemaintenancewindow) | false | | licenseKey | The license key of the cluster. When empty, the cluster operates in trial mode and after the expiry date (default 30 days) the operator will cease any reconciliation attempt. For details, please refer to the license agreement that comes with the operator. | string | false | +| monitoring | The configuration of the monitoring infrastructure of this cluster | *[MonitoringConfiguration](#monitoringconfiguration) | false | ## ClusterStatus @@ -229,6 +231,7 @@ ClusterStatus defines the observed state of Cluster | pvcCount | How many PVCs have been created by this cluster | int32 | false | | jobCount | How many Jobs have been created by this cluster | int32 | false | | danglingPVC | List of all the PVCs created by this cluster and still available which are not attached to a Pod | []string | false | +| initializingPVC | List of all the PVCs that are being initialized by this cluster | []string | false | | licenseStatus | Status of the license | licensekey.Status | false | | writeService | Current write pod | string | false | | readService | Current list of read pods | string | false | @@ -248,6 +251,16 @@ DataBackupConfiguration is the configuration of the backup of the data directory | jobs | The number of parallel jobs to be used to upload the backup, defaults to 2 | *int32 | false | +## MonitoringConfiguration + +MonitoringConfiguration is the type containing all the monitoring configuration for a certain cluster + +| Field | Description | Scheme | Required | +| -------------------- | ------------------------------ | -------------------- | -------- | +| customQueriesConfigMap | The list of config maps containing the custom queries | []corev1.ConfigMapKeySelector | false | +| customQueriesSecret | The list of secrets containing the custom queries | []corev1.SecretKeySelector | false | + + ## NodeMaintenanceWindow NodeMaintenanceWindow contains information that the operator will use while upgrading the underlying node. diff --git a/advocacy_docs/kubernetes/cloud_native_postgresql/architecture.mdx b/advocacy_docs/kubernetes/cloud_native_postgresql/architecture.mdx index 3f8c87dd562..4dccd20f717 100644 --- a/advocacy_docs/kubernetes/cloud_native_postgresql/architecture.mdx +++ b/advocacy_docs/kubernetes/cloud_native_postgresql/architecture.mdx @@ -39,16 +39,15 @@ purposes. Applications must be aware of the limitations that [Hot Standby](https://www.postgresql.org/docs/current/hot-standby.html) presents and familiar with the way PostgreSQL operates when dealing with these workloads. -Applications can access any PostgreSQL instance at any time through the `-r` -service made available by the operator at connection time. +Applications can access hot standby replicas through the `-ro` service made available +by the operator. This service enables the application to offload read-only queries from the +primary node. The following diagram shows the architecture: -![Applications reading from any instance in round robin](./images/architecture-r.png) +![Applications reading from hot standby replicas in round robin](./images/architecture-read-only.png) -Applications can also access hot standby replicas through the `-ro` service made available -by the operator. This service enables the application to offload read-only queries from the -primary node. +Applications can also access any PostgreSQL instance at any time through the `-r` service at connection time. ## Application deployments diff --git a/advocacy_docs/kubernetes/cloud_native_postgresql/credits.mdx b/advocacy_docs/kubernetes/cloud_native_postgresql/credits.mdx index 2597cd81bb5..9eaff9e1de0 100644 --- a/advocacy_docs/kubernetes/cloud_native_postgresql/credits.mdx +++ b/advocacy_docs/kubernetes/cloud_native_postgresql/credits.mdx @@ -15,7 +15,9 @@ developed, and tested by the EnterpriseDB Cloud Native team: - Niccolò Fei - Jonathan Gonzalez - Danish Khan +- Anand Nednur - Marco Nenciarini +- Gabriele Quaresima - Jitendra Wadle - Adam Wright diff --git a/advocacy_docs/kubernetes/cloud_native_postgresql/images/architecture-read-only.png b/advocacy_docs/kubernetes/cloud_native_postgresql/images/architecture-read-only.png new file mode 100644 index 00000000000..85a11f18568 Binary files /dev/null and b/advocacy_docs/kubernetes/cloud_native_postgresql/images/architecture-read-only.png differ diff --git a/advocacy_docs/kubernetes/cloud_native_postgresql/index.mdx b/advocacy_docs/kubernetes/cloud_native_postgresql/index.mdx index 0c089b8df5f..998fa2540ae 100644 --- a/advocacy_docs/kubernetes/cloud_native_postgresql/index.mdx +++ b/advocacy_docs/kubernetes/cloud_native_postgresql/index.mdx @@ -17,6 +17,7 @@ navigation: - quickstart - cloud_setup - bootstrap + - resource_management - security - failure_modes - rolling_update @@ -24,6 +25,7 @@ navigation: - postgresql_conf - storage - samples + - monitoring - expose_pg_services - ssl_connections - kubernetes_upgrade diff --git a/advocacy_docs/kubernetes/cloud_native_postgresql/installation.mdx b/advocacy_docs/kubernetes/cloud_native_postgresql/installation.mdx index 3c77f19faad..b90d18d9164 100644 --- a/advocacy_docs/kubernetes/cloud_native_postgresql/installation.mdx +++ b/advocacy_docs/kubernetes/cloud_native_postgresql/installation.mdx @@ -6,15 +6,17 @@ product: 'Cloud Native Operator' ## Installation on Kubernetes +### Directly using the operator manifest + The operator can be installed like any other resource in Kubernetes, through a YAML manifest applied via `kubectl`. -You can install the [latest operator manifest](https://get.enterprisedb.io/cnp/postgresql-operator-1.1.0.yaml) +You can install the [latest operator manifest](https://get.enterprisedb.io/cnp/postgresql-operator-1.2.0.yaml) as follows: ```sh kubectl apply -f \ - https://get.enterprisedb.io/cnp/postgresql-operator-1.1.0.yaml + https://get.enterprisedb.io/cnp/postgresql-operator-1.2.0.yaml ``` Once you have run the `kubectl` command, Cloud Native PostgreSQL will be installed in your Kubernetes cluster. @@ -25,6 +27,16 @@ You can verify that with: kubectl get deploy -n postgresql-operator-system postgresql-operator-controller-manager ``` +### Using the Operator Lifecycle Manager (OLM) + +OperatorHub is a community-sourced index of operators available via the +[Operator Lifecycle Manager](https://github.com/operator-framework/operator-lifecycle-manager), +which is a package managing system for operators. + +You can install Cloud Native PostgreSQL using the metadata available in the +[Cloud Native PostgreSQL page](https://operatorhub.io/operator/cloud-native-postgresql) +from the [OperatorHub.io website](https://operatorhub.io), following the installation steps listed on that page. + ## Installation on Openshift ### Via the web interface diff --git a/advocacy_docs/kubernetes/cloud_native_postgresql/license_keys.mdx b/advocacy_docs/kubernetes/cloud_native_postgresql/license_keys.mdx index 94c60725a99..827291d6aad 100644 --- a/advocacy_docs/kubernetes/cloud_native_postgresql/license_keys.mdx +++ b/advocacy_docs/kubernetes/cloud_native_postgresql/license_keys.mdx @@ -4,19 +4,65 @@ originalFilePath: 'src/license_keys.md' product: 'Cloud Native Operator' --- -Each `Cluster` resource has a `licenseKey` parameter in its definition. - -A `licenseKey` is always required for the operator to work. +A license key is always required for the operator to work. The only exception is when you run the operator with Community PostgreSQL: -in this case, if the `licenseKey` parameter is unset, a cluster will be -started with the default trial license - which automatically expires after 30 days. +in this case, if the license key is unset, a cluster will be started with the default +trial license - which automatically expires after 30 days. !!! Important After the license expiration, the operator will cease any reconciliation attempt on the cluster, effectively stopping to manage its status. The pods and the data will still be available. +## Company level license keys + +A license key allows you to create an unlimited number of PostgreSQL +clusters in your installation. + +The license key needs to be available in a `ConfigMap` in the same +namespace where the operator is deployed. + +In Kubernetes the operator is deployed by default in +the `postgresql-operator-system` namespace. +When instead OLM is used (i.e. on OpenShift), the operator is installed +by default in the `openshift-operators` namespace. + +Given the namespace name, and the license key, you can create +the config map with the following command: + +``` +kubectl create configmap -n [NAMESPACE_NAME_HERE] \ + postgresql-operator-controller-manager-config \ + --from-literal=EDB_LICENSE_KEY=[LICENSE_KEY_HERE] +``` + +The following command can be used to reload the config map: + +``` +kubectl rollout restart deployment -n [NAMESPACE_NAME_HERE] \ + postgresql-operator-controller-manager +``` + +The validity of the license key can be checked inside the cluster status. + +```sh +kubectl get cluster cluster_example -o yaml +[...] +status: + [...] + licenseStatus: + licenseExpiration: "2021-11-06T09:36:02Z" + licenseStatus: Trial + valid: true + isImplicit: false + isTrial: true +[...] +``` + +## Cluster level license keys + +Each `Cluster` resource has a `licenseKey` parameter in its definition. You can find the expiration date, as well as more information about the license, in the cluster status: @@ -29,6 +75,8 @@ status: licenseExpiration: "2021-11-06T09:36:02Z" licenseStatus: Trial valid: true + isImplicit: false + isTrial: true [...] ``` @@ -38,4 +86,4 @@ the expiration date or move the cluster to a production license. Cloud Native PostgreSQL is distributed under the EnterpriseDB Limited Usage License Agreement, available at [enterprisedb.com/limited-use-license](https://www.enterprisedb.com/limited-use-license). -Cloud Native PostgreSQL: Copyright (C) 2019-2020 EnterpriseDB. +Cloud Native PostgreSQL: Copyright (C) 2019-2021 EnterpriseDB. diff --git a/advocacy_docs/kubernetes/cloud_native_postgresql/monitoring.mdx b/advocacy_docs/kubernetes/cloud_native_postgresql/monitoring.mdx new file mode 100644 index 00000000000..d6b04e4225c --- /dev/null +++ b/advocacy_docs/kubernetes/cloud_native_postgresql/monitoring.mdx @@ -0,0 +1,95 @@ +--- +title: 'Monitoring' +originalFilePath: 'src/monitoring.md' +product: 'Cloud Native Operator' +--- + +For each PostgreSQL instance, the operator provides an exporter of metrics for +[Prometheus](https://prometheus.io/) via HTTP, on port 8000. +The operator comes with a predefined set of metrics, as well as a highly +configurable and customizable system to define additional queries via one or +more `ConfigMap` objects - and, future versions, `Secret` too. + +The exporter can be accessed as follows: + +```shell +curl http://:8000/metrics +``` + +All monitoring queries are: + +- transactionally atomic (one transaction per query) +- executed with the `pg_monitor` role + +Please refer to the +["Default roles" section in PostgreSQL documentation](https://www.postgresql.org/docs/current/default-roles.html) +for details on the `pg_monitor` role. + +## User defined metrics + +Users will be able to define metrics through the available interface +that the operator provides. This interface is currently in *beta* state and +only supports definition of custom queries as `ConfigMap` and `Secret` objects +using a YAML file that is inspired by the [queries.yaml file](https://github.com/prometheus-community/postgres_exporter/blob/main/queries.yaml) +of the PostgreSQL Prometheus Exporter. + +Queries must be defined in a `ConfigMap` to be referenced in the `monitoring` +section of the `Cluster` definition, as in the following example: + +```yaml +apiVersion: postgresql.k8s.enterprisedb.io/v1 +kind: Cluster +metadata: + name: cluster-example +spec: + instances: 3 + + storage: + size: 1Gi + + monitoring: + customQueriesConfigMap: + - name: example-monitoring + key: custom-queries +``` + +Specifically, the `monitoring` section looks for an array with the name +`customQueriesConfigMap`, which, as the name suggests, needs a list of +`ConfigMap` key references to be used as the source of custom queries. + +For example: + +```yaml +--- +apiVersion: v1 +kind: ConfigMap +metadata: + namespace: default + name: example-monitoring +data: + custom-queries: | + pg_replication: + query: "SELECT CASE WHEN NOT pg_is_in_recovery() + THEN 0 + ELSE GREATEST (0, + EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))) + END AS lag" + primary: true + metrics: + - lag: + usage: "GAUGE" + description: "Replication lag behind primary in seconds" +``` + +The object must have a name and be in the same namespace as the `Cluster`. +Note that the above query will be executed on the `primary` node, with the +following output. + +```text +# HELP custom_pg_replication_lag Replication lag behind primary in seconds +# TYPE custom_pg_replication_lag gauge +custom_pg_replication_lag 0 +``` + +This framework enables the definition of custom metrics to monitor the database +or the application inside the PostgreSQL cluster. diff --git a/advocacy_docs/kubernetes/cloud_native_postgresql/resource_management.mdx b/advocacy_docs/kubernetes/cloud_native_postgresql/resource_management.mdx new file mode 100644 index 00000000000..6257b775957 --- /dev/null +++ b/advocacy_docs/kubernetes/cloud_native_postgresql/resource_management.mdx @@ -0,0 +1,100 @@ +--- +title: 'Resource management' +originalFilePath: 'src/resource_management.md' +product: 'Cloud Native Operator' +--- + +In a typical Kubernetes cluster, pods run with unlimited resources. By default, +they might be allowed to use as much CPU and RAM as needed. + +Cloud Native PostgreSQL allows administrators to control and manage resource usage by the pods of the cluster, +through the `resources` section of the manifest, with two knobs: + +- `requests`: initial requirement +- `limits`: maximum usage, in case of dynamic increase of resource needs + +For example, you can request an initial amount of RAM of 32MiB (scalable to 128MiB) and 50m of CPU (scalable to 100m) +as follows: + +```yaml + resources: + requests: + memory: "32Mi" + cpu: "50m" + limits: + memory: "128Mi" + cpu: "100m" +``` + +Memory requests and limits are associated with containers, but it is useful to think of a pod as having a memory request +and limit. The pod's memory request is the sum of the memory requests for all the containers in the pod. + +Pod scheduling is based on requests and not on limits. A pod is scheduled to run on a Node only if the Node has enough +available memory to satisfy the pod's memory request. + +For each resource, we divide containers into 3 Quality of Service (QoS) classes, in decreasing order of priority: + +- *Guaranteed* +- *Burstable* +- *Best-Effort* + +For more details, please refer to the ["Configure Quality of Service for Pods"](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#qos-classes) +section in the Kubernetes documentation. + +For a PostgreSQL workload it is recommended to set a "Guaranteed" QoS. + +To avoid resources related issues in Kubernetes, we can refer to the best practices for "out of resource" handling +while creating a cluster: + +- Specify your required values for memory and CPU in the resources section of the manifest file. + This way, you can avoid the `OOM Killed` (where "OOM" stands for Out Of Memory) and `CPU throttle` or any other + resources related issues on running instances. +- For your cluster's pods to get assigned to the "Guaranteed" QoS class, you must set limits and requests + for both memory and CPU to the same value. +- Specify your required PostgreSQL memory parameters consistently with the pod resources (as you would do + in a VM or physical machine scenario - see below). +- Set up database server pods on a dedicated node using nodeSelector. + See the ["nodeSelector field of the affinityconfiguration resource on the API reference page"](api_reference.md#affinityconfiguration). + +You can refer to the following example manifest: + +```yaml +apiVersion: postgresql.k8s.enterprisedb.io/v1 +kind: Cluster +metadata: + name: postgresql-resources +spec: + + instances: 3 + + postgresql: + parameters: + shared_buffers: "256MB" + + resources: + requests: + memory: "1024Mi" + cpu: 1 + limits: + memory: "1024Mi" + cpu: 1 + + storage: + size: 1Gi +``` + +In the above example, we have specified `shared_buffers` parameter with a value of `256MB` - i.e., how much memory is +dedicated to the PostgreSQL server for caching data (the default value for this parameter is `128MB` in case +it's not defined). + +A reasonable starting value for `shared_buffers` is 25% of the memory in your system. +For example: if your `shared_buffers` is 256 MB, then the recommended value for your container memory size is 1 GB, +which means that within a pod all the containers will have a total of 1 GB memory that Kubernetes will always preserve, +enabling our containers to work as expected. +For more details, please refer to the ["Resource Consumption"](https://www.postgresql.org/docs/current/runtime-config-resource.html) +section in the PostgreSQL documentation. + +!!! Seealso "Managing Compute Resources for Containers" + For more details on resource management, please refer to the + ["Managing Compute Resources for Containers"](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) + page from the Kubernetes documentation. diff --git a/advocacy_docs/kubernetes/cloud_native_postgresql/rolling_update.mdx b/advocacy_docs/kubernetes/cloud_native_postgresql/rolling_update.mdx index 8bef6cedd15..b6fb6400a8b 100644 --- a/advocacy_docs/kubernetes/cloud_native_postgresql/rolling_update.mdx +++ b/advocacy_docs/kubernetes/cloud_native_postgresql/rolling_update.mdx @@ -26,17 +26,17 @@ from the one with the highest serial. The primary is the last node to be upgraded. This operation is configurable and managed by the `primaryUpdateStrategy` option, accepting these two values: -- `switchover`: the rolling update process is managed by Kubernetes +- `unsupervised`: the rolling update process is managed by Kubernetes and is entirely automated, with the *switchover* operation starting once all the replicas have been upgraded -- `manual`: the rolling update process is suspended immediately +- `supervised`: the rolling update process is suspended immediately after all replicas have been upgraded and can only be completed with a manual switchover triggered by an administrator with `kubectl cnp promote [cluster] [pod]`. The plugin can be downloaded from the [`kubectl-cnp` project page](https://github.com/EnterpriseDB/kubectl-cnp) on GitHub. -The default and recommended value is `switchover`. +The default and recommended value is `unsupervised`. The upgrade keeps the Cloud Native PostgreSQL identity and does not re-clone the data. Pods will be deleted and created again with the same PVCs. diff --git a/advocacy_docs/kubernetes/cloud_native_postgresql/samples/cluster-example-monitoring.yaml b/advocacy_docs/kubernetes/cloud_native_postgresql/samples/cluster-example-monitoring.yaml new file mode 100644 index 00000000000..88edcf51db9 --- /dev/null +++ b/advocacy_docs/kubernetes/cloud_native_postgresql/samples/cluster-example-monitoring.yaml @@ -0,0 +1,235 @@ +apiVersion: postgresql.k8s.enterprisedb.io/v1 +kind: Cluster +metadata: + name: cluster-example +spec: + instances: 3 + + storage: + size: 1Gi + + monitoring: + customQueriesConfigMap: + - name: example-monitoring + key: custom-queries + customQueriesSecret: + - name: example-monitoring-secret + key: pg-database +--- +apiVersion: v1 +kind: ConfigMap +metadata: + namespace: default + name: example-monitoring +data: + custom-queries: | + pg_replication: + query: "SELECT CASE WHEN NOT pg_is_in_recovery() THEN 0 ELSE GREATEST (0, EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))) END AS lag" + primary: true + metrics: + - lag: + usage: "GAUGE" + description: "Replication lag behind primary in seconds" + + pg_postmaster: # wokeignore:rule=master + query: "SELECT pg_postmaster_start_time as start_time_seconds from pg_postmaster_start_time()" # wokeignore:rule=master + primary: true + metrics: + - start_time_seconds: + usage: "GAUGE" + description: "Time at which postmaster started" # wokeignore:rule=master + + pg_stat_user_tables: + query: | + SELECT + current_database() datname, + schemaname, + relname, + seq_scan, + seq_tup_read, + idx_scan, + idx_tup_fetch, + n_tup_ins, + n_tup_upd, + n_tup_del, + n_tup_hot_upd, + n_live_tup, + n_dead_tup, + n_mod_since_analyze, + COALESCE(last_vacuum, '1970-01-01Z') as last_vacuum, + COALESCE(last_autovacuum, '1970-01-01Z') as last_autovacuum, + COALESCE(last_analyze, '1970-01-01Z') as last_analyze, + COALESCE(last_autoanalyze, '1970-01-01Z') as last_autoanalyze, + vacuum_count, + autovacuum_count, + analyze_count, + autoanalyze_count + FROM + pg_stat_user_tables + metrics: + - datname: + usage: "LABEL" + description: "Name of current database" + - schemaname: + usage: "LABEL" + description: "Name of the schema that this table is in" + - relname: + usage: "LABEL" + description: "Name of this table" + - seq_scan: + usage: "COUNTER" + description: "Number of sequential scans initiated on this table" + - seq_tup_read: + usage: "COUNTER" + description: "Number of live rows fetched by sequential scans" + - idx_scan: + usage: "COUNTER" + description: "Number of index scans initiated on this table" + - idx_tup_fetch: + usage: "COUNTER" + description: "Number of live rows fetched by index scans" + - n_tup_ins: + usage: "COUNTER" + description: "Number of rows inserted" + - n_tup_upd: + usage: "COUNTER" + description: "Number of rows updated" + - n_tup_del: + usage: "COUNTER" + description: "Number of rows deleted" + - n_tup_hot_upd: + usage: "COUNTER" + description: "Number of rows HOT updated (i.e., with no separate index update required)" + - n_live_tup: + usage: "GAUGE" + description: "Estimated number of live rows" + - n_dead_tup: + usage: "GAUGE" + description: "Estimated number of dead rows" + - n_mod_since_analyze: + usage: "GAUGE" + description: "Estimated number of rows changed since last analyze" + - last_vacuum: + usage: "GAUGE" + description: "Last time at which this table was manually vacuumed (not counting VACUUM FULL)" + - last_autovacuum: + usage: "GAUGE" + description: "Last time at which this table was vacuumed by the autovacuum daemon" + - last_analyze: + usage: "GAUGE" + description: "Last time at which this table was manually analyzed" + - last_autoanalyze: + usage: "GAUGE" + description: "Last time at which this table was analyzed by the autovacuum daemon" + - vacuum_count: + usage: "COUNTER" + description: "Number of times this table has been manually vacuumed (not counting VACUUM FULL)" + - autovacuum_count: + usage: "COUNTER" + description: "Number of times this table has been vacuumed by the autovacuum daemon" + - analyze_count: + usage: "COUNTER" + description: "Number of times this table has been manually analyzed" + - autoanalyze_count: + usage: "COUNTER" + description: "Number of times this table has been analyzed by the autovacuum daemon" + + pg_statio_user_tables: + query: "SELECT current_database() datname, schemaname, relname, heap_blks_read, heap_blks_hit, idx_blks_read, idx_blks_hit, toast_blks_read, toast_blks_hit, tidx_blks_read, tidx_blks_hit FROM pg_statio_user_tables" + metrics: + - datname: + usage: "LABEL" + description: "Name of current database" + - schemaname: + usage: "LABEL" + description: "Name of the schema that this table is in" + - relname: + usage: "LABEL" + description: "Name of this table" + - heap_blks_read: + usage: "COUNTER" + description: "Number of disk blocks read from this table" + - heap_blks_hit: + usage: "COUNTER" + description: "Number of buffer hits in this table" + - idx_blks_read: + usage: "COUNTER" + description: "Number of disk blocks read from all indexes on this table" + - idx_blks_hit: + usage: "COUNTER" + description: "Number of buffer hits in all indexes on this table" + - toast_blks_read: + usage: "COUNTER" + description: "Number of disk blocks read from this table's TOAST table (if any)" + - toast_blks_hit: + usage: "COUNTER" + description: "Number of buffer hits in this table's TOAST table (if any)" + - tidx_blks_read: + usage: "COUNTER" + description: "Number of disk blocks read from this table's TOAST table indexes (if any)" + - tidx_blks_hit: + usage: "COUNTER" + description: "Number of buffer hits in this table's TOAST table indexes (if any)" + + pg_stat_activity: + query: | + WITH + metrics AS ( + SELECT + application_name, + SUM(EXTRACT(EPOCH FROM (CURRENT_TIMESTAMP - state_change))::bigint)::float AS process_idle_seconds_sum, + COUNT(*) AS process_idle_seconds_count + FROM pg_stat_activity + WHERE state = 'idle' + GROUP BY application_name + ), + buckets AS ( + SELECT + application_name, + le, + SUM( + CASE WHEN EXTRACT(EPOCH FROM (CURRENT_TIMESTAMP - state_change)) <= le + THEN 1 + ELSE 0 + END + )::bigint AS bucket + FROM + pg_stat_activity, + UNNEST(ARRAY[1, 2, 5, 15, 30, 60, 90, 120, 300]) AS le + GROUP BY application_name, le + ORDER BY application_name, le + ) + SELECT + application_name, + process_idle_seconds_sum, + process_idle_seconds_count, + ARRAY_AGG(le) AS process_idle_seconds, + ARRAY_AGG(bucket) AS process_idle_seconds_bucket + FROM metrics JOIN buckets USING (application_name) + GROUP BY 1, 2, 3 + metrics: + - application_name: + usage: "LABEL" + description: "Application Name" + - process_idle_seconds: + usage: "HISTOGRAM" + description: "Idle time of server processes" +--- +apiVersion: v1 +kind: Secret +metadata: + namespace: default + name: example-monitoring-secret +stringData: + pg-database: | + pg_database: + query: "SELECT pg_database.datname, pg_database_size(pg_database.datname) as size_bytes FROM pg_database" + primary: true + cache_seconds: 30 + metrics: + - datname: + usage: "LABEL" + description: "Name of the database" + - size_bytes: + usage: "GAUGE" + description: "Disk space used by the database" diff --git a/advocacy_docs/kubernetes/cloud_native_postgresql/security.mdx b/advocacy_docs/kubernetes/cloud_native_postgresql/security.mdx index 9baf637d668..a42e034ee3b 100644 --- a/advocacy_docs/kubernetes/cloud_native_postgresql/security.mdx +++ b/advocacy_docs/kubernetes/cloud_native_postgresql/security.mdx @@ -4,17 +4,24 @@ originalFilePath: 'src/security.md' product: 'Cloud Native Operator' --- -This section contains information about security for Cloud Native PostgreSQL, -from a few standpoints: source code, Kubernetes, and PostgreSQL. +This section contains information about security for Cloud Native PostgreSQL +analyzed at 3 different layers: Code, Container and Cluster. !!! Warning The information contained in this page must not exonerate you from - performing regular InfoSec duties on your Kubernetes cluster. + performing regular InfoSec duties on your Kubernetes cluster. Please + familiarize with the ["Overview of Cloud Native Security"](https://kubernetes.io/docs/concepts/security/overview/) + page from the Kubernetes documentation. + +!!! Seealso "About the 4C's Security Model" + Please refer to ["The 4C’s Security Model in Kubernetes"](https://www.enterprisedb.com/blog/4cs-security-model-kubernetes) + blog article to get a better understanding and context of the approach EDB + has taken with security in Cloud Native PostgreSQL. -## Source code static analysis +## Code Source code of Cloud Native PostgreSQL is *systematically scanned* for static analysis purposes, -including **security problems**, using a popular open-source for Go called +including **security problems**, using a popular open-source linter for Go called [GolangCI-Lint](https://github.com/golangci/golangci-lint) directly in the CI/CD pipeline. GolangCI-Lint can run several *linters* on the same source code. @@ -31,7 +38,39 @@ the code such as hard-coded credentials, integer overflows and SQL injections - Source code is also regularly inspected through [Coverity Scan by Synopsys](https://scan.coverity.com/) via EnterpriseDB's internal CI/CD pipeline. -## Kubernetes +## Container + +Every container image that is part of Cloud Native PostgreSQL is automatically built via CI/CD pipelines following every commit. +Such images include not only the operator's, but also the operands' - specifically every supported PostgreSQL and EDB Postgres Advanced version. +Within the pipelines, images are scanned with: + +- [Dockle](https://github.com/goodwithtech/dockle): for best practices in terms + of the container build process +- [Clair](https://github.com/quay/clair): for vulnerabilities found in both the + underlying operating system as well as libraries and applications that they run + +!!! Important + All operand images are automatically rebuilt once a day by our pipelines in case + of security updates at the base image and package level, providing **patch level updates** + for the container images that EDB distributes. + +The following guidelines and frameworks have been taken into account for container-level security: + +- the ["Container Image Creation and Deployment Guide"](https://dl.dod.cyber.mil/wp-content/uploads/devsecops/pdf/DevSecOps_Enterprise_Container_Image_Creation_and_Deployment_Guide_2.6-Public-Release.pdf), + developed by the Defense Information Systems Agency (DISA) of the United States Department of Defense (DoD) +- the ["CIS Benchmark for Docker"](https://www.cisecurity.org/benchmark/docker/), + developed by the Center for Internet Security (CIS) + +!!! Seealso "About the Container level security" + Please refer to ["Security and Containers in Cloud Native PostgreSQL"](https://www.enterprisedb.com/blog/security-and-containers-cloud-native-postgresql) + blog article for more information about the approach that EDB has taken on + security at container level in Cloud Native PostgreSQL. + +## Cluster + +Security at the cluster level takes into account all Kubernetes components that +form both the control plane and the nodes, as well as the applications that run in +the cluster (PostgreSQL included). ### Pod Security Policies @@ -56,100 +95,7 @@ Network policies are beyond the scope of this document. Please refer to the ["Network policies"](https://kubernetes.io/docs/concepts/services-networking/network-policies/) section of the Kubernetes documentation for further information. -### Resources - -In a typical Kubernetes cluster, containers run with unlimited resources. By default, -they might be allowed to use as much CPU and RAM as needed. - -Cloud Native PostgreSQL allows administrators to control and manage resource usage by the pods of the cluster, -through the `resources` section of the manifest, with two knobs: - -- `requests`: initial requirement -- `limits`: maximum usage, in case of dynamic increase of resource needs - -For example, you can request an initial amount of RAM of 32MiB (scalable to 128MiB) and 50m of CPU (scalable to 100m) as follows: - -```yaml - resources: - requests: - memory: "32Mi" - cpu: "50m" - limits: - memory: "128Mi" - cpu: "100m" -``` - -Memory requests and limits are associated with containers, but it is useful to think of a pod as having a memory request -and limit. The memory request for the pod is the sum of the memory requests for all the containers in the pod. - -Pod scheduling is based on requests and not limits. A pod is scheduled to run on a Node only if the Node has enough -available memory to satisfy the pod's memory request. - -For each resource, we divide containers into 3 Quality of Service (QoS) classes, in decreasing order of priority: - -- *Guaranteed* -- *Burstable* -- *Best-Effort* - -For more details, please refer to the ["Configure Quality of Service for Pods"](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#qos-classes) section in the Kubernetes documentation. - -For a PostgreSQL workload it is recommended to set a "Guaranteed" QoS. - -In order to avoid resources related issues in Kubernetes, we can refer to the best practices for "out of resource" handling while creating -a cluster: - -- Specify your required values for memory and CPU in the resources section of the manifest file. - This way you can avoid the `OOM Killed` (where "OOM" stands for Out Of Memory) and `CPU throttle` or any other resources - related issues on running instances. -- In order for the pods of your cluster to get assigned to the "Guaranteed" QoS class, you must set limits and requests - for both memory and CPU to the same value. -- Specify your required PostgreSQL memory parameters consistently with the pod resources (like you would do in a VM or physical machine scenario - see below). -- Set up database server pods on a dedicated node using nodeSelector. - See the ["nodeSelector field of the affinityconfiguration resource on the API reference page"](api_reference.md#affinityconfiguration). - -You can refer the following example manifest: - -```yaml -apiVersion: postgresql.k8s.enterprisedb.io/v1 -kind: Cluster -metadata: - name: postgresql-resources -spec: - - instances: 3 - - postgresql: - parameters: - shared_buffers: "256MB" - - resources: - requests: - memory: "1024Mi" - cpu: 1 - limits: - memory: "1024Mi" - cpu: 1 - - storage: - size: 1Gi -``` - -In the above example, we have specified `shared_buffers` parameter with a value of `256MB` - i.e. how much memory is -dedicated to the PostgreSQL server for caching data (the default value for this parameter is `128MB` in case it's not defined). - -A reasonable starting value for `shared_buffers` is 25% of the memory in your system. -For example: if your `shared_buffers` is 256 MB, then the recommended value for your container memory size is 1 GB, -which means that within a pod all the containers will have a total of 1 GB memory that Kubernetes will always preserve, -enabling our containers to work as expected. -For more details, please refer to the ["Resource Consumption"](https://www.postgresql.org/docs/current/runtime-config-resource.html) -section in the PostgreSQL documentation. - -!!! See also "Managing Compute Resources for Containers" - For more details on resource management, please refer to the - ["Managing Compute Resources for Containers"](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) - page from the Kubernetes documentation. - -## PostgreSQL +### PostgreSQL The current implementation of Cloud Native PostgreSQL automatically creates passwords and `.pgpass` files for the `postgres` superuser and the database owner. diff --git a/docs/how-tos/sync-cnp-docs.md b/docs/how-tos/sync-cnp-docs.md new file mode 100644 index 00000000000..afea70fb812 --- /dev/null +++ b/docs/how-tos/sync-cnp-docs.md @@ -0,0 +1,20 @@ +# Sync Cloud-Native-PostgreSQL Docs + +Currently we need to manually sync over [cloud-native-postgresql][cnp]("CNP") +docs whenever there's a new release. The long term goal is to automate this via +GitHub action dispatch and automated event handling. + +1. The CNP team informs us that there's a new version. +1. Check out the appropriate version from the [CNP][] repo. +1. Replace `docs:temp_kubernetes/original/` with + `cloud-native-postgresql:docs/`. +1. Transpile original source documentation into MDX format: + + ```sh + python scripts/source/source_cloud_native_operator.py + ``` + +1. Replace `advocacy_docs/kubernetes/cloud-native-postgresql/` with + `temp_kubernetes/build/`. + +[cnp]: https://github.com/EnterpriseDB/cloud-native-postgresql diff --git a/temp_kubernetes/original/graffle/architecture-read-only.graffle b/temp_kubernetes/original/graffle/architecture-read-only.graffle new file mode 100644 index 00000000000..571d335efb0 Binary files /dev/null and b/temp_kubernetes/original/graffle/architecture-read-only.graffle differ diff --git a/temp_kubernetes/original/mkdocs.yml b/temp_kubernetes/original/mkdocs.yml index a33c4321580..245dd2fa97e 100644 --- a/temp_kubernetes/original/mkdocs.yml +++ b/temp_kubernetes/original/mkdocs.yml @@ -18,6 +18,7 @@ nav: - quickstart.md - cloud_setup.md - bootstrap.md + - resource_management.md - security.md - failure_modes.md - rolling_update.md @@ -25,6 +26,7 @@ nav: - postgresql_conf.md - storage.md - samples.md + - monitoring.md - expose_pg_services.md - ssl_connections.md - kubernetes_upgrade.md diff --git a/temp_kubernetes/original/requirements.txt b/temp_kubernetes/original/requirements.txt deleted file mode 100755 index a7f665233c5..00000000000 --- a/temp_kubernetes/original/requirements.txt +++ /dev/null @@ -1,3 +0,0 @@ -mkdocs -mkdocs-bootswatch -pandocfilters diff --git a/temp_kubernetes/original/src/api_reference.md b/temp_kubernetes/original/src/api_reference.md index cb67f3d4808..0b31b71b1f0 100644 --- a/temp_kubernetes/original/src/api_reference.md +++ b/temp_kubernetes/original/src/api_reference.md @@ -32,6 +32,7 @@ Below you will find a description of the defined resources: * [ClusterSpec](#clusterspec) * [ClusterStatus](#clusterstatus) * [DataBackupConfiguration](#databackupconfiguration) +* [MonitoringConfiguration](#monitoringconfiguration) * [NodeMaintenanceWindow](#nodemaintenancewindow) * [PostgresConfiguration](#postgresconfiguration) * [RecoveryTarget](#recoverytarget) @@ -62,7 +63,7 @@ BackupList contains a list of Backup | Field | Description | Scheme | Required | | -------------------- | ------------------------------ | -------------------- | -------- | | metadata | Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds | [metav1.ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#listmeta-v1-meta) | false | -| items | List of backups | [][Backup](#backup) | true | +| items | List of backups | \[][Backup](#backup) | true | ## BackupSpec @@ -179,7 +180,7 @@ ClusterList contains a list of Cluster | Field | Description | Scheme | Required | | -------------------- | ------------------------------ | -------------------- | -------- | | metadata | Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds | [metav1.ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#listmeta-v1-meta) | false | -| items | List of clusters | [][Cluster](#cluster) | true | +| items | List of clusters | \[][Cluster](#cluster) | true | ## ClusterSpec @@ -208,6 +209,7 @@ ClusterSpec defines the desired state of Cluster | backup | The configuration to be used for backups | *[BackupConfiguration](#backupconfiguration) | false | | nodeMaintenanceWindow | Define a maintenance window for the Kubernetes nodes | *[NodeMaintenanceWindow](#nodemaintenancewindow) | false | | licenseKey | The license key of the cluster. When empty, the cluster operates in trial mode and after the expiry date (default 30 days) the operator will cease any reconciliation attempt. For details, please refer to the license agreement that comes with the operator. | string | false | +| monitoring | The configuration of the monitoring infrastructure of this cluster | *[MonitoringConfiguration](#monitoringconfiguration) | false | ## ClusterStatus @@ -225,6 +227,7 @@ ClusterStatus defines the observed state of Cluster | pvcCount | How many PVCs have been created by this cluster | int32 | false | | jobCount | How many Jobs have been created by this cluster | int32 | false | | danglingPVC | List of all the PVCs created by this cluster and still available which are not attached to a Pod | []string | false | +| initializingPVC | List of all the PVCs that are being initialized by this cluster | []string | false | | licenseStatus | Status of the license | licensekey.Status | false | | writeService | Current write pod | string | false | | readService | Current list of read pods | string | false | @@ -244,6 +247,16 @@ DataBackupConfiguration is the configuration of the backup of the data directory | jobs | The number of parallel jobs to be used to upload the backup, defaults to 2 | *int32 | false | +## MonitoringConfiguration + +MonitoringConfiguration is the type containing all the monitoring configuration for a certain cluster + +| Field | Description | Scheme | Required | +| -------------------- | ------------------------------ | -------------------- | -------- | +| customQueriesConfigMap | The list of config maps containing the custom queries | []corev1.ConfigMapKeySelector | false | +| customQueriesSecret | The list of secrets containing the custom queries | []corev1.SecretKeySelector | false | + + ## NodeMaintenanceWindow NodeMaintenanceWindow contains information that the operator will use while upgrading the underlying node. @@ -341,7 +354,7 @@ ScheduledBackupList contains a list of ScheduledBackup | Field | Description | Scheme | Required | | -------------------- | ------------------------------ | -------------------- | -------- | | metadata | Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds | [metav1.ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#listmeta-v1-meta) | false | -| items | List of clusters | [][ScheduledBackup](#scheduledbackup) | true | +| items | List of clusters | \[][ScheduledBackup](#scheduledbackup) | true | ## ScheduledBackupSpec diff --git a/temp_kubernetes/original/src/architecture.md b/temp_kubernetes/original/src/architecture.md index bcd2f7ba6f8..5b27ded904b 100644 --- a/temp_kubernetes/original/src/architecture.md +++ b/temp_kubernetes/original/src/architecture.md @@ -35,16 +35,15 @@ purposes. Applications must be aware of the limitations that [Hot Standby](https://www.postgresql.org/docs/current/hot-standby.html) presents and familiar with the way PostgreSQL operates when dealing with these workloads. -Applications can access any PostgreSQL instance at any time through the `-r` -service made available by the operator at connection time. +Applications can access hot standby replicas through the `-ro` service made available +by the operator. This service enables the application to offload read-only queries from the +primary node. The following diagram shows the architecture: -![Applications reading from any instance in round robin](./images/architecture-r.png) +![Applications reading from hot standby replicas in round robin](./images/architecture-read-only.png) -Applications can also access hot standby replicas through the `-ro` service made available -by the operator. This service enables the application to offload read-only queries from the -primary node. +Applications can also access any PostgreSQL instance at any time through the `-r` service at connection time. ## Application deployments diff --git a/temp_kubernetes/original/src/credits.md b/temp_kubernetes/original/src/credits.md index 74c0d92c67e..03b571bc312 100644 --- a/temp_kubernetes/original/src/credits.md +++ b/temp_kubernetes/original/src/credits.md @@ -11,7 +11,9 @@ developed, and tested by the EnterpriseDB Cloud Native team: - Niccolò Fei - Jonathan Gonzalez - Danish Khan +- Anand Nednur - Marco Nenciarini +- Gabriele Quaresima - Jitendra Wadle - Adam Wright diff --git a/temp_kubernetes/original/src/images/architecture-read-only.png b/temp_kubernetes/original/src/images/architecture-read-only.png new file mode 100644 index 00000000000..85a11f18568 Binary files /dev/null and b/temp_kubernetes/original/src/images/architecture-read-only.png differ diff --git a/temp_kubernetes/original/src/installation.md b/temp_kubernetes/original/src/installation.md index 47964a141dc..ddfa49b1e4c 100644 --- a/temp_kubernetes/original/src/installation.md +++ b/temp_kubernetes/original/src/installation.md @@ -2,15 +2,17 @@ ## Installation on Kubernetes +### Directly using the operator manifest + The operator can be installed like any other resource in Kubernetes, through a YAML manifest applied via `kubectl`. -You can install the [latest operator manifest](https://get.enterprisedb.io/cnp/postgresql-operator-1.1.0.yaml) +You can install the [latest operator manifest](https://get.enterprisedb.io/cnp/postgresql-operator-1.2.0.yaml) as follows: ```sh kubectl apply -f \ - https://get.enterprisedb.io/cnp/postgresql-operator-1.1.0.yaml + https://get.enterprisedb.io/cnp/postgresql-operator-1.2.0.yaml ``` Once you have run the `kubectl` command, Cloud Native PostgreSQL will be installed in your Kubernetes cluster. @@ -21,6 +23,16 @@ You can verify that with: kubectl get deploy -n postgresql-operator-system postgresql-operator-controller-manager ``` +### Using the Operator Lifecycle Manager (OLM) + +OperatorHub is a community-sourced index of operators available via the +[Operator Lifecycle Manager](https://github.com/operator-framework/operator-lifecycle-manager), +which is a package managing system for operators. + +You can install Cloud Native PostgreSQL using the metadata available in the +[Cloud Native PostgreSQL page](https://operatorhub.io/operator/cloud-native-postgresql) +from the [OperatorHub.io website](https://operatorhub.io), following the installation steps listed on that page. + ## Installation on Openshift ### Via the web interface diff --git a/temp_kubernetes/original/src/license_keys.md b/temp_kubernetes/original/src/license_keys.md index a88782dc74d..a495ea65d38 100644 --- a/temp_kubernetes/original/src/license_keys.md +++ b/temp_kubernetes/original/src/license_keys.md @@ -1,18 +1,64 @@ # License and License Keys -Each `Cluster` resource has a `licenseKey` parameter in its definition. - -A `licenseKey` is always required for the operator to work. +A license key is always required for the operator to work. The only exception is when you run the operator with Community PostgreSQL: -in this case, if the `licenseKey` parameter is unset, a cluster will be -started with the default trial license - which automatically expires after 30 days. +in this case, if the license key is unset, a cluster will be started with the default +trial license - which automatically expires after 30 days. !!! Important After the license expiration, the operator will cease any reconciliation attempt on the cluster, effectively stopping to manage its status. The pods and the data will still be available. +## Company level license keys + +A license key allows you to create an unlimited number of PostgreSQL +clusters in your installation. + +The license key needs to be available in a `ConfigMap` in the same +namespace where the operator is deployed. + +In Kubernetes the operator is deployed by default in +the `postgresql-operator-system` namespace. +When instead OLM is used (i.e. on OpenShift), the operator is installed +by default in the `openshift-operators` namespace. + +Given the namespace name, and the license key, you can create +the config map with the following command: + +``` +kubectl create configmap -n [NAMESPACE_NAME_HERE] \ + postgresql-operator-controller-manager-config \ + --from-literal=EDB_LICENSE_KEY=[LICENSE_KEY_HERE] +``` + +The following command can be used to reload the config map: + +``` +kubectl rollout restart deployment -n [NAMESPACE_NAME_HERE] \ + postgresql-operator-controller-manager +``` + +The validity of the license key can be checked inside the cluster status. + +```sh +kubectl get cluster cluster_example -o yaml +[...] +status: + [...] + licenseStatus: + licenseExpiration: "2021-11-06T09:36:02Z" + licenseStatus: Trial + valid: true + isImplicit: false + isTrial: true +[...] +``` + +## Cluster level license keys + +Each `Cluster` resource has a `licenseKey` parameter in its definition. You can find the expiration date, as well as more information about the license, in the cluster status: @@ -25,6 +71,8 @@ status: licenseExpiration: "2021-11-06T09:36:02Z" licenseStatus: Trial valid: true + isImplicit: false + isTrial: true [...] ``` @@ -34,4 +82,4 @@ the expiration date or move the cluster to a production license. Cloud Native PostgreSQL is distributed under the EnterpriseDB Limited Usage License Agreement, available at [enterprisedb.com/limited-use-license](https://www.enterprisedb.com/limited-use-license). -Cloud Native PostgreSQL: Copyright (C) 2019-2020 EnterpriseDB. +Cloud Native PostgreSQL: Copyright (C) 2019-2021 EnterpriseDB. diff --git a/temp_kubernetes/original/src/monitoring.md b/temp_kubernetes/original/src/monitoring.md new file mode 100644 index 00000000000..357ec6f961d --- /dev/null +++ b/temp_kubernetes/original/src/monitoring.md @@ -0,0 +1,91 @@ +# Monitoring + +For each PostgreSQL instance, the operator provides an exporter of metrics for +[Prometheus](https://prometheus.io/) via HTTP, on port 8000. +The operator comes with a predefined set of metrics, as well as a highly +configurable and customizable system to define additional queries via one or +more `ConfigMap` objects - and, future versions, `Secret` too. + +The exporter can be accessed as follows: + +```shell +curl http://:8000/metrics +``` + +All monitoring queries are: + +- transactionally atomic (one transaction per query) +- executed with the `pg_monitor` role + +Please refer to the +["Default roles" section in PostgreSQL documentation](https://www.postgresql.org/docs/current/default-roles.html) +for details on the `pg_monitor` role. + +## User defined metrics + +Users will be able to define metrics through the available interface +that the operator provides. This interface is currently in *beta* state and +only supports definition of custom queries as `ConfigMap` and `Secret` objects +using a YAML file that is inspired by the [queries.yaml file](https://github.com/prometheus-community/postgres_exporter/blob/main/queries.yaml) +of the PostgreSQL Prometheus Exporter. + +Queries must be defined in a `ConfigMap` to be referenced in the `monitoring` +section of the `Cluster` definition, as in the following example: + +```yaml +apiVersion: postgresql.k8s.enterprisedb.io/v1 +kind: Cluster +metadata: + name: cluster-example +spec: + instances: 3 + + storage: + size: 1Gi + + monitoring: + customQueriesConfigMap: + - name: example-monitoring + key: custom-queries +``` + +Specifically, the `monitoring` section looks for an array with the name +`customQueriesConfigMap`, which, as the name suggests, needs a list of +`ConfigMap` key references to be used as the source of custom queries. + +For example: + +```yaml +--- +apiVersion: v1 +kind: ConfigMap +metadata: + namespace: default + name: example-monitoring +data: + custom-queries: | + pg_replication: + query: "SELECT CASE WHEN NOT pg_is_in_recovery() + THEN 0 + ELSE GREATEST (0, + EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))) + END AS lag" + primary: true + metrics: + - lag: + usage: "GAUGE" + description: "Replication lag behind primary in seconds" +``` + +The object must have a name and be in the same namespace as the `Cluster`. +Note that the above query will be executed on the `primary` node, with the +following output. + +```text +# HELP custom_pg_replication_lag Replication lag behind primary in seconds +# TYPE custom_pg_replication_lag gauge +custom_pg_replication_lag 0 +``` + +This framework enables the definition of custom metrics to monitor the database +or the application inside the PostgreSQL cluster. diff --git a/temp_kubernetes/original/src/resource_management.md b/temp_kubernetes/original/src/resource_management.md new file mode 100644 index 00000000000..cf4188c181e --- /dev/null +++ b/temp_kubernetes/original/src/resource_management.md @@ -0,0 +1,96 @@ +# Resource management + +In a typical Kubernetes cluster, pods run with unlimited resources. By default, +they might be allowed to use as much CPU and RAM as needed. + +Cloud Native PostgreSQL allows administrators to control and manage resource usage by the pods of the cluster, +through the `resources` section of the manifest, with two knobs: + +- `requests`: initial requirement +- `limits`: maximum usage, in case of dynamic increase of resource needs + +For example, you can request an initial amount of RAM of 32MiB (scalable to 128MiB) and 50m of CPU (scalable to 100m) +as follows: + +```yaml + resources: + requests: + memory: "32Mi" + cpu: "50m" + limits: + memory: "128Mi" + cpu: "100m" +``` + +Memory requests and limits are associated with containers, but it is useful to think of a pod as having a memory request +and limit. The pod's memory request is the sum of the memory requests for all the containers in the pod. + +Pod scheduling is based on requests and not on limits. A pod is scheduled to run on a Node only if the Node has enough +available memory to satisfy the pod's memory request. + +For each resource, we divide containers into 3 Quality of Service (QoS) classes, in decreasing order of priority: + +- *Guaranteed* +- *Burstable* +- *Best-Effort* + +For more details, please refer to the ["Configure Quality of Service for Pods"](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#qos-classes) +section in the Kubernetes documentation. + +For a PostgreSQL workload it is recommended to set a "Guaranteed" QoS. + +To avoid resources related issues in Kubernetes, we can refer to the best practices for "out of resource" handling +while creating a cluster: + +- Specify your required values for memory and CPU in the resources section of the manifest file. + This way, you can avoid the `OOM Killed` (where "OOM" stands for Out Of Memory) and `CPU throttle` or any other + resources related issues on running instances. +- For your cluster's pods to get assigned to the "Guaranteed" QoS class, you must set limits and requests + for both memory and CPU to the same value. +- Specify your required PostgreSQL memory parameters consistently with the pod resources (as you would do + in a VM or physical machine scenario - see below). +- Set up database server pods on a dedicated node using nodeSelector. + See the ["nodeSelector field of the affinityconfiguration resource on the API reference page"](api_reference.md#affinityconfiguration). + +You can refer to the following example manifest: + +```yaml +apiVersion: postgresql.k8s.enterprisedb.io/v1 +kind: Cluster +metadata: + name: postgresql-resources +spec: + + instances: 3 + + postgresql: + parameters: + shared_buffers: "256MB" + + resources: + requests: + memory: "1024Mi" + cpu: 1 + limits: + memory: "1024Mi" + cpu: 1 + + storage: + size: 1Gi +``` + +In the above example, we have specified `shared_buffers` parameter with a value of `256MB` - i.e., how much memory is +dedicated to the PostgreSQL server for caching data (the default value for this parameter is `128MB` in case +it's not defined). + +A reasonable starting value for `shared_buffers` is 25% of the memory in your system. +For example: if your `shared_buffers` is 256 MB, then the recommended value for your container memory size is 1 GB, +which means that within a pod all the containers will have a total of 1 GB memory that Kubernetes will always preserve, +enabling our containers to work as expected. +For more details, please refer to the ["Resource Consumption"](https://www.postgresql.org/docs/current/runtime-config-resource.html) +section in the PostgreSQL documentation. + +!!! Seealso "Managing Compute Resources for Containers" + For more details on resource management, please refer to the + ["Managing Compute Resources for Containers"](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) + page from the Kubernetes documentation. diff --git a/temp_kubernetes/original/src/rolling_update.md b/temp_kubernetes/original/src/rolling_update.md index 57009b5aa7b..ec67e41756b 100644 --- a/temp_kubernetes/original/src/rolling_update.md +++ b/temp_kubernetes/original/src/rolling_update.md @@ -22,17 +22,17 @@ from the one with the highest serial. The primary is the last node to be upgraded. This operation is configurable and managed by the `primaryUpdateStrategy` option, accepting these two values: -- `switchover`: the rolling update process is managed by Kubernetes +- `unsupervised`: the rolling update process is managed by Kubernetes and is entirely automated, with the *switchover* operation starting once all the replicas have been upgraded -- `manual`: the rolling update process is suspended immediately +- `supervised`: the rolling update process is suspended immediately after all replicas have been upgraded and can only be completed with a manual switchover triggered by an administrator with `kubectl cnp promote [cluster] [pod]`. The plugin can be downloaded from the [`kubectl-cnp` project page](https://github.com/EnterpriseDB/kubectl-cnp) on GitHub. -The default and recommended value is `switchover`. +The default and recommended value is `unsupervised`. The upgrade keeps the Cloud Native PostgreSQL identity and does not re-clone the data. Pods will be deleted and created again with the same PVCs. diff --git a/temp_kubernetes/original/src/samples/cluster-example-monitoring.yaml b/temp_kubernetes/original/src/samples/cluster-example-monitoring.yaml new file mode 100644 index 00000000000..88edcf51db9 --- /dev/null +++ b/temp_kubernetes/original/src/samples/cluster-example-monitoring.yaml @@ -0,0 +1,235 @@ +apiVersion: postgresql.k8s.enterprisedb.io/v1 +kind: Cluster +metadata: + name: cluster-example +spec: + instances: 3 + + storage: + size: 1Gi + + monitoring: + customQueriesConfigMap: + - name: example-monitoring + key: custom-queries + customQueriesSecret: + - name: example-monitoring-secret + key: pg-database +--- +apiVersion: v1 +kind: ConfigMap +metadata: + namespace: default + name: example-monitoring +data: + custom-queries: | + pg_replication: + query: "SELECT CASE WHEN NOT pg_is_in_recovery() THEN 0 ELSE GREATEST (0, EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))) END AS lag" + primary: true + metrics: + - lag: + usage: "GAUGE" + description: "Replication lag behind primary in seconds" + + pg_postmaster: # wokeignore:rule=master + query: "SELECT pg_postmaster_start_time as start_time_seconds from pg_postmaster_start_time()" # wokeignore:rule=master + primary: true + metrics: + - start_time_seconds: + usage: "GAUGE" + description: "Time at which postmaster started" # wokeignore:rule=master + + pg_stat_user_tables: + query: | + SELECT + current_database() datname, + schemaname, + relname, + seq_scan, + seq_tup_read, + idx_scan, + idx_tup_fetch, + n_tup_ins, + n_tup_upd, + n_tup_del, + n_tup_hot_upd, + n_live_tup, + n_dead_tup, + n_mod_since_analyze, + COALESCE(last_vacuum, '1970-01-01Z') as last_vacuum, + COALESCE(last_autovacuum, '1970-01-01Z') as last_autovacuum, + COALESCE(last_analyze, '1970-01-01Z') as last_analyze, + COALESCE(last_autoanalyze, '1970-01-01Z') as last_autoanalyze, + vacuum_count, + autovacuum_count, + analyze_count, + autoanalyze_count + FROM + pg_stat_user_tables + metrics: + - datname: + usage: "LABEL" + description: "Name of current database" + - schemaname: + usage: "LABEL" + description: "Name of the schema that this table is in" + - relname: + usage: "LABEL" + description: "Name of this table" + - seq_scan: + usage: "COUNTER" + description: "Number of sequential scans initiated on this table" + - seq_tup_read: + usage: "COUNTER" + description: "Number of live rows fetched by sequential scans" + - idx_scan: + usage: "COUNTER" + description: "Number of index scans initiated on this table" + - idx_tup_fetch: + usage: "COUNTER" + description: "Number of live rows fetched by index scans" + - n_tup_ins: + usage: "COUNTER" + description: "Number of rows inserted" + - n_tup_upd: + usage: "COUNTER" + description: "Number of rows updated" + - n_tup_del: + usage: "COUNTER" + description: "Number of rows deleted" + - n_tup_hot_upd: + usage: "COUNTER" + description: "Number of rows HOT updated (i.e., with no separate index update required)" + - n_live_tup: + usage: "GAUGE" + description: "Estimated number of live rows" + - n_dead_tup: + usage: "GAUGE" + description: "Estimated number of dead rows" + - n_mod_since_analyze: + usage: "GAUGE" + description: "Estimated number of rows changed since last analyze" + - last_vacuum: + usage: "GAUGE" + description: "Last time at which this table was manually vacuumed (not counting VACUUM FULL)" + - last_autovacuum: + usage: "GAUGE" + description: "Last time at which this table was vacuumed by the autovacuum daemon" + - last_analyze: + usage: "GAUGE" + description: "Last time at which this table was manually analyzed" + - last_autoanalyze: + usage: "GAUGE" + description: "Last time at which this table was analyzed by the autovacuum daemon" + - vacuum_count: + usage: "COUNTER" + description: "Number of times this table has been manually vacuumed (not counting VACUUM FULL)" + - autovacuum_count: + usage: "COUNTER" + description: "Number of times this table has been vacuumed by the autovacuum daemon" + - analyze_count: + usage: "COUNTER" + description: "Number of times this table has been manually analyzed" + - autoanalyze_count: + usage: "COUNTER" + description: "Number of times this table has been analyzed by the autovacuum daemon" + + pg_statio_user_tables: + query: "SELECT current_database() datname, schemaname, relname, heap_blks_read, heap_blks_hit, idx_blks_read, idx_blks_hit, toast_blks_read, toast_blks_hit, tidx_blks_read, tidx_blks_hit FROM pg_statio_user_tables" + metrics: + - datname: + usage: "LABEL" + description: "Name of current database" + - schemaname: + usage: "LABEL" + description: "Name of the schema that this table is in" + - relname: + usage: "LABEL" + description: "Name of this table" + - heap_blks_read: + usage: "COUNTER" + description: "Number of disk blocks read from this table" + - heap_blks_hit: + usage: "COUNTER" + description: "Number of buffer hits in this table" + - idx_blks_read: + usage: "COUNTER" + description: "Number of disk blocks read from all indexes on this table" + - idx_blks_hit: + usage: "COUNTER" + description: "Number of buffer hits in all indexes on this table" + - toast_blks_read: + usage: "COUNTER" + description: "Number of disk blocks read from this table's TOAST table (if any)" + - toast_blks_hit: + usage: "COUNTER" + description: "Number of buffer hits in this table's TOAST table (if any)" + - tidx_blks_read: + usage: "COUNTER" + description: "Number of disk blocks read from this table's TOAST table indexes (if any)" + - tidx_blks_hit: + usage: "COUNTER" + description: "Number of buffer hits in this table's TOAST table indexes (if any)" + + pg_stat_activity: + query: | + WITH + metrics AS ( + SELECT + application_name, + SUM(EXTRACT(EPOCH FROM (CURRENT_TIMESTAMP - state_change))::bigint)::float AS process_idle_seconds_sum, + COUNT(*) AS process_idle_seconds_count + FROM pg_stat_activity + WHERE state = 'idle' + GROUP BY application_name + ), + buckets AS ( + SELECT + application_name, + le, + SUM( + CASE WHEN EXTRACT(EPOCH FROM (CURRENT_TIMESTAMP - state_change)) <= le + THEN 1 + ELSE 0 + END + )::bigint AS bucket + FROM + pg_stat_activity, + UNNEST(ARRAY[1, 2, 5, 15, 30, 60, 90, 120, 300]) AS le + GROUP BY application_name, le + ORDER BY application_name, le + ) + SELECT + application_name, + process_idle_seconds_sum, + process_idle_seconds_count, + ARRAY_AGG(le) AS process_idle_seconds, + ARRAY_AGG(bucket) AS process_idle_seconds_bucket + FROM metrics JOIN buckets USING (application_name) + GROUP BY 1, 2, 3 + metrics: + - application_name: + usage: "LABEL" + description: "Application Name" + - process_idle_seconds: + usage: "HISTOGRAM" + description: "Idle time of server processes" +--- +apiVersion: v1 +kind: Secret +metadata: + namespace: default + name: example-monitoring-secret +stringData: + pg-database: | + pg_database: + query: "SELECT pg_database.datname, pg_database_size(pg_database.datname) as size_bytes FROM pg_database" + primary: true + cache_seconds: 30 + metrics: + - datname: + usage: "LABEL" + description: "Name of the database" + - size_bytes: + usage: "GAUGE" + description: "Disk space used by the database" diff --git a/temp_kubernetes/original/src/security.md b/temp_kubernetes/original/src/security.md index 3dce0c06863..c2211205897 100644 --- a/temp_kubernetes/original/src/security.md +++ b/temp_kubernetes/original/src/security.md @@ -1,16 +1,23 @@ # Security -This section contains information about security for Cloud Native PostgreSQL, -from a few standpoints: source code, Kubernetes, and PostgreSQL. +This section contains information about security for Cloud Native PostgreSQL +analyzed at 3 different layers: Code, Container and Cluster. !!! Warning The information contained in this page must not exonerate you from - performing regular InfoSec duties on your Kubernetes cluster. + performing regular InfoSec duties on your Kubernetes cluster. Please + familiarize with the ["Overview of Cloud Native Security"](https://kubernetes.io/docs/concepts/security/overview/) + page from the Kubernetes documentation. + +!!! Seealso "About the 4C's Security Model" + Please refer to ["The 4C’s Security Model in Kubernetes"](https://www.enterprisedb.com/blog/4cs-security-model-kubernetes) + blog article to get a better understanding and context of the approach EDB + has taken with security in Cloud Native PostgreSQL. -## Source code static analysis +## Code Source code of Cloud Native PostgreSQL is *systematically scanned* for static analysis purposes, -including **security problems**, using a popular open-source for Go called +including **security problems**, using a popular open-source linter for Go called [GolangCI-Lint](https://github.com/golangci/golangci-lint) directly in the CI/CD pipeline. GolangCI-Lint can run several *linters* on the same source code. @@ -27,7 +34,39 @@ the code such as hard-coded credentials, integer overflows and SQL injections - Source code is also regularly inspected through [Coverity Scan by Synopsys](https://scan.coverity.com/) via EnterpriseDB's internal CI/CD pipeline. -## Kubernetes +## Container + +Every container image that is part of Cloud Native PostgreSQL is automatically built via CI/CD pipelines following every commit. +Such images include not only the operator's, but also the operands' - specifically every supported PostgreSQL and EDB Postgres Advanced version. +Within the pipelines, images are scanned with: + +- [Dockle](https://github.com/goodwithtech/dockle): for best practices in terms + of the container build process +- [Clair](https://github.com/quay/clair): for vulnerabilities found in both the + underlying operating system as well as libraries and applications that they run + +!!! Important + All operand images are automatically rebuilt once a day by our pipelines in case + of security updates at the base image and package level, providing **patch level updates** + for the container images that EDB distributes. + +The following guidelines and frameworks have been taken into account for container-level security: + +- the ["Container Image Creation and Deployment Guide"](https://dl.dod.cyber.mil/wp-content/uploads/devsecops/pdf/DevSecOps_Enterprise_Container_Image_Creation_and_Deployment_Guide_2.6-Public-Release.pdf), + developed by the Defense Information Systems Agency (DISA) of the United States Department of Defense (DoD) +- the ["CIS Benchmark for Docker"](https://www.cisecurity.org/benchmark/docker/), + developed by the Center for Internet Security (CIS) + +!!! Seealso "About the Container level security" + Please refer to ["Security and Containers in Cloud Native PostgreSQL"](https://www.enterprisedb.com/blog/security-and-containers-cloud-native-postgresql) + blog article for more information about the approach that EDB has taken on + security at container level in Cloud Native PostgreSQL. + +## Cluster + +Security at the cluster level takes into account all Kubernetes components that +form both the control plane and the nodes, as well as the applications that run in +the cluster (PostgreSQL included). ### Pod Security Policies @@ -52,100 +91,7 @@ Network policies are beyond the scope of this document. Please refer to the ["Network policies"](https://kubernetes.io/docs/concepts/services-networking/network-policies/) section of the Kubernetes documentation for further information. -### Resources - -In a typical Kubernetes cluster, containers run with unlimited resources. By default, -they might be allowed to use as much CPU and RAM as needed. - -Cloud Native PostgreSQL allows administrators to control and manage resource usage by the pods of the cluster, -through the `resources` section of the manifest, with two knobs: - -- `requests`: initial requirement -- `limits`: maximum usage, in case of dynamic increase of resource needs - -For example, you can request an initial amount of RAM of 32MiB (scalable to 128MiB) and 50m of CPU (scalable to 100m) as follows: - -```yaml - resources: - requests: - memory: "32Mi" - cpu: "50m" - limits: - memory: "128Mi" - cpu: "100m" -``` - -Memory requests and limits are associated with containers, but it is useful to think of a pod as having a memory request -and limit. The memory request for the pod is the sum of the memory requests for all the containers in the pod. - -Pod scheduling is based on requests and not limits. A pod is scheduled to run on a Node only if the Node has enough -available memory to satisfy the pod's memory request. - -For each resource, we divide containers into 3 Quality of Service (QoS) classes, in decreasing order of priority: - -- *Guaranteed* -- *Burstable* -- *Best-Effort* - -For more details, please refer to the ["Configure Quality of Service for Pods"](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#qos-classes) section in the Kubernetes documentation. - -For a PostgreSQL workload it is recommended to set a "Guaranteed" QoS. - -In order to avoid resources related issues in Kubernetes, we can refer to the best practices for "out of resource" handling while creating -a cluster: - -- Specify your required values for memory and CPU in the resources section of the manifest file. - This way you can avoid the `OOM Killed` (where "OOM" stands for Out Of Memory) and `CPU throttle` or any other resources - related issues on running instances. -- In order for the pods of your cluster to get assigned to the "Guaranteed" QoS class, you must set limits and requests - for both memory and CPU to the same value. -- Specify your required PostgreSQL memory parameters consistently with the pod resources (like you would do in a VM or physical machine scenario - see below). -- Set up database server pods on a dedicated node using nodeSelector. - See the ["nodeSelector field of the affinityconfiguration resource on the API reference page"](api_reference.md#affinityconfiguration). - -You can refer the following example manifest: - -```yaml -apiVersion: postgresql.k8s.enterprisedb.io/v1 -kind: Cluster -metadata: - name: postgresql-resources -spec: - - instances: 3 - - postgresql: - parameters: - shared_buffers: "256MB" - - resources: - requests: - memory: "1024Mi" - cpu: 1 - limits: - memory: "1024Mi" - cpu: 1 - - storage: - size: 1Gi -``` - -In the above example, we have specified `shared_buffers` parameter with a value of `256MB` - i.e. how much memory is -dedicated to the PostgreSQL server for caching data (the default value for this parameter is `128MB` in case it's not defined). - -A reasonable starting value for `shared_buffers` is 25% of the memory in your system. -For example: if your `shared_buffers` is 256 MB, then the recommended value for your container memory size is 1 GB, -which means that within a pod all the containers will have a total of 1 GB memory that Kubernetes will always preserve, -enabling our containers to work as expected. -For more details, please refer to the ["Resource Consumption"](https://www.postgresql.org/docs/current/runtime-config-resource.html) -section in the PostgreSQL documentation. - -!!! See also "Managing Compute Resources for Containers" - For more details on resource management, please refer to the - ["Managing Compute Resources for Containers"](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) - page from the Kubernetes documentation. - -## PostgreSQL +### PostgreSQL The current implementation of Cloud Native PostgreSQL automatically creates passwords and `.pgpass` files for the `postgres` superuser and the database owner.