diff --git a/.github/workflows/helm-chart-test.yml b/.github/workflows/helm-chart-test.yml index 887e6b5da..674031795 100644 --- a/.github/workflows/helm-chart-test.yml +++ b/.github/workflows/helm-chart-test.yml @@ -67,10 +67,10 @@ jobs: with: name: ${{ matrix.test-strategy }}_${{ env.CHART_FILE_NAME }} path: ${{ env.CHART_PACKAGE_PATH }} - - name: Upload Helm chart template rendered + - name: Upload chart test artifacts if: always() uses: actions/upload-artifact@v4 with: - name: ${{ matrix.test-strategy }}_chart_template_rendered.yaml - path: ./tests/tests/output_deployment.yaml + name: ${{ matrix.test-strategy }}-artifacts + path: ./tests/tests/ if-no-files-found: ignore diff --git a/Makefile b/Makefile index 5df2eb27e..f9e55dd2c 100644 --- a/Makefile +++ b/Makefile @@ -395,7 +395,7 @@ chart_test_edge: VERSION=$(TAG_VERSION) NAMESPACE=$(NAMESPACE) ./tests/charts/make/chart_test.sh NodeEdge chart_test_parallel_autoscaling: - VERSION=$(TAG_VERSION) NAMESPACE=$(NAMESPACE) ./tests/charts/make/chart_test.sh ParallelAutoscaling + VERSION=$(TAG_VERSION) NAMESPACE=$(NAMESPACE) ./tests/charts/make/chart_test.sh JobAutoscaling .PHONY: \ all \ diff --git a/charts/selenium-grid/README.md b/charts/selenium-grid/README.md index d14b83c3f..3568d4cd5 100644 --- a/charts/selenium-grid/README.md +++ b/charts/selenium-grid/README.md @@ -6,16 +6,21 @@ This chart enables the creation of a Selenium Grid Server in Kubernetes. * [Selenium-Grid Helm Chart](#selenium-grid-helm-chart) * [Contents](#contents) + * [Introduction](#introduction) * [Installing the chart](#installing-the-chart) * [Enable Selenium Grid Autoscaling](#enable-selenium-grid-autoscaling) - * [Settings when scaling with deployments](#settings-when-scaling-with-deployments-) + * [Settings common for both `job` and `deployment` scalingType](#settings-common-for-both-job-and-deployment-scalingtype) + * [Settings when scalingType with `deployment`](#settings-when-scalingtype-with-deployment-) + * [Settings when scalingType with `job`](#settings-when-scalingtype-with-job) * [Updating Selenium-Grid release](#updating-selenium-grid-release) * [Uninstalling Selenium Grid release](#uninstalling-selenium-grid-release) * [Ingress Configuration](#ingress-configuration) - * [References values](#references-values) * [Configuration](#configuration) * [Configuration global](#configuration-global) * [Configuration `global.K8S_PUBLIC_IP`](#configuration-globalk8spublicip) + * [Configuration of Nodes](#configuration-of-nodes) + * [Container ports and Service ports](#container-ports-and-service-ports) + * [Probes](#probes) * [Configuration of Selenium Grid chart](#configuration-of-selenium-grid-chart) * [Configuration of KEDA](#configuration-of-keda) * [Configuration of Ingress NGINX Controller](#configuration-of-ingress-nginx-controller) @@ -23,6 +28,13 @@ This chart enables the creation of a Selenium Grid Server in Kubernetes. * [Configuration for isolated components](#configuration-for-isolated-components) +## Introduction + +We offer a Helm chart to simplify the deployment of Selenium Grid Docker images to Kubernetes. +- Chart changes are tracked in [CHANGELOG](CHANGELOG.md). +- Sanity/Regression tests for the chart features are tracked in [TESTING](TESTING.md). +- There are some reference values file that used to test and deploy Selenium Grid chart. You can find them in [tests/charts/refValues](../../tests/charts/refValues) and [tests/charts/ci](../../tests/charts/ci). + ## Installing the chart If you want to install the latest master version of Selenium Grid onto your cluster you can do that by using the helm charts repository located at https://www.selenium.dev/docker-selenium. @@ -67,14 +79,82 @@ or [jobs](https://keda.sh/docs/latest/concepts/scaling-jobs/) and the charts sup chart support both modes. It is controlled with `autoscaling.scalingType` that can be set to either job (default) or deployment. -### Settings when scaling with deployments +### Settings common for both `job` and `deployment` scalingType + +There are few settings that are common for both scaling types. These are grouped under `autoscaling.scaledOptions`. + +In case individual node should be scaled differently, you can override the upstream settings with `.scaledOptions` for each node type. For example: + +```yaml +autoscaling: + scaledOptions: + minReplicaCount: 0 + maxReplicaCount: 8 + pollingInterval: 20 + +chromeNode: + scaledOptions: + minReplicaCount: 1 + maxReplicaCount: 16 + pollingInterval: 10 +``` + +### Settings when scalingType with `deployment` + +By default, `autoscaling.terminationGracePeriodSeconds` is set to 3600 seconds. This is used when scalingType is set to `deployment`. You can adjust this value, it will affect to all nodes. + +In case individual node which needs to set different period, you can override the upstream settings with `.terminationGracePeriodSeconds` for each node type. Note that override value must be greater than upstream setting to take effect. For example: + +```yaml +autoscaling: + terminationGracePeriodSeconds: 3600 #default +chromeNode: + terminationGracePeriodSeconds: 7200 #override +firefoxNode: + terminationGracePeriodSeconds: 1800 #not override +``` + +When scaling using deployments the HPA choose pods to terminate randomly. If the chosen pod is currently executing a test rather +than being idle, then there is `terminationGracePeriodSeconds` seconds before the test is expected to complete. If your test is +still executing after `terminationGracePeriodSeconds` seconds, it would result in failure as the pod will be killed. + +During `terminationGracePeriodSeconds` period, there is `preStop` hook to execute command to wait for the pod can be shut down gracefully which can be defined in `.deregisterLifecycle` +- There is a `_helpers` template with name `seleniumGrid.node.deregisterLifecycle` render value for pod `lifecycle.preStop`. By default, hook to execute the script to drain node and wait for current session to complete if any. The script is stored in node ConfigMap, more details can be seen in config `nodeConfigMap.` +- You can define your custom `preStop` hook which is applied for all nodes via `autoscaling.deregisterLifecycle` +- In case individual node which needs different hook, you can override the upstream settings with `.deregisterLifecycle` for each node type. If you want to disable upstream hook in a node, pass the value as `false` +- If an individual node has settings `.lifecycle` itself, it would take the highest precedence to override the above use cases. + +```yaml +autoscaling: + deregisterLifecycle: + preStop: + exec: + command: ["bash", "-c", "echo 'Your custom preStop hook applied for all nodes'"] +chromeNode: + deregisterLifecycle: false #disable upstream hook in chrome node +firefoxNode: + deregisterLifecycle: + preStop: + exec: + command: ["bash", "-c", "echo 'Your custom preStop hook specific for firefox node'"] +edgeNode: + lifecycle: + preStop: + exec: + command: ["bash", "-c", "echo 'preStop hook is defined in edge node lifecycle itself'"] +``` + +For other settings that KEDA [ScaledObject spec](https://keda.sh/docs/latest/concepts/scaling-deployments/#scaledobject-spec) supports, you can set them via `autoscaling.scaledObjectOptions`. For example: + +```yaml +autoscaling: + scaledObjectOptions: + cooldownPeriod: 60 +``` + +### Settings when scalingType with `job` -The `terminationGracePeriodSeconds` is set to 30 seconds by default. When scaling using deployments -the HPA choose pods to terminate randomly. If the chosen pod is currently executing a test rather -than being idle, then there is 30 seconds before the test is expected to complete. If your test is -still executing after 30 seconds, it would result in failure as the pod will be killed. If you want -to give more time for your tests to complete, you may set `terminationGracePeriodSeconds` to value -upto 3600 seconds. +Settings that KEDA [ScaledJob spec](https://keda.sh/docs/latest/concepts/scaling-jobs/#scaledjob-spec) supports can be set via `autoscaling.scaledJobOptions`. ## Updating Selenium-Grid release @@ -148,28 +228,22 @@ nginx.ingress.kubernetes.io/client-body-buffer-size nginx.ingress.kubernetes.io/proxy-buffers-number ``` -## Reference values - -There are some values file that used to test and deploy Selenium Grid chart. You can find them in -- [tests/charts/refValues](../../tests/charts/refValues). -- [tests/charts/ci](../../tests/charts/ci). - ## Configuration ### Configuration global For now, global configuration supported is: -| Parameter | Default | Description | -|---------------------------------------|-----------------------|---------------------------------------| -| `global.K8S_PUBLIC_IP` | `""` | Public IP of the host running K8s | -| `global.seleniumGrid.imageRegistry` | `selenium` | Distribution registry to pull images | -| `global.seleniumGrid.imageTag` | `4.16.1-20231219` | Image tag for all selenium components | -| `global.seleniumGrid.nodesImageTag` | `4.16.1-20231219` | Image tag for browser's nodes | -| `global.seleniumGrid.videoImageTag` | `ffmpeg-6.1-20231219` | Image tag for browser's video recoder | -| `global.seleniumGrid.imagePullSecret` | `""` | Pull secret to be used for all images | -| `global.seleniumGrid.imagePullSecret` | `""` | Pull secret to be used for all images | -| `global.seleniumGrid.affinity` | `{}` | Affinity assigned globally | -| `global.seleniumGrid.logLevel` | `INFO` | Set log level for all components | +| Parameter | Default | Description | +|---------------------------------------|-----------------------|----------------------------------------| +| `global.K8S_PUBLIC_IP` | `""` | Public IP of the host running K8s | +| `global.seleniumGrid.imageRegistry` | `selenium` | Distribution registry to pull images | +| `global.seleniumGrid.imageTag` | `4.16.1-20231219` | Image tag for all selenium components | +| `global.seleniumGrid.nodesImageTag` | `4.16.1-20231219` | Image tag for browser's nodes | +| `global.seleniumGrid.videoImageTag` | `ffmpeg-6.1-20231219` | Image tag for browser's video recorder | +| `global.seleniumGrid.imagePullSecret` | `""` | Pull secret to be used for all images | +| `global.seleniumGrid.imagePullSecret` | `""` | Pull secret to be used for all images | +| `global.seleniumGrid.affinity` | `{}` | Affinity assigned globally | +| `global.seleniumGrid.logLevel` | `INFO` | Set log level for all components | #### Configuration `global.K8S_PUBLIC_IP` @@ -196,6 +270,78 @@ SE_NODE_GRID_URL: 'http://admin:admin@10.10.10.10/selenium' ``` Besides that, from the outside of the cluster, you can access via NodePort http://10.10.10.10:30444/selenium +### Configuration of Nodes + +#### Container ports and Service ports + +By default, Node will use port `5555` to listen on container (following [this](https://www.selenium.dev/documentation/grid/configuration/cli_options/#server)) and expose via Service. You can update this value via `.port` in respective node type. This will be used to set `SE_NODE_PORT` environment variable to pass to option `--port` when starting the node and update in Service accordingly. + +By default, if httpGet probes are enabled, it will use `.port` value in respective node type unless you override it via e.g. `.startupProbe.port` `.readinessProbe.port` or `.livenessProbe.port` in respective node type. + +In a node container, there are other running services can be exposed. For example: VNC, NoVNC, SSH, etc. You can easily expose them on container via `.ports` and on Service `service.ports` in respective node type. + +```yaml +chromeNode: + port: 6666 # Update `SE_NODE_PORT` to 6666 + nodePort: 30666 # Specify a NodePort to expose `SE_NODE_PORT` to outside traffic + ports: + - 5900 # You can give port number alone, default protocol is TCP + - 7900 + service: + type: NodePort # Expose entire ports on Service via NodePort + ports: + - name: vnc-port + protocol: TCP + port: 5900 + targetPort: 5900 + nodePort: 30590 # Specify a NodePort to expose VNC port + - name: novnc-port + protocol: TCP + port: 7900 + targetPort: 7900 + # NodePort will be assigned randomly if not set +edgeNode: + ports: # You also can give object following manifest of container ports + - containerPort: 5900 + name: vnc + protocol: TCP + - containerPort: 7900 + name: novnc + protocol: TCP +``` + +#### Probes + +By default, `startupProbe` is enabled and `readinessProbe` and `livenessProbe` are disabled. You can enable/disable them via `.startupProbe.enabled` `.readinessProbe.enabled` `.livenessProbe.enabled` in respective node type. + +By default, probes are using `httpGet` method to check the node state. It will use `.port` value in respective node type unless you override it via e.g. `.startupProbe.port` `.readinessProbe.port` or `.livenessProbe.port` in respective node type. + +Other settings of probe support to override under `.startupProbe` `.readinessProbe` `.livenessProbe` in respective node type. + +```markdown + schema + path + port + initialDelaySeconds + failureThreshold + timeoutSeconds + periodSeconds + successThreshold +``` + +You can easily configure the probes (as Kubernetes [supports](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)) to override the default settings. For example: + +```yaml +edgeNode: + port: 5555 + startupProbe: + enabled: true + tcpSocket: + port: 5555 + failureThreshold: 10 + periodSeconds: 5 +``` + ### Configuration of Selenium Grid chart This table contains the configuration parameters of the chart and their default values: @@ -243,9 +389,9 @@ This table contains the configuration parameters of the chart and their default | `chromeNode.imageTag` | `4.16.1-20231219` | Image of chrome nodes | | `chromeNode.imagePullPolicy` | `IfNotPresent` | Image pull policy (see https://kubernetes.io/docs/concepts/containers/images/#updating-images) | | `chromeNode.imagePullSecret` | `""` | Image pull secret (see https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry) | -| `chromeNode.ports` | `[5555]` | Port list to enable on container | -| `chromeNode.seleniumPort` | `5900` | Selenium port (spec.ports[0].targetPort in kubernetes service) | -| `chromeNode.seleniumServicePort` | `6900` | Selenium port exposed in service (spec.ports[0].port in kubernetes service) | +| `chromeNode.ports` | `[]` | Extra ports list to enable on container (e.g VNC, NoVNC, SSH if any) | +| `chromeNode.port` | `5555` | Port is used to set `SE_NODE_PORT` | +| `chromeNode.nodePort` | `nil` | NodePort where chrome-node exposed | | `chromeNode.annotations` | `{}` | Annotations for chrome-node pods | | `chromeNode.labels` | `{}` | Labels for chrome-node pods | | `chromeNode.resources` | `See values.yaml` | Resources for chrome-node pods | @@ -263,8 +409,9 @@ This table contains the configuration parameters of the chart and their default | `chromeNode.service.ports` | `[]` | Extra ports exposed in node service | | `chromeNode.service.annotations` | `{}` | Custom annotations for service | | `chromeNode.dshmVolumeSizeLimit` | `1Gi` | Size limit for DSH volume mounted in container (if not set, default is "1Gi") | -| `chromeNode.startupProbe` | `{}` | Probe to check pod is started successfully | -| `chromeNode.livenessProbe` | `{}` | Liveness probe settings | +| `chromeNode.startupProbe.enabled` | `true` | Enable Probe to check pod is started successfully (the following configs see `values.yaml`) | +| `chromeNode.readinessProbe.enabled` | `false` | Enable Readiness probe settings (the following configs see `values.yaml`) | +| `chromeNode.livenessProbe.enabled` | `false` | Enable Liveness probe settings (the following configs see `values.yaml`) | | `chromeNode.terminationGracePeriodSeconds` | `30` | Time to graceful terminate container (default: 30s) | | `chromeNode.lifecycle` | `{}` | hooks to make pod correctly shutdown or started | | `chromeNode.extraVolumeMounts` | `[]` | Extra mounts of declared ExtraVolumes into pod | @@ -283,9 +430,9 @@ This table contains the configuration parameters of the chart and their default | `firefoxNode.imageTag` | `4.16.1-20231219` | Image of firefox nodes | | `firefoxNode.imagePullPolicy` | `IfNotPresent` | Image pull policy (see https://kubernetes.io/docs/concepts/containers/images/#updating-images) | | `firefoxNode.imagePullSecret` | `""` | Image pull secret (see https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry) | -| `firefoxNode.ports` | `[5555]` | Port list to enable on container | -| `firefoxNode.seleniumPort` | `5900` | Selenium port (spec.ports[0].targetPort in kubernetes service) | -| `firefoxNode.seleniumServicePort` | `6900` | Selenium port exposed in service (spec.ports[0].port in kubernetes service) | +| `firefoxNode.ports` | `[]` | Extra ports list to enable on container (e.g VNC, NoVNC, SSH if any) | +| `firefoxNode.port` | `5555` | Port is used to set `SE_NODE_PORT` | +| `firefoxNode.nodePort` | `nil` | NodePort where firefox-node exposed | | `firefoxNode.annotations` | `{}` | Annotations for firefox-node pods | | `firefoxNode.labels` | `{}` | Labels for firefox-node pods | | `firefoxNode.resources` | `See values.yaml` | Resources for firefox-node pods | @@ -303,8 +450,9 @@ This table contains the configuration parameters of the chart and their default | `firefoxNode.service.ports` | `[]` | Extra ports exposed in node service | | `firefoxNode.service.annotations` | `{}` | Custom annotations for service | | `firefoxNode.dshmVolumeSizeLimit` | `1Gi` | Size limit for DSH volume mounted in container (if not set, default is "1Gi") | -| `firefoxNode.startupProbe` | `{}` | Probe to check pod is started successfully | -| `firefoxNode.livenessProbe` | `{}` | Liveness probe settings | +| `firefoxNode.startupProbe.enabled` | `true` | Enable Probe to check pod is started successfully (the following configs see `values.yaml`) | +| `firefoxNode.readinessProbe.enabled` | `false` | Enable Readiness probe settings (the following configs see `values.yaml`) | +| `firefoxNode.livenessProbe.enabled` | `false` | Enable Liveness probe settings (the following configs see `values.yaml`) | | `firefoxNode.terminationGracePeriodSeconds` | `30` | Time to graceful terminate container (default: 30s) | | `firefoxNode.lifecycle` | `{}` | hooks to make pod correctly shutdown or started | | `firefoxNode.extraVolumeMounts` | `[]` | Extra mounts of declared ExtraVolumes into pod | @@ -323,9 +471,9 @@ This table contains the configuration parameters of the chart and their default | `edgeNode.imageTag` | `4.16.1-20231219` | Image of edge nodes | | `edgeNode.imagePullPolicy` | `IfNotPresent` | Image pull policy (see https://kubernetes.io/docs/concepts/containers/images/#updating-images) | | `edgeNode.imagePullSecret` | `""` | Image pull secret (see https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry) | -| `edgeNode.ports` | `[5555]` | Port list to enable on container | -| `edgeNode.seleniumPort` | `5900` | Selenium port (spec.ports[0].targetPort in kubernetes service) | -| `edgeNode.seleniumServicePort` | `6900` | Selenium port exposed in service (spec.ports[0].port in kubernetes service) | +| `edgeNode.ports` | `[]` | Extra ports list to enable on container (e.g VNC, NoVNC, SSH if any) | +| `edgeNode.port` | `5555` | Port is used to set `SE_NODE_PORT` | +| `edgeNode.nodePort` | `nil` | NodePort where edge-node exposed | | `edgeNode.annotations` | `{}` | Annotations for edge-node pods | | `edgeNode.labels` | `{}` | Labels for edge-node pods | | `edgeNode.resources` | `See values.yaml` | Resources for edge-node pods | @@ -343,8 +491,9 @@ This table contains the configuration parameters of the chart and their default | `edgeNode.service.ports` | `[]` | Extra ports exposed in node service | | `edgeNode.service.annotations` | `{}` | Custom annotations for service | | `edgeNode.dshmVolumeSizeLimit` | `1Gi` | Size limit for DSH volume mounted in container (if not set, default is "1Gi") | -| `edgeNode.startupProbe` | `{}` | Probe to check pod is started successfully | -| `edgeNode.livenessProbe` | `{}` | Liveness probe settings | +| `edgeNode.startupProbe.enabled` | `true` | Enable Probe to check pod is started successfully (the following configs see `values.yaml`) | +| `edgeNode.readinessProbe.enabled` | `false` | Enable Readiness probe settings (the following configs see `values.yaml`) | +| `edgeNode.livenessProbe.enabled` | `false` | Enable Liveness probe settings (the following configs see `values.yaml`) | | `edgeNode.terminationGracePeriodSeconds` | `30` | Time to graceful terminate container (default: 30s) | | `edgeNode.lifecycle` | `{}` | hooks to make pod correctly shutdown or started | | `edgeNode.extraVolumeMounts` | `[]` | Extra mounts of declared ExtraVolumes into pod | @@ -357,7 +506,7 @@ This table contains the configuration parameters of the chart and their default | `edgeNode.scaledObjectOptions` | See `values.yaml` | Override the global `autoscaling.scaledObjectOptions` with specific scaled options for edge nodes | | `videoRecorder.enabled` | `false` | Enable video recorder for node | | `videoRecorder.imageRegistry` | `nil` | Distribution registry to pull the image (this overwrites `.global.seleniumGrid.imageRegistry` value) | -| `videoRecorder.imageName` | `video` | Selenium video recoder image name | +| `videoRecorder.imageName` | `video` | Selenium video recorder image name | | `videoRecorder.imageTag` | `ffmpeg-6.1-20231219` | Image tag of video recorder | | `videoRecorder.imagePullPolicy` | `IfNotPresent` | Image pull policy (see https://kubernetes.io/docs/concepts/containers/images/#updating-images) | | `videoRecorder.uploader` | `false` | Name of the uploader to use. The value `false` is used to disable uploader. Supported default `s3` | diff --git a/charts/selenium-grid/TESTING.md b/charts/selenium-grid/TESTING.md index 7e94f85e4..fd29f09c1 100644 --- a/charts/selenium-grid/TESTING.md +++ b/charts/selenium-grid/TESTING.md @@ -10,7 +10,7 @@ All related testing to this helm chart will be documented in this file. | | Basic Auth is enabled | ✗ | | | Auto scaling | Auto scaling with `enableWithExistingKEDA` is `true` | ✓ | Cluster | | | Auto scaling with `scalingType` is `job` | ✓ | Cluster | -| | Auto scaling with `scalingType` is `deployment` | ✗ | | +| | Auto scaling with `scalingType` is `deployment` | ✓ | Cluster | | | Auto scaling with `autoscaling.scaledOptions.minReplicaCount` is `0` | ✓ | Cluster | | | Parallel tests execution against node autoscaling | ✓ | Cluster | | Ingress | Ingress is enabled without `hostname` | ✓ | Cluster | @@ -27,6 +27,10 @@ All related testing to this helm chart will be documented in this file. | | Components are able to set `.affinity` | ✓ | Template | | Tracing | Enable tracing via `SE_ENABLE_TRACING` | ✓ | Cluster | | | Disable tracing via `SE_ENABLE_TRACING` | ✓ | Cluster | +| `Node` component | `SE_NODE_PORT` can set a port different via `.port` | ✓ | Cluster | +| | Extra ports can be exposed on container via `.ports` | ✓ | Cluster | +| | Extra ports can be exposed on Service via `.service.ports` | ✓ | Cluster | +| | Service type change to `NodePort`, specific NodePort can be set | ✓ | Cluster | ## Test Chart Template - By using `helm template` command, the chart template is tested without installing it to Kubernetes cluster. diff --git a/charts/selenium-grid/templates/_helpers.tpl b/charts/selenium-grid/templates/_helpers.tpl index 7902d1938..baa2bc3be 100644 --- a/charts/selenium-grid/templates/_helpers.tpl +++ b/charts/selenium-grid/templates/_helpers.tpl @@ -80,6 +80,56 @@ Ingress fullname {{- default "selenium-ingress" .Values.ingress.nameOverride | trunc 63 | trimSuffix "-" -}} {{- end -}} +{{/* +Probe httpGet schema +*/}} +{{- define "seleniumGrid.probe.httpGet.schema" -}} +{{- "HTTP" -}} +{{- end -}} + +{{/* +Check user define custom probe method +*/}} +{{- define "seleniumGrid.probe.fromUserDefine" -}} +{{- $overrideProbe := dict -}} +{{- with .exec -}} +{{- $overrideProbe = dict "exec" . -}} +{{- end }} +{{- with .httpGet -}} +{{- $overrideProbe = dict "httpGet" . -}} +{{- end }} +{{- with .tcpSocket -}} +{{- $overrideProbe = dict "tcpSocket" . -}} +{{- end }} +{{- with .grpc -}} +{{- $overrideProbe = dict "grpc" . -}} +{{- end -}} +{{- $overrideProbe | toYaml -}} +{{- end -}} + +{{/* +Get probe settings +*/}} +{{- define "seleniumGrid.probe.settings" -}} +{{- $settings := dict -}} +{{- with .initialDelaySeconds -}} + {{- $settings = set $settings "initialDelaySeconds" . -}} +{{- end }} +{{- with .periodSeconds -}} + {{- $settings = set $settings "periodSeconds" . -}} +{{- end }} +{{- with .timeoutSeconds -}} + {{- $settings = set $settings "timeoutSeconds" . -}} +{{- end }} +{{- with .successThreshold -}} + {{- $settings = set $settings "successThreshold" . -}} +{{- end }} +{{- with .failureThreshold -}} + {{- $settings = set $settings "failureThreshold" . -}} +{{- end -}} +{{- $settings | toYaml -}} +{{- end -}} + {{- define "seleniumGrid.ingress.nginx.annotations.default" -}} {{- with .Values.ingress.nginx }} {{- with .proxyTimeout }} @@ -199,9 +249,12 @@ template: {{- $imageRegistry := default .Values.global.seleniumGrid.imageRegistry .node.imageRegistry }} image: {{ printf "%s/%s:%s" $imageRegistry .node.imageName $imageTag }} imagePullPolicy: {{ .node.imagePullPolicy }} - {{- with .node.extraEnvironmentVariables }} - env: {{- tpl (toYaml .) $ | nindent 10 }} - {{- end }} + env: + - name: SE_NODE_PORT + value: {{ .node.port | quote }} + {{- with .node.extraEnvironmentVariables }} + {{- tpl (toYaml .) $ | nindent 10 }} + {{- end }} envFrom: - configMapRef: name: {{ .Values.busConfigMap.name }} @@ -212,16 +265,26 @@ template: {{- with .node.extraEnvFrom }} {{- tpl (toYaml .) $ | nindent 10 }} {{- end }} - {{- if gt (len .node.ports) 0 }} ports: - {{- range .node.ports }} - - containerPort: {{ . }} + - containerPort: {{ .node.port }} + protocol: TCP + {{- if gt (len .node.ports) 0 }} + {{- $ports := .node.ports -}} + {{- if (regexMatch "[0-9]+$" (index $ports 0 | toString)) -}} + {{- range .node.ports }} + - containerPort: {{ . | int }} protocol: TCP + {{- end }} + {{- else -}} + {{- tpl (toYaml .node.ports) $ | nindent 10 }} {{- end }} {{- end }} volumeMounts: - name: dshm mountPath: /dev/shm + - name: {{ .Values.nodeConfigMap.scriptVolumeMountName }} + mountPath: /opt/selenium/{{ .Values.nodeConfigMap.preStopScript }} + subPath: {{ .Values.nodeConfigMap.preStopScript }} {{- if .node.extraVolumeMounts }} {{- tpl (toYaml .node.extraVolumeMounts) $ | nindent 10 }} {{- end }} @@ -231,12 +294,54 @@ template: {{- with .node.securityContext }} securityContext: {{- toYaml . | nindent 10 }} {{- end }} - {{- include "seleniumGrid.lifecycle" . | nindent 8 -}} + {{- include "seleniumGrid.node.lifecycle" . | nindent 8 -}} + {{- if .node.startupProbe.enabled }} {{- with .node.startupProbe }} - startupProbe: {{- toYaml . | nindent 10 }} + startupProbe: + {{- if (ne (include "seleniumGrid.probe.fromUserDefine" .) "{}") }} + {{- include "seleniumGrid.probe.fromUserDefine" . | nindent 10 }} + {{- else }} + httpGet: + scheme: {{ default (include "seleniumGrid.probe.httpGet.schema" .) .schema }} + path: {{ .path }} + port: {{ default ($.node.port) .port }} + {{- end }} + {{- if (ne (include "seleniumGrid.probe.settings" .) "{}") }} + {{- include "seleniumGrid.probe.settings" . | nindent 10 }} + {{- end }} + {{- end }} + {{- end }} + {{- if .node.readinessProbe.enabled }} + {{- with .node.readinessProbe }} + readinessProbe: + {{- if (ne (include "seleniumGrid.probe.fromUserDefine" .) "{}") }} + {{- include "seleniumGrid.probe.fromUserDefine" . | nindent 12 }} + {{- else }} + httpGet: + scheme: {{ default (include "seleniumGrid.probe.httpGet.schema" .) .schema }} + path: {{ .path }} + port: {{ default ($.node.port) .port }} + {{- end }} + {{- if (ne (include "seleniumGrid.probe.settings" .) "{}") }} + {{- include "seleniumGrid.probe.settings" . | nindent 10 }} + {{- end }} + {{- end }} {{- end }} + {{- if .node.livenessProbe.enabled }} {{- with .node.livenessProbe }} - livenessProbe: {{- toYaml . | nindent 10 }} + livenessProbe: + {{- if (ne (include "seleniumGrid.probe.fromUserDefine" .) "{}") }} + {{- include "seleniumGrid.probe.fromUserDefine" . | nindent 10 }} + {{- else }} + httpGet: + scheme: {{ default (include "seleniumGrid.probe.httpGet.schema" .) .schema }} + path: {{ .path }} + port: {{ default ($.node.port) .port }} + {{- end }} + {{- if (ne (include "seleniumGrid.probe.settings" .) "{}") }} + {{- include "seleniumGrid.probe.settings" . | nindent 10 }} + {{- end }} + {{- end }} {{- end }} {{- if .node.sidecars }} {{- toYaml .node.sidecars | nindent 6 }} @@ -325,6 +430,10 @@ template: {{- end }} terminationGracePeriodSeconds: {{ .node.terminationGracePeriodSeconds }} volumes: + - name: {{ .Values.nodeConfigMap.scriptVolumeMountName }} + configMap: + name: {{ .Values.nodeConfigMap.name }} + defaultMode: {{ .Values.nodeConfigMap.defaultMode }} - name: dshm emptyDir: medium: Memory @@ -425,19 +534,55 @@ Graphql unsafeSsl of the hub or the router {{- end -}} {{/* -Get the lifecycle of the pod. When KEDA is activated and the lifecycle is used for a pod of a -deployment preStop hook to deregister from the selenium hub. +Define preStop hook for the node pod. Node preStop script is stored in a ConfigMap and mounted as a volume. */}} -{{- define "seleniumGrid.lifecycle" }} -{{ $lifecycle := tpl (toYaml (default (dict) .node.lifecycle)) $ }} -{{- if and (eq .Values.autoscaling.scalingType "deployment") (eq (include "seleniumGrid.useKEDA" .) "true") -}} -{{ $lifecycle = merge ($lifecycle | fromYaml ) .Values.autoscaling.deregisterLifecycle | toYaml }} +{{- define "seleniumGrid.node.deregisterLifecycle" -}} +preStop: + exec: + command: ["bash", "-c", "/opt/selenium/{{ .Values.nodeConfigMap.preStopScript }}"] +{{- end -}} + +{{/* +Get the lifecycle of the pod is used for a Node to deregister from the Hub/Router. +1. IF KEDA is activated, scalingType is "deployment", and individual node deregisterLifecycle is not set, use autoscaling.deregisterLifecycle +2. ELSE (KEDA is not activated and node deregisterLifecycle is set), use .deregisterLifecycle in individual node +3. IF individual node with .lifecycle is set, it takes highest precedence to override the preStop in above use cases +*/}} +{{- define "seleniumGrid.node.lifecycle" }} +{{- $defaultDeregisterLifecycle := tpl (include "seleniumGrid.node.deregisterLifecycle" .) $ -}} +{{- $lifecycle := toYaml (dict) -}} +{{- if and (and (eq .Values.autoscaling.scalingType "deployment") (eq (include "seleniumGrid.useKEDA" .) "true")) (empty .node.deregisterLifecycle) -}} + {{- $lifecycle = merge ($lifecycle | fromYaml) (tpl (toYaml (default ($defaultDeregisterLifecycle | fromYaml) .Values.autoscaling.deregisterLifecycle)) $ | fromYaml) | toYaml -}} +{{- else -}} + {{- if eq (.node.deregisterLifecycle | toString | lower) "false" -}} + {{- $lifecycle = toYaml (dict) -}} + {{- else -}} + {{- $lifecycle = (tpl (toYaml (default ($defaultDeregisterLifecycle | fromYaml) .node.deregisterLifecycle) ) $ | fromYaml) | toYaml -}} + {{- end -}} +{{- end -}} +{{- if not (empty .node.lifecycle) -}} + {{- $lifecycle = mergeOverwrite ($lifecycle | fromYaml) (tpl (toYaml .node.lifecycle) $ | fromYaml) | toYaml -}} {{- end -}} {{ if and $lifecycle (ne $lifecycle "{}") -}} lifecycle: {{ $lifecycle | nindent 2 }} {{- end -}} {{- end -}} +{{/* +Define terminationGracePeriodSeconds of the node pod. +1. IF KEDA is activated, scalingType is "deployment", use autoscaling.terminationGracePeriodSeconds +2. IF node.terminationGracePeriodSeconds is greater than autoscaling.terminationGracePeriodSeconds, use node.terminationGracePeriodSeconds +*/}} +{{- define "seleniumGrid.node.terminationGracePeriodSeconds" -}} +{{- $autoscalingPeriod := default 0 .Values.autoscaling.terminationGracePeriodSeconds -}} +{{- $nodePeriod := default 0 .node.terminationGracePeriodSeconds -}} +{{- $period := $nodePeriod -}} +{{- if and (eq .Values.autoscaling.scalingType "deployment") (eq (include "seleniumGrid.useKEDA" .) "true") -}} + {{- $period = ternary $nodePeriod $autoscalingPeriod (gt $nodePeriod $autoscalingPeriod) -}} +{{- end -}} +{{- $period -}} +{{- end -}} + {{/* Default specs of VolumeMounts and Volumes for video recorder */}} diff --git a/charts/selenium-grid/templates/chrome-node-service.yaml b/charts/selenium-grid/templates/chrome-node-service.yaml index efe8f1873..a6965c3d0 100644 --- a/charts/selenium-grid/templates/chrome-node-service.yaml +++ b/charts/selenium-grid/templates/chrome-node-service.yaml @@ -22,8 +22,11 @@ spec: ports: - name: tcp-chrome protocol: TCP - port: {{ .Values.chromeNode.seleniumServicePort }} - targetPort: {{ .Values.chromeNode.seleniumPort }} + port: {{ .Values.chromeNode.port }} + targetPort: {{ .Values.chromeNode.port }} + {{- if and (eq $.Values.chromeNode.service.type "NodePort") .Values.chromeNode.nodePort }} + nodePort: {{ .Values.chromeNode.nodePort }} + {{- end }} {{- with .Values.chromeNode.service.ports }} {{- range . }} - name: {{ .name }} diff --git a/charts/selenium-grid/templates/edge-node-service.yaml b/charts/selenium-grid/templates/edge-node-service.yaml index d54dd70a3..896be0ac9 100644 --- a/charts/selenium-grid/templates/edge-node-service.yaml +++ b/charts/selenium-grid/templates/edge-node-service.yaml @@ -22,8 +22,11 @@ spec: ports: - name: tcp-edge protocol: TCP - port: {{ .Values.edgeNode.seleniumServicePort }} - targetPort: {{ .Values.edgeNode.seleniumPort }} + port: {{ .Values.edgeNode.port }} + targetPort: {{ .Values.edgeNode.port }} + {{- if and (eq $.Values.edgeNode.service.type "NodePort") .Values.edgeNode.nodePort }} + nodePort: {{ .Values.edgeNode.nodePort }} + {{- end }} {{- with .Values.edgeNode.service.ports }} {{- range . }} - name: {{ .name }} diff --git a/charts/selenium-grid/templates/event-bus-configmap.yaml b/charts/selenium-grid/templates/event-bus-configmap.yaml index 05f279bba..ad2b33756 100644 --- a/charts/selenium-grid/templates/event-bus-configmap.yaml +++ b/charts/selenium-grid/templates/event-bus-configmap.yaml @@ -1,4 +1,4 @@ -{{- $eventBusHost := ternary (include "seleniumGrid.eventBus.fullname" .) (include "seleniumGrid.hub.fullname" .) .Values.isolateComponents -}} +{{- $eventBusHost := printf "%s.%s" (ternary (include "seleniumGrid.eventBus.fullname" .) (include "seleniumGrid.hub.fullname" .) .Values.isolateComponents) (.Release.Namespace) -}} {{- $eventBusPublishPort := ternary .Values.components.eventBus.publishPort .Values.hub.publishPort .Values.isolateComponents -}} {{- $eventBusSubscribePort := ternary .Values.components.eventBus.subscribePort .Values.hub.subscribePort .Values.isolateComponents -}} apiVersion: v1 diff --git a/charts/selenium-grid/templates/firefox-node-service.yaml b/charts/selenium-grid/templates/firefox-node-service.yaml index 43ba40380..a0de36ca5 100644 --- a/charts/selenium-grid/templates/firefox-node-service.yaml +++ b/charts/selenium-grid/templates/firefox-node-service.yaml @@ -22,8 +22,11 @@ spec: ports: - name: tcp-firefox protocol: TCP - port: {{ .Values.firefoxNode.seleniumServicePort }} - targetPort: {{ .Values.firefoxNode.seleniumPort }} + port: {{ .Values.firefoxNode.port }} + targetPort: {{ .Values.firefoxNode.port }} + {{- if and (eq $.Values.firefoxNode.service.type "NodePort") .Values.firefoxNode.nodePort }} + nodePort: {{ .Values.firefoxNode.nodePort }} + {{- end }} {{- with .Values.firefoxNode.service.ports }} {{- range . }} - name: {{ .name }} diff --git a/charts/selenium-grid/templates/hub-deployment.yaml b/charts/selenium-grid/templates/hub-deployment.yaml index b9b28a446..71473b209 100644 --- a/charts/selenium-grid/templates/hub-deployment.yaml +++ b/charts/selenium-grid/templates/hub-deployment.yaml @@ -42,27 +42,53 @@ spec: protocol: TCP - containerPort: {{ .Values.hub.subscribePort }} protocol: TCP - {{- if .Values.hub.livenessProbe.enabled }} - livenessProbe: + {{- if .Values.hub.startupProbe.enabled }} + {{- with .Values.hub.startupProbe }} + startupProbe: + {{- if (ne (include "seleniumGrid.probe.fromUserDefine" .) "{}") }} + {{- include "seleniumGrid.probe.fromUserDefine" . | nindent 10 }} + {{- else }} httpGet: - path: {{ .Values.hub.livenessProbe.path }} - port: {{ .Values.hub.port }} - initialDelaySeconds: {{ .Values.hub.livenessProbe.initialDelaySeconds }} - periodSeconds: {{ .Values.hub.livenessProbe.periodSeconds }} - timeoutSeconds: {{ .Values.hub.livenessProbe.timeoutSeconds }} - successThreshold: {{ .Values.hub.livenessProbe.successThreshold }} - failureThreshold: {{ .Values.hub.livenessProbe.failureThreshold }} + scheme: {{ default (include "seleniumGrid.probe.httpGet.schema" .) .schema }} + path: {{ .path }} + port: {{ default ($.Values.hub.port) .port }} + {{- end }} + {{- if (ne (include "seleniumGrid.probe.settings" .) "{}") }} + {{- include "seleniumGrid.probe.settings" . | nindent 12 }} + {{- end }} + {{- end }} {{- end }} {{- if .Values.hub.readinessProbe.enabled }} + {{- with .Values.hub.readinessProbe }} readinessProbe: + {{- if (ne (include "seleniumGrid.probe.fromUserDefine" .) "{}") }} + {{- include "seleniumGrid.probe.fromUserDefine" . | nindent 10 }} + {{- else }} httpGet: - path: {{ .Values.hub.readinessProbe.path }} - port: {{ .Values.hub.port }} - initialDelaySeconds: {{ .Values.hub.readinessProbe.initialDelaySeconds }} - periodSeconds: {{ .Values.hub.readinessProbe.periodSeconds }} - timeoutSeconds: {{ .Values.hub.readinessProbe.timeoutSeconds }} - successThreshold: {{ .Values.hub.readinessProbe.successThreshold }} - failureThreshold: {{ .Values.hub.readinessProbe.failureThreshold }} + scheme: {{ default (include "seleniumGrid.probe.httpGet.schema" .) .schema }} + path: {{ .path }} + port: {{ default ($.Values.hub.port) .port }} + {{- end }} + {{- if (ne (include "seleniumGrid.probe.settings" .) "{}") }} + {{- include "seleniumGrid.probe.settings" . | nindent 12 }} + {{- end }} + {{- end }} + {{- end }} + {{- if .Values.hub.livenessProbe.enabled }} + {{- with .Values.hub.livenessProbe }} + livenessProbe: + {{- if (ne (include "seleniumGrid.probe.fromUserDefine" .) "{}") }} + {{- include "seleniumGrid.probe.fromUserDefine" . | nindent 10 }} + {{- else }} + httpGet: + scheme: {{ default (include "seleniumGrid.probe.httpGet.schema" .) .schema }} + path: {{ .path }} + port: {{ default ($.Values.hub.port) .port }} + {{- end }} + {{- if (ne (include "seleniumGrid.probe.settings" .) "{}") }} + {{- include "seleniumGrid.probe.settings" . | nindent 12 }} + {{- end }} + {{- end }} {{- end }} env: {{- with .Values.hub.subPath }} diff --git a/charts/selenium-grid/templates/node-configmap.yaml b/charts/selenium-grid/templates/node-configmap.yaml index da4edab40..807e2036c 100644 --- a/charts/selenium-grid/templates/node-configmap.yaml +++ b/charts/selenium-grid/templates/node-configmap.yaml @@ -14,3 +14,11 @@ metadata: data: SE_DRAIN_AFTER_SESSION_COUNT: '{{- and (eq (include "seleniumGrid.useKEDA" .) "true") (eq .Values.autoscaling.scalingType "job") | ternary "1" "0" -}}' SE_NODE_GRID_URL: '{{ include "seleniumGrid.url" .}}' + {{ .Values.nodeConfigMap.preStopScript }}: | + #!/bin/bash + if curl -sf 127.0.0.1:${SE_NODE_PORT}/status; then + curl -X POST 127.0.0.1:${SE_NODE_PORT}/se/grid/node/drain --header 'X-REGISTRATION-SECRET;' + while curl -sf 127.0.0.1:${SE_NODE_PORT}/status; do sleep 1; done + else + echo "Node is already drained. Shutting down gracefully!" + fi diff --git a/charts/selenium-grid/templates/router-deployment.yaml b/charts/selenium-grid/templates/router-deployment.yaml index cc0cbbf29..e02829d46 100644 --- a/charts/selenium-grid/templates/router-deployment.yaml +++ b/charts/selenium-grid/templates/router-deployment.yaml @@ -67,27 +67,54 @@ spec: ports: - containerPort: {{ .Values.components.router.port }} protocol: TCP - {{- if .Values.components.router.livenessProbe.enabled }} - livenessProbe: + {{- if .Values.components.router.startupProbe.enabled }} + {{- with .Values.components.router.startupProbe }} + startupProbe: + {{- if (ne (include "seleniumGrid.probe.fromUserDefine" .) "{}") }} + {{- include "seleniumGrid.probe.fromUserDefine" . | nindent 10 }} + {{- else }} httpGet: - path: {{ .Values.components.router.livenessProbe.path }} - port: {{ .Values.components.router.port }} - initialDelaySeconds: {{ .Values.components.router.livenessProbe.initialDelaySeconds }} - periodSeconds: {{ .Values.components.router.livenessProbe.periodSeconds }} - timeoutSeconds: {{ .Values.components.router.livenessProbe.timeoutSeconds }} - successThreshold: {{ .Values.components.router.livenessProbe.successThreshold }} - failureThreshold: {{ .Values.components.router.livenessProbe.failureThreshold }} + scheme: {{ default (include "seleniumGrid.probe.httpGet.schema" .) .schema }} + path: {{ .path }} + port: {{ default ($.Values.components.router.port) .port }} + {{- end }} + {{- if (ne (include "seleniumGrid.probe.settings" .) "{}") }} + {{- include "seleniumGrid.probe.settings" . | nindent 12 }} + {{- end }} + {{- end }} {{- end }} {{- if .Values.components.router.readinessProbe.enabled }} + {{- with .Values.components.router.readinessProbe }} readinessProbe: + {{- if (ne (include "seleniumGrid.probe.fromUserDefine" .) "{}") }} + {{- include "seleniumGrid.probe.fromUserDefine" . | nindent 10 }} + {{- else }} + httpGet: + scheme: {{ default (include "seleniumGrid.probe.httpGet.schema" .) .schema }} + path: {{ .path }} + port: {{ default ($.Values.components.router.port) .port }} + {{- end }} + {{- if (ne (include "seleniumGrid.probe.settings" .) "{}") }} + {{- include "seleniumGrid.probe.settings" . | nindent 12 }} + {{- end }} + {{- end }} + {{- end }} + {{- if .Values.components.router.livenessProbe.enabled }} + livenessProbe: + {{- with .Values.components.router.livenessProbe }} + livenessProbe: + {{- if (ne (include "seleniumGrid.probe.fromUserDefine" .) "{}") }} + {{- include "seleniumGrid.probe.fromUserDefine" . | nindent 10 }} + {{- else }} httpGet: - path: {{ .Values.components.router.readinessProbe.path }} - port: {{ .Values.components.router.port }} - initialDelaySeconds: {{ .Values.components.router.readinessProbe.initialDelaySeconds }} - periodSeconds: {{ .Values.components.router.readinessProbe.periodSeconds }} - timeoutSeconds: {{ .Values.components.router.readinessProbe.timeoutSeconds }} - successThreshold: {{ .Values.components.router.readinessProbe.successThreshold }} - failureThreshold: {{ .Values.components.router.readinessProbe.failureThreshold }} + scheme: {{ default (include "seleniumGrid.probe.httpGet.schema" .) .schema }} + path: {{ .path }} + port: {{ default ($.Values.components.router.port) .port }} + {{- end }} + {{- if (ne (include "seleniumGrid.probe.settings" .) "{}") }} + {{- include "seleniumGrid.probe.settings" . | nindent 12 }} + {{- end }} + {{- end }} {{- end }} {{- with .Values.components.router.resources }} resources: {{- toYaml . | nindent 12 }} diff --git a/charts/selenium-grid/values.yaml b/charts/selenium-grid/values.yaml index b6e7ef07c..f4a094c16 100644 --- a/charts/selenium-grid/values.yaml +++ b/charts/selenium-grid/values.yaml @@ -74,6 +74,12 @@ busConfigMap: # ConfigMap that contains common environment variables for browser nodes nodeConfigMap: name: selenium-node-config + # Default mode for ConfigMap is mounted as file + defaultMode: 0755 + # File name of preStop script in ConfigMap + preStopScript: nodePreStop.sh + # Name of volume mount is used to mount scripts in the ConfigMap + scriptVolumeMountName: node-helper-scripts # Custom annotations for configmap annotations: {} @@ -104,11 +110,11 @@ components: # Router port port: 4444 nodePort: 30444 - # Liveness probe settings - livenessProbe: + # Wait for pod startup + startupProbe: enabled: true path: /readyz - initialDelaySeconds: 10 + initialDelaySeconds: 5 failureThreshold: 10 timeoutSeconds: 10 periodSeconds: 10 @@ -122,6 +128,15 @@ components: timeoutSeconds: 10 periodSeconds: 10 successThreshold: 1 + # Liveness probe settings + livenessProbe: + enabled: true + path: /readyz + initialDelaySeconds: 10 + failureThreshold: 10 + timeoutSeconds: 10 + periodSeconds: 10 + successThreshold: 1 # Resources for router container resources: {} # SecurityContext for router container @@ -319,11 +334,11 @@ hub: # Selenium Hub port port: 4444 nodePort: 31444 - # Liveness probe settings - livenessProbe: + # Wait for pod startup + startupProbe: enabled: true path: /readyz - initialDelaySeconds: 10 + initialDelaySeconds: 5 failureThreshold: 10 timeoutSeconds: 10 periodSeconds: 10 @@ -337,6 +352,15 @@ hub: timeoutSeconds: 10 periodSeconds: 10 successThreshold: 1 + # Liveness probe settings + livenessProbe: + enabled: true + path: /readyz + initialDelaySeconds: 10 + failureThreshold: 10 + timeoutSeconds: 10 + periodSeconds: 10 + successThreshold: 1 # Custom sub path for the hub deployment subPath: "" # Custom environment variables for selenium-hub @@ -418,15 +442,13 @@ autoscaling: scaledObjectOptions: scaleTargetRef: kind: Deployment + # Define terminationGracePeriodSeconds for scalingType "deployment". Period for `deregisterLifecycle` to gracefully shut down the node before force killing it + terminationGracePeriodSeconds: 3600 + # Define preStop command to shuts down the node gracefully when scalingType is set to "deployment" deregisterLifecycle: - preStop: - exec: - command: - - bash - - -c - - | - curl -X POST 127.0.0.1:5555/se/grid/node/drain --header 'X-REGISTRATION-SECRET;' && \ - while curl 127.0.0.1:5555/status; do sleep 1; done; + # preStop: + # exec: + # command: [ "bash", "-c", "/opt/selenium/nodePreStop.sh" ] # Configuration for chrome nodes chromeNode: @@ -451,13 +473,13 @@ chromeNode: # Image pull secret (see https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) imagePullSecret: "" - # Port list to enable on container - ports: - - 5555 - # Selenium port (spec.ports[0].targetPort in kubernetes service) - seleniumPort: 5900 - # Selenium port exposed in service (spec.ports[0].port in kubernetes service) - seleniumServicePort: 6900 + # Extra ports list to enable on the node container (e.g. SSH, VNC, NoVNC, etc.) + ports: [] + # - 5900 + # - 7900 + # Node component port + port: 5555 + nodePort: # Annotations for chrome-node pods annotations: {} # Labels for chrome-node pods @@ -510,9 +532,9 @@ chromeNode: loadBalancerIP: "" # Extra ports exposed in node service ports: - # - name: node-port - # port: 5555 - # targetPort: 5555 + # - name: vnc-port + # port: 5900 + # targetPort: 5900 # Custom annotations for service annotations: {} # Size limit for DSH volume mounted in container (if not set, default is "1Gi") @@ -521,18 +543,40 @@ chromeNode: priorityClassName: "" # Wait for pod startup - startupProbe: {} - # httpGet: - # path: /status - # port: 5555 - # failureThreshold: 120 - # periodSeconds: 5 + startupProbe: + enabled: true + path: /status + initialDelaySeconds: 0 + failureThreshold: 10 + timeoutSeconds: 5 + periodSeconds: 5 + successThreshold: 1 + + # Readiness probe settings + readinessProbe: + enabled: false + path: /status + initialDelaySeconds: 10 + failureThreshold: 10 + timeoutSeconds: 10 + periodSeconds: 10 + successThreshold: 1 # Liveness probe settings - livenessProbe: {} + livenessProbe: + enabled: false + path: /status + initialDelaySeconds: 15 + failureThreshold: 10 + timeoutSeconds: 10 + periodSeconds: 10 + successThreshold: 1 # Time to wait for pod termination terminationGracePeriodSeconds: 30 + # Define preStop command to shuts down the chrome node gracefully. This overwrites autoscaling.deregisterLifecycle + deregisterLifecycle: + # Define postStart and preStop events. This overwrites the defined preStop in deregisterLifecycle if any lifecycle: {} extraVolumeMounts: [] # - name: my-extra-volume @@ -585,13 +629,13 @@ firefoxNode: # Image pull secret (see https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) imagePullSecret: "" - # Port list to enable on container - ports: - - 5555 - # Selenium port (spec.ports[0].targetPort in kubernetes service) - seleniumPort: 5900 - # Selenium port exposed in service (spec.ports[0].port in kubernetes service) - seleniumServicePort: 6900 + # Extra ports list to enable on the node container (e.g. SSH, VNC, NoVNC, etc.) + ports: [] + # - 5900 + # - 7900 + # Node component port + port: 5555 + nodePort: # Annotations for firefox-node pods annotations: {} # Labels for firefox-node pods @@ -644,9 +688,9 @@ firefoxNode: loadBalancerIP: "" # Extra ports exposed in node service ports: - # - name: node-port - # port: 5555 - # targetPort: 5555 + # - name: vnc-port + # port: 5900 + # targetPort: 5900 # Custom annotations for service annotations: {} # Size limit for DSH volume mounted in container (if not set, default is "1Gi") @@ -655,18 +699,40 @@ firefoxNode: priorityClassName: "" # Wait for pod startup - startupProbe: {} - # httpGet: - # path: /status - # port: 5555 - # failureThreshold: 120 - # periodSeconds: 5 + startupProbe: + enabled: true + path: /status + initialDelaySeconds: 0 + failureThreshold: 10 + timeoutSeconds: 5 + periodSeconds: 5 + successThreshold: 1 + + # Readiness probe settings + readinessProbe: + enabled: false + path: /status + initialDelaySeconds: 10 + failureThreshold: 10 + timeoutSeconds: 10 + periodSeconds: 10 + successThreshold: 1 # Liveness probe settings - livenessProbe: {} + livenessProbe: + enabled: false + path: /status + initialDelaySeconds: 15 + failureThreshold: 10 + timeoutSeconds: 10 + periodSeconds: 10 + successThreshold: 1 # Time to wait for pod termination terminationGracePeriodSeconds: 30 + # Define preStop command to shuts down the chrome node gracefully. This overwrites autoscaling.deregisterLifecycle + deregisterLifecycle: + # Define postStart and preStop events. This overwrites the defined preStop in deregisterLifecycle if any lifecycle: {} extraVolumeMounts: [] # - name: my-extra-volume @@ -717,12 +783,13 @@ edgeNode: # Image pull secret (see https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) imagePullSecret: "" - ports: - - 5555 - # Selenium port (spec.ports[0].targetPort in kubernetes service) - seleniumPort: 5900 - # Selenium port exposed in service (spec.ports[0].port in kubernetes service) - seleniumServicePort: 6900 + # Extra ports list to enable on the node container (e.g. SSH, VNC, NoVNC, etc.) + ports: [] + # - 5900 + # - 7900 + # Node component port + port: 5555 + nodePort: # Annotations for edge-node pods annotations: {} # Labels for edge-node pods @@ -775,9 +842,9 @@ edgeNode: loadBalancerIP: "" # Extra ports exposed in node service ports: - # - name: node-port - # port: 5555 - # targetPort: 5555 + # - name: vnc-port + # port: 5900 + # targetPort: 5900 # Custom annotations for service annotations: {} # Size limit for DSH volume mounted in container (if not set, default is "1Gi") @@ -786,18 +853,40 @@ edgeNode: priorityClassName: "" # Wait for pod startup - startupProbe: {} - # httpGet: - # path: /status - # port: 5555 - # failureThreshold: 120 - # periodSeconds: 5 + startupProbe: + enabled: true + path: /status + initialDelaySeconds: 0 + failureThreshold: 10 + timeoutSeconds: 5 + periodSeconds: 5 + successThreshold: 1 + + # Readiness probe settings + readinessProbe: + enabled: false + path: /status + initialDelaySeconds: 10 + failureThreshold: 10 + timeoutSeconds: 10 + periodSeconds: 10 + successThreshold: 1 # Liveness probe settings - livenessProbe: {} + livenessProbe: + enabled: false + path: /status + initialDelaySeconds: 15 + failureThreshold: 10 + timeoutSeconds: 10 + periodSeconds: 10 + successThreshold: 1 # Time to wait for pod termination terminationGracePeriodSeconds: 30 + # Define preStop command to shuts down the chrome node gracefully. This overwrites autoscaling.deregisterLifecycle + deregisterLifecycle: + # Define postStart and preStop events. This overwrites the defined preStop in deregisterLifecycle if any lifecycle: {} extraVolumeMounts: [] # - name: my-extra-volume diff --git a/tests/SeleniumTests/__init__.py b/tests/SeleniumTests/__init__.py index fa95acefa..4ca352683 100644 --- a/tests/SeleniumTests/__init__.py +++ b/tests/SeleniumTests/__init__.py @@ -127,7 +127,7 @@ def test_title_and_maximize_window(self): self.driver.maximize_window() self.assertTrue(self.driver.title == 'The Internet') -class ParallelAutoscaling(): +class JobAutoscaling(): def run(self, test_classes): with concurrent.futures.ThreadPoolExecutor() as executor: futures = [] @@ -138,7 +138,7 @@ def run(self, test_classes): for future in concurrent.futures.as_completed(futures): future.result() -class ParallelAutoscalingTests(unittest.TestCase): +class JobAutoscalingTests(unittest.TestCase): def test_parallel_autoscaling(self): - runner = ParallelAutoscaling() + runner = JobAutoscaling() runner.run([ChromeTests, EdgeTests, FirefoxTests]) diff --git a/tests/charts/bootstrap.sh b/tests/charts/bootstrap.sh index d961b63ae..f1ade17d1 100755 --- a/tests/charts/bootstrap.sh +++ b/tests/charts/bootstrap.sh @@ -13,9 +13,9 @@ python -m pip install pyyaml==6.0.1 \ cd .. helm template dummy --values tests/charts/templates/render/dummy.yaml \ - charts/selenium-grid > ./tests/tests/output_deployment.yaml + charts/selenium-grid > ./tests/tests/dummy_template_manifests.yaml -python tests/charts/templates/test.py "./tests/tests/output_deployment.yaml" +python tests/charts/templates/test.py "./tests/tests/dummy_template_manifests.yaml" ret_code=$? if [ "${CI:-false}" = "false" ]; then diff --git a/tests/charts/ci/DeploymentAutoScaling-values.yaml b/tests/charts/ci/DeploymentAutoScaling-values.yaml new file mode 100644 index 000000000..b9e5b120e --- /dev/null +++ b/tests/charts/ci/DeploymentAutoScaling-values.yaml @@ -0,0 +1,72 @@ +autoscaling: + enableWithExistingKEDA: true + scalingType: deployment + scaledOptions: + minReplicaCount: 0 + maxReplicaCount: 3 + scaledObjectOptions: + cooldownPeriod: 30 + terminationGracePeriodSeconds: 360 +# Configuration for chrome nodes +chromeNode: + # (test): user is able to change `SE_NODE_PORT` + port: 6666 + # (test): user is able to set NodePort to expose `SE_NODE_PORT` + nodePort: 30666 + # (test): user is able to define list extra container ports + ports: + - 5900 + - 7900 + service: + type: NodePort + # (test): user is able to define extra ports for Node service + ports: + - name: vnc-port + protocol: TCP + port: 5900 + targetPort: 5900 + nodePort: 30590 + - name: novnc-port + protocol: TCP + port: 7900 + targetPort: 7900 + # NodePort will be assigned randomly if not set + nameOverride: my-chrome-name + extraEnvironmentVariables: &extraEnvironmentVariables + - name: SE_OPTS + value: "--enable-managed-downloads true" + - name: SE_DRAIN_AFTER_SESSION_COUNT + value: "0" + readinessProbe: + enabled: &readinessProbe true + livenessProbe: + enabled: &livenessProbe true +# Configuration for edge nodes +edgeNode: + # (test): user is able to define extra container ports + ports: + - containerPort: 5900 + name: vnc + protocol: TCP + - containerPort: 7900 + name: novnc + protocol: TCP + nameOverride: my-edge-name + extraEnvironmentVariables: *extraEnvironmentVariables + # (test): user is able to override probe method + startupProbe: + enabled: true + tcpSocket: + port: 8888 + readinessProbe: + enabled: *readinessProbe + livenessProbe: + enabled: *livenessProbe +# Configuration for firefox nodes +firefoxNode: + nameOverride: my-firefox-name + extraEnvironmentVariables: *extraEnvironmentVariables + readinessProbe: + enabled: *readinessProbe + livenessProbe: + enabled: *livenessProbe diff --git a/tests/charts/ci/ParallelAutoscaling-values.yaml b/tests/charts/ci/JobAutoscaling-values.yaml similarity index 67% rename from tests/charts/ci/ParallelAutoscaling-values.yaml rename to tests/charts/ci/JobAutoscaling-values.yaml index 78302b94f..d8fc0bc79 100644 --- a/tests/charts/ci/ParallelAutoscaling-values.yaml +++ b/tests/charts/ci/JobAutoscaling-values.yaml @@ -1,26 +1,43 @@ isolateComponents: false + autoscaling: + enableWithExistingKEDA: true + scalingType: job strategy: default scaledOptions: minReplicaCount: 0 maxReplicaCount: 5 + pollingInterval: 20 +# Configuration for chrome nodes chromeNode: nameOverride: my-chrome-name extraEnvironmentVariables: - name: SE_OPTS value: "--enable-managed-downloads true" + readinessProbe: + enabled: &readinessProbe false + livenessProbe: + enabled: &livenessProbe false # Configuration for edge nodes edgeNode: nameOverride: my-edge-name extraEnvironmentVariables: - name: SE_OPTS value: "--enable-managed-downloads true" + readinessProbe: + enabled: *readinessProbe + livenessProbe: + enabled: *livenessProbe # Configuration for firefox nodes firefoxNode: nameOverride: my-firefox-name extraEnvironmentVariables: - name: SE_OPTS value: "--enable-managed-downloads true" + readinessProbe: + enabled: *readinessProbe + livenessProbe: + enabled: *livenessProbe ingress: paths: diff --git a/tests/charts/ci/NodeChrome-values.yaml b/tests/charts/ci/NodeChrome-values.yaml index 09396f42d..d7c9df098 100644 --- a/tests/charts/ci/NodeChrome-values.yaml +++ b/tests/charts/ci/NodeChrome-values.yaml @@ -1,6 +1,7 @@ # This is used in Helm chart testing # Configuration for chrome nodes chromeNode: + port: 6666 nameOverride: my-chrome-name extraEnvironmentVariables: - name: SE_OPTS diff --git a/tests/charts/ci/NodeEdge-values.yaml b/tests/charts/ci/NodeEdge-values.yaml index 27220e275..7c78ccbe7 100644 --- a/tests/charts/ci/NodeEdge-values.yaml +++ b/tests/charts/ci/NodeEdge-values.yaml @@ -4,6 +4,7 @@ chromeNode: enabled: false # Configuration for edge nodes edgeNode: + port: 8888 nameOverride: my-edge-name extraEnvironmentVariables: - name: SE_OPTS diff --git a/tests/charts/ci/NodeFirefox-values.yaml b/tests/charts/ci/NodeFirefox-values.yaml index 21bb6a285..72cce27c5 100644 --- a/tests/charts/ci/NodeFirefox-values.yaml +++ b/tests/charts/ci/NodeFirefox-values.yaml @@ -7,6 +7,7 @@ edgeNode: enabled: false # Configuration for firefox nodes firefoxNode: + port: 7777 nameOverride: my-firefox-name extraEnvironmentVariables: - name: SE_OPTS diff --git a/tests/charts/ci/autoscaling-values.yaml b/tests/charts/ci/autoscaling-values.yaml deleted file mode 100644 index 7fa80b2bb..000000000 --- a/tests/charts/ci/autoscaling-values.yaml +++ /dev/null @@ -1,5 +0,0 @@ -autoscaling: - enableWithExistingKEDA: true - scalingType: job - scaledOptions: - minReplicaCount: 0 diff --git a/tests/charts/make/chart_test.sh b/tests/charts/make/chart_test.sh index 2d87c799e..194bb6056 100755 --- a/tests/charts/make/chart_test.sh +++ b/tests/charts/make/chart_test.sh @@ -1,4 +1,6 @@ #!/bin/bash +mkdir -p tests/tests +set -o xtrace echo "Set ENV variables" CLUSTER_NAME=${CLUSTER_NAME:-"chart-testing"} @@ -31,7 +33,7 @@ cleanup() { on_failure() { local exit_status=$? echo "Describe all resources in the ${SELENIUM_NAMESPACE} namespace for debugging purposes" - kubectl describe all -n ${SELENIUM_NAMESPACE} + kubectl describe all -n ${SELENIUM_NAMESPACE} > tests/tests/describe_all_resources_${MATRIX_BROWSER}.txt echo "There is step failed with exit status $exit_status" cleanup exit $exit_status @@ -42,20 +44,25 @@ trap 'on_failure' ERR HELM_COMMAND_SET_AUTOSCALING="" if [ "${SELENIUM_GRID_AUTOSCALING}" = "true" ]; then - HELM_COMMAND_SET_AUTOSCALING="--values ${TEST_VALUES_PATH}/autoscaling-values.yaml \ + HELM_COMMAND_SET_AUTOSCALING="--values ${TEST_VALUES_PATH}/DeploymentAutoScaling-values.yaml \ --set autoscaling.enableWithExistingKEDA=${SELENIUM_GRID_AUTOSCALING} \ --set autoscaling.scaledOptions.minReplicaCount=${SELENIUM_GRID_AUTOSCALING_MIN_REPLICA}" fi -echo "Deploy Selenium Grid Chart" -helm upgrade --install ${RELEASE_NAME} \ +HELM_COMMAND_ARGS="${RELEASE_NAME} \ --values ${TEST_VALUES_PATH}/auth-ingress-values.yaml \ --values ${TEST_VALUES_PATH}/tracing-values.yaml \ --values ${TEST_VALUES_PATH}/${MATRIX_BROWSER}-values.yaml \ ${HELM_COMMAND_SET_AUTOSCALING} \ --set global.seleniumGrid.imageTag=${VERSION} --set global.seleniumGrid.imageRegistry=${NAMESPACE} \ --set global.seleniumGrid.nodesImageTag=${VERSION} \ -${CHART_PATH} --namespace ${SELENIUM_NAMESPACE} --create-namespace +${CHART_PATH} --namespace ${SELENIUM_NAMESPACE} --create-namespace" + +echo "Render manifests YAML for this deployment" +helm template ${HELM_COMMAND_ARGS} > tests/tests/cluster_deployment_manifests_${MATRIX_BROWSER}.yaml + +echo "Deploy Selenium Grid Chart" +helm upgrade --install ${HELM_COMMAND_ARGS} echo "Run Tests" export SELENIUM_GRID_HOST=${SELENIUM_GRID_HOST} @@ -71,6 +78,6 @@ echo "Get pods status" kubectl get pods -n ${SELENIUM_NAMESPACE} echo "Get all resources in the ${SELENIUM_NAMESPACE} namespace" -kubectl get all -n ${SELENIUM_NAMESPACE} +kubectl get all -n ${SELENIUM_NAMESPACE} > tests/tests/describe_all_resources_${MATRIX_BROWSER}.txt cleanup diff --git a/tests/charts/refValues/sample-aws.yaml b/tests/charts/refValues/sample-aws.yaml index d32d101cf..06172f0e3 100644 --- a/tests/charts/refValues/sample-aws.yaml +++ b/tests/charts/refValues/sample-aws.yaml @@ -60,20 +60,12 @@ chromeNode: value: "true" - name: SE_OPTS value: "--enable-managed-downloads true" - startupProbe: &nodeStartupProbe - httpGet: - path: /status - port: 5555 - failureThreshold: 120 - periodSeconds: 1 firefoxNode: extraEnvironmentVariables: *extraEnvironmentVariablesNodes - startupProbe: *nodeStartupProbe edgeNode: extraEnvironmentVariables: *extraEnvironmentVariablesNodes - startupProbe: *nodeStartupProbe videoRecorder: enabled: false diff --git a/tests/charts/refValues/simplex-minikube.yaml b/tests/charts/refValues/simplex-minikube.yaml index e491deda3..5239714c7 100644 --- a/tests/charts/refValues/simplex-minikube.yaml +++ b/tests/charts/refValues/simplex-minikube.yaml @@ -33,7 +33,7 @@ ingress: number: 4444 basicAuth: - enabled: true + enabled: false isolateComponents: true @@ -69,20 +69,12 @@ chromeNode: value: "true" - name: SE_OPTS value: "--enable-managed-downloads true" - startupProbe: &nodeStartupProbe - httpGet: - path: /status - port: 5555 - failureThreshold: 120 - periodSeconds: 1 firefoxNode: extraEnvironmentVariables: *extraEnvironmentVariablesNodes - startupProbe: *nodeStartupProbe edgeNode: extraEnvironmentVariables: *extraEnvironmentVariablesNodes - startupProbe: *nodeStartupProbe videoRecorder: enabled: false diff --git a/tests/test.py b/tests/test.py index b8b462c2b..5be6fbe8c 100644 --- a/tests/test.py +++ b/tests/test.py @@ -57,7 +57,7 @@ 'StandaloneFirefox': 'FirefoxTests', # Chart Parallel Test - 'ParallelAutoscaling': 'ParallelAutoscalingTests' + 'JobAutoscaling': 'JobAutoscalingTests' } FROM_IMAGE_ARGS = {