Customize the Prometheus export port

[Yicheng-Lu-llll]

Why are these changes needed?

Currently, KubeRay always exposes a Prometheus metrics endpoint on port 8080, and users are unable to customize this export port (see issue #864). However, the Ray documentation indicates that users are able to set up a custom Prometheus export port.

To better meet the needs of users and maintain consistency with Ray's capabilities, I recommend enabling the customization of the Prometheus export port for KubeRay users.

How to set Prometheus export port ?

• Add metrics-export-port: "{Port Number}" to rayStartParams for head and worker pods.
• Add containerPort: {Port Number} to template.spec.containers[0].ports for head and worker pods.
  Note, It's user's responsibility to maintain rayStartParam ports and container ports mapping(see here)

For example:
(full version can be found here)

apiVersion: ray.io/v1alpha1
kind: RayCluster
spec:
  headGroupSpec:
    rayStartParams:
      metrics-export-port: "9001"
    template:
      spec:
        containers:
        - name: ray-head
          image: rayproject/ray:2.3.0
          ports:
          - containerPort: 9001
            name: metrics
            protocol: TCP

  workerGroupSpecs:
  - replicas: 1
    minReplicas: 1
    maxReplicas: 10
    rayStartParams:
      metrics-export-port: "9001" 
    template:
      spec:
        containers:
        - name: ray-worker
          image: rayproject/ray:2.3.0
          ports:
          - containerPort: 9001
            name: metrics
            protocol: TCP

Related issue number

Closes #864

Checks

ray-cluster.custom-prometheus-export-port.yaml set metrics endpoint in port 9091
(Can be found here)

kubectl apply -f /home/ubuntu/kuberay/ray-cluster.custom-prometheus-export-port.yaml
# install the ServiceMonitor, PodMonitor and PrometheusRule
./install/prometheus/install.sh
kubectl port-forward --address 0.0.0.0 prometheus-prometheus-kube-prometheus-prometheus-0 -n prometheus-system 9090:9090
# go to http://localhost:9090

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[gvspraveen]

if i am reading this right, if user does not provide metrics port then we wont provide a default anymore.

Curious why we cannot have both?

1. By default exposes Prometheus on port 8080
2. Allow users to override

[Yicheng-Lu-llll]

Oh！ we will have the default Prometheus port 8080.

	// add metrics port for exposing to the promethues stack.
	metricsPort := v1.ContainerPort{
		Name:          "metrics",
		ContainerPort: int32(DefaultMetricsPort),
	}
	dupIndex := -1
	for i, port := range podTemplate.Spec.Containers[0].Ports {
		if port.Name == metricsPort.Name {
			dupIndex = i
			break
		}
	}
        // dupIndex < 0 means the user does not set metricsPort, so we set the default metricsPort.
	if dupIndex < 0 {
		podTemplate.Spec.Containers[0].Ports = append(podTemplate.Spec.Containers[0].Ports, metricsPort)
	}
       // If the user does set metricsPort, we will now not override with the default metricsPort

[jhasm]

Changes look good to me, but I do not see any tests. Are you still working on the tests or are the tests covered somewhere else?

[Yicheng-Lu-llll]

Added tests. Thank you!

[kevin85421]

LGTM