Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Customize the Prometheus export port #954

Conversation

Yicheng-Lu-llll
Copy link
Contributor

@Yicheng-Lu-llll Yicheng-Lu-llll commented Mar 10, 2023

Why are these changes needed?

Currently, KubeRay always exposes a Prometheus metrics endpoint on port 8080, and users are unable to customize this export port (see issue #864). However, the Ray documentation indicates that users are able to set up a custom Prometheus export port.

To better meet the needs of users and maintain consistency with Ray's capabilities, I recommend enabling the customization of the Prometheus export port for KubeRay users.

How to set Prometheus export port ?

  • Add metrics-export-port: "{Port Number}" to rayStartParams for head and worker pods.
  • Add containerPort: {Port Number} to template.spec.containers[0].ports for head and worker pods.
    Note, It's user's responsibility to maintain rayStartParam ports and container ports mapping(see here)

For example:
(full version can be found here)

apiVersion: ray.io/v1alpha1
kind: RayCluster
spec:
  headGroupSpec:
    rayStartParams:
      metrics-export-port: "9001"
    template:
      spec:
        containers:
        - name: ray-head
          image: rayproject/ray:2.3.0
          ports:
          - containerPort: 9001
            name: metrics
            protocol: TCP

  workerGroupSpecs:
  - replicas: 1
    minReplicas: 1
    maxReplicas: 10
    rayStartParams:
      metrics-export-port: "9001" 
    template:
      spec:
        containers:
        - name: ray-worker
          image: rayproject/ray:2.3.0
          ports:
          - containerPort: 9001
            name: metrics
            protocol: TCP

Related issue number

Closes #864

Checks

ray-cluster.custom-prometheus-export-port.yaml set metrics endpoint in port 9091
(Can be found here)

kubectl apply -f /home/ubuntu/kuberay/ray-cluster.custom-prometheus-export-port.yaml
# install the ServiceMonitor, PodMonitor and PrometheusRule
./install/prometheus/install.sh
kubectl port-forward --address 0.0.0.0 prometheus-prometheus-kube-prometheus-prometheus-0 -n prometheus-system 9090:9090
# go to http://localhost:9090

1678424357945

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • This PR is not tested :(

@Yicheng-Lu-llll Yicheng-Lu-llll changed the title Enable customize the Prometheus export port Customize the Prometheus export port Mar 10, 2023
@kevin85421 kevin85421 self-requested a review March 12, 2023 05:50
@@ -137,8 +137,6 @@ func DefaultHeadPodTemplate(instance rayiov1alpha1.RayCluster, headSpec rayiov1a
}
if dupIndex < 0 {
podTemplate.Spec.Containers[0].Ports = append(podTemplate.Spec.Containers[0].Ports, metricsPort)
} else {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if i am reading this right, if user does not provide metrics port then we wont provide a default anymore.

Curious why we cannot have both?

  1. By default exposes Prometheus on port 8080
  2. Allow users to override

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh! we will have the default Prometheus port 8080.

	// add metrics port for exposing to the promethues stack.
	metricsPort := v1.ContainerPort{
		Name:          "metrics",
		ContainerPort: int32(DefaultMetricsPort),
	}
	dupIndex := -1
	for i, port := range podTemplate.Spec.Containers[0].Ports {
		if port.Name == metricsPort.Name {
			dupIndex = i
			break
		}
	}
        // dupIndex < 0 means the user does not set metricsPort, so we set the default metricsPort.
	if dupIndex < 0 {
		podTemplate.Spec.Containers[0].Ports = append(podTemplate.Spec.Containers[0].Ports, metricsPort)
	}
       // If the user does set metricsPort, we will now not override with the default metricsPort

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good to me, but I do not see any tests. Are you still working on the tests or are the tests covered somewhere else?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added tests. Thank you!

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>
Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>
Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>
Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>
@Yicheng-Lu-llll Yicheng-Lu-llll force-pushed the containerPortForMetricsCannotBeSet branch from a91e140 to d005170 Compare March 16, 2023 22:36
Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>
Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>
Copy link
Member

@kevin85421 kevin85421 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kevin85421 kevin85421 merged commit 4b5085f into ray-project:master Mar 17, 2023
lowang-bh pushed a commit to lowang-bh/kuberay that referenced this pull request Sep 24, 2023
Customize the Prometheus export port
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] Helm CRD RayService workerGroup containerPort for metrics cannot be set
4 participants