AiMLops Project 3.1 System improvements

Managing Certificates and Ingress for MLflow in Kubernetes

This guide explains how to apply, verify, troubleshoot, and delete certificates and Ingress resources in a Kubernetes cluster for MLflow HTTPS configuration.

1. Apply Required Resources

These commands apply the necessary Kubernetes resources to set up HTTPS for MLflow using cert-manager and an nginx ingress controller.

kubectl apply -f clusterIssuer.yaml
kubectl apply -f mlflow-certificate.yaml
kubectl apply -f mlflow-ingress.yaml

What These Commands Do:

• clusterIssuer.yaml – Creates a ClusterIssuer, which is responsible for issuing TLS certificates using a self-signed CA or Let's Encrypt.
• mlflow-certificate.yaml – Requests a TLS certificate for the MLflow domain (e.g., mlflow.local).
• mlflow-ingress.yaml – Configures the Ingress to route traffic to MLflow and use the TLS certificate.

2. Verify Certificate and Certificate Requests

After applying the resources, verify if the certificates are correctly issued:

kubectl get certificate -n mlflow
kubectl get certificaterequest -n mlflow
kubectl get orders -n mlflow

What These Commands Do:

• get certificate – Lists all TLS certificates in the mlflow namespace.
• get certificaterequest – Displays the current status of certificate requests.
• get orders – Shows certificate signing requests handled by cert-manager.

3. Troubleshoot and Inspect Certificates

If there are issues with the certificate, check detailed information:

kubectl describe certificate mlflow-cert -n mlflow
kubectl describe certificaterequest -n mlflow
kubectl describe order -n mlflow
kubectl delete secret mlflow-tls -n mlflow

What These Commands Do:

• describe certificate mlflow-cert – Provides details about the mlflow-cert, including its status and conditions.
• describe certificaterequest – Shows why a certificate request might be failing.
• describe order – Provides information about how the certificate is being issued.
• delete secret mlflow-tls – Removes the TLS secret, forcing a re-issuance of the certificate.

4. Delete Certificate and Ingress Resources

If a certificate is not working correctly, you might need to delete and reapply it:

kubectl delete clusterIssuer self-signature-issuer -n mlflow
kubectl delete certificate mlflow-cert -n mlflow
kubectl delete ingress mlflow-ingress -n mlflow

What These Commands Do:

• delete clusterIssuer – Removes the ClusterIssuer.
• delete certificate – Deletes the MLflow certificate, allowing for regeneration.
• delete ingress – Deletes the Ingress resource.

5. Debugging the Nginx Ingress

To check if the Nginx Ingress Controller is working properly:

kubectl get ingress -n mlflow
kubectl describe ingress mlflow-ingress -n mlflow
kubectl logs -l app.kubernetes.io/name=ingress-nginx -n ingress-nginx
kubectl exec -it <nginx-ingress-pod> -n ingress-nginx -- cat /etc/nginx/nginx.conf

What These Commands Do:

• get ingress – Lists the Ingress resources.
• describe ingress – Shows detailed information about how traffic is being routed.
• logs -l app.kubernetes.io/name=ingress-nginx – Fetches logs from the Nginx Ingress Controller.
• exec -it <nginx-ingress-pod> -- cat /etc/nginx/nginx.conf – Allows you to inspect the nginx.conf file inside the pod.

6. Testing HTTPS Connectivity

After everything is applied, test if MLflow is reachable over HTTPS:

curl -v https://mlflow.local
kubectl port-forward service/mlflow 5000:5000 -n mlflow
kubectl get services -n mlflow

What These Commands Do:

• curl -v https://mlflow.local – Checks if HTTPS is working.
• port-forward service/mlflow 5000:5000 – Temporarily exposes MLflow on localhost:5000 for testing.
• get services – Ensures the mlflow service is running.

How HTTPS Gets the Certificate Using ClusterIssuer and Nginx Ingress

This section explains how cert-manager, ClusterIssuer, Nginx Ingress, and MLflow Ingress work together to issue and use a TLS certificate.

Step 1: Certificate Issuance

• The ClusterIssuer (from clusterIssuer.yaml) generates or requests certificates.
• Cert-manager requests a certificate from the ClusterIssuer when mlflow-certificate.yaml is applied.
• The certificate is stored in a Kubernetes Secret named mlflow-tls.

Key Resource: mlflow-certificate.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mlflow-cert
  namespace: mlflow
spec:
  secretName: mlflow-tls
  issuerRef:
    name: self-signature-issuer
    kind: ClusterIssuer
  dnsNames:
    - mlflow.local

Step 2: Nginx Ingress Uses the Certificate

• The mlflow-ingress.yaml file tells the Ingress controller (Nginx) to use TLS termination.
• Nginx uses the mlflow-tls secret as the source of the SSL certificate.

Key Resource: mlflow-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mlflow-ingress
  namespace: mlflow
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - mlflow.local
      secretName: mlflow-tls

Step 3: Traffic Flow Through Nginx

1. A user requests https://mlflow.local.
2. Nginx checks the TLS certificate in the mlflow-tls secret.
3. If the certificate is valid, the HTTPS handshake is completed.
4. Nginx forwards the request to the MLflow service.

Verification

• Check if the certificate is issued: kubectl get certificate -n mlflow
• Check if the secret is created: kubectl get secret mlflow-tls -n mlflow
• Check the Ingress configuration: kubectl describe ingress mlflow-ingress -n mlflow
• Test HTTPS with Curl: curl -v https://mlflow.local

Recap

Component	Role
ClusterIssuer	Issues certificates via cert-manager
Cert-Manager	Requests, manages, and renews certificates
Kubernetes Secret (mlflow-tls)	Stores the issued certificate and private key
Nginx Ingress Controller	Acts as a reverse proxy, handling HTTPS and forwarding requests
MLflow Ingress (mlflow-ingress.yaml)	Tells Nginx how to route requests to the MLflow service
MLflow Service (mlflow.yaml)	Exposes the MLflow server inside Kubernetes