AiMLops Project 3.5 Questions Summary

Questions and Issues Summary

This document collects and summarizes all the questions, requests, and error messages discussed regarding the MLOps Platform setup script.

📝 Initial Request

"Can you recreate the whole script?"

You requested a recreation of the full setup.sh Bash script that manages the installation of a Kubeflow-based platform. This includes:

• Disk/CPU checks
• Deployment option prompts
• Tool installations
• Kind cluster creation
• Resource application via Kustomize

⚠️ Main Errors Encountered

1. Unbound Variable: CLUSTER_NAME

Error Message:

./setup.sh: line 33: CLUSTER_NAME: unbound variable

Cause: The script tries to access $CLUSTER_NAME before it is defined or exported in the environment.

Fix: Ensure that config.env exists and contains a definition like:

CLUSTER_NAME=my-cluster

Also ensure the line:

source $PLATFORM_CONFIG

is placed after $PLATFORM_CONFIG is initialized.

2. Kustomize Path Resolution Errors

Error Message:

accumulating resources from '../../apps/pipeline/upstream/env/platform-agnostic-emissary': must resolve to a file

Cause: The kustomize build command is referencing relative paths that do not exist in your directory structure.

Fixes:

• Make sure all kustomization.yaml files have correct resources: paths.
• Ensure all submodules or dependent directories are cloned or initialized.
• You may need to run:

git submodule update --init --recursive

3. Kustomize Error: ConfigMap Not Found

Error Message:

nothing selected by ConfigMap.[noVer].[noGrp]/ssl-config.[noNs]

Cause: The Kustomize overlay references a ConfigMap named ssl-config which hasn't been created.

Fixes:

• Ensure Kubernetes_ssl_configmap_creation.sh is executed before Kustomize build.
• Validate the ConfigMap with:

kubectl get configmap ssl-config -A

4. Certificate Manager Webhook Issues

Error Message:

Error from server (InternalError): error when creating "...letsclusterissuer.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority

Cause: Cert-manager webhook was re-enabled before the webhook pod became ready and its certificate was trusted.

Fixes:

• Add proper wait conditions before applying cert-manager resources
• Ensure the webhook's CA bundle is properly propagated

🗂️ File Requests

You requested:

• Recreation of the full setup.sh
• Generation of a .md file containing all errors
• A summary .md file of all the questions asked during the conversation

Topics to Understand for Effective MLOps Platform Troubleshooting

1. Bash Script Debugging and Best Practices

• Key concepts:

  • Using set -x for verbose debugging
  • Proper error capture with set -e
  • Defensive variable handling with set -u
  • Using static analysis tools like ShellCheck

• Best practices:

  • Always quote variables: "$VAR" not $VAR
  • Use absolute paths when possible
  • Include debugging code (set -x) that can be enabled with a flag

2. Kubernetes Resource Dependencies

• Understanding:

  • Kubernetes applies resources in parallel
  • CRDs must be installed before using custom resources
  • Some resources have implicit dependencies (Service → Endpoints)

• Techniques:

  • Use kubectl wait for critical dependencies
  • Employ helm hooks or Job resources for sequencing
  • Check resource status before proceeding

3. Kustomize Architecture and Patterns

• Concepts:

  • Base vs. overlay structure
  • Common patterns for multi-environment deployments
  • Patches vs. transformers

• Debugging approaches:

  • Use --dry-run to see what will be applied
  • Test overlays individually
  • Validate paths using kustomize build

4. MLOps Platform Component Integration

• Key components:

  • Kubeflow for pipelines
  • MLflow for experiment tracking
  • Ray for distributed training
  • Infrastructure components (databases, message queues)

• Integration challenges:

  • Authentication between components
  • Shared storage requirements
  • Network policies and security considerations

5. TLS and Certificate Management in Kubernetes

• Concepts:

  • TLS termination options (edge, passthrough, re-encrypt)
  • Certificate issuers (Let's Encrypt, self-signed)
  • Certificate lifecycle management

• Common issues:

  • Webhook validation certificate trust
  • Certificate renewal
  • Wildcard certificates and multi-domain support

6. Git Submodule Management

• Understanding:

  • How submodules reference specific commits
  • The difference between submodule update and clone
  • Managing dependencies through multiple repositories

• Best practices:

  • Include submodule initialization in setup scripts
  • Document submodule dependencies clearly
  • Consider alternatives like Git subtrees or package managers

7. Kubernetes Admission Controllers and Webhooks

• Key concepts:

  • Validating vs. mutating webhooks
  • TLS certificate requirements
  • Failure modes and timeouts

• Debugging strategies:

  • Check webhook logs
  • Verify TLS certificate chain
  • Use admission controller bypass tools

Conclusion

Understanding these topics will help you diagnose and fix most issues with the MLOps platform setup. Each area represents a potential point of failure, and having a systematic approach to troubleshooting each component will speed up the resolution process.

When developing scripts for Kubernetes-based platforms:

1. Write defensive Bash scripts with proper error handling
2. Use a systematic approach to dependencies between resources
3. Test kustomize builds in isolation before integrating
4. Implement proper waiting and retries for webhook-based admission controllers
5. Document expected configurations thoroughly