top command to view crossplane resource usage and metrics
#5036
Comments
|
Here is a quick Introduction & GoalsThe goal is to create a new command in the crossplane CLI which would:
ConstraintsThere is a fine line between having a fully fledged Prometheus metrics visualized with Grafana and having a simple terminal output. We do not want to implement Prometheus v2, but rather expose simple metrics and resources utilization which gives a quick glimpse into the state of the crossplane machinery. ContextThe crossplane top command will be part of the crossplane CLI and will inherit its context and communication patterns. The below diagram shows the components that the command interacts with. sequenceDiagram
autonumber
actor User
participant Terminal as crossplane top
participant MetricsEndpoint as crossplane/provider/function pod /metrics endpoint
User->>Terminal: Keyboard input
Terminal->>MetricsEndpoint: HTTP call to /metrics endpoint on pod
Note right of MetricsEndpoint: HTTP call to kubelet to retrieve resoureces utilization
MetricsEndpoint-->>Terminal: Metrics/Resource utilization with latest values
Terminal-->>User: Display output
Note left of Terminal: also possible to use <br/>watch for auto-updates
Solution StrategyThe primary objective is to introduce a streamlined command that provides users with instantaneous insights into resource utilization and key metrics without resorting to more heavyweight observability tooling. Here are a few key design decisions grouped into sections. Metrics Acquisition Unlike traditional approaches that rely on Prometheus, this solution will extract metrics directly from User Experience Consistency Terminal Interface Tailored Metrics Selection Parsing and Presentation: Data Snapshot: Mock DesignsSample command output with crossplane and provider pods running:
It is also possible to add header information which would act like a combination of
Out of ScopeA few ideas that are worth noting but are out of scope for the initial implementation:
Future improvementsFor now the plan is to cover crossplane, providers and functions pods. In the future, we could add other crossplane native API types such as External Secret Store Plugins and maybe others. Crossplane Docs · v1.14 · Crossplane API POC ImplementationThe Current POC implementation can be found in the crossplane-top repository. package main
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/tools/clientcmd"
"k8s.io/metrics/pkg/client/clientset/versioned"
)
func getKubeConfig() (string, error) {
kubeconfig := os.Getenv("KUBECONFIG")
if kubeconfig != "" {
return kubeconfig, nil
}
home, err := os.UserHomeDir()
if err != nil {
return "", fmt.Errorf("could not get user home directory: %w", err)
}
return filepath.Join(home, ".kube", "config"), nil
}
func main() {
if err := run(); err != nil {
fmt.Fprintf(os.Stderr, "error: %v\n", err)
os.Exit(1)
}
}
func run() error {
// Build the config from the kubeconfig path
kubeconfig, err := getKubeConfig()
if err != nil {
return fmt.Errorf("failed to get kubeconfig: %w", err)
}
config, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
if err != nil {
return fmt.Errorf("could not build config from flags: %w", err)
}
// Create the clientset for Kubernetes
k8sClientset, err := kubernetes.NewForConfig(config)
if err != nil {
return fmt.Errorf("could not create the clientset for Kubernetes: %w", err)
}
// Create the clientset for Metrics
metricsClientset, err := versioned.NewForConfig(config)
if err != nil {
return fmt.Errorf("could not create the clientset for Metrics: %w", err)
}
// Fetch all pods from all namespaces in case of Crossplane pods being installed elswhere
pods, err := k8sClientset.CoreV1().Pods(metav1.NamespaceAll).List(context.TODO(), metav1.ListOptions{})
if err != nil {
return fmt.Errorf("could not fetch all pods from all namespaces: %w", err)
}
fmt.Printf("%-20s %-55s %-12s %-15s\n", "NAMESPACE", "NAME", "CPU(cores)", "MEMORY(Mi)")
// Loop through pods and print metrics
for _, pod := range pods.Items {
for labelKey := range pod.GetLabels() {
if strings.HasPrefix(labelKey, "pkg.crossplane.io/provider") || strings.HasPrefix(labelKey, "pkg.crossplane.io/function") {
podMetrics, err := metricsClientset.MetricsV1beta1().PodMetricses(pod.Namespace).Get(context.TODO(), pod.Name, metav1.GetOptions{})
if err != nil {
fmt.Printf("Error getting metrics for pod %s: %v\n", pod.Name, err)
continue
}
for _, container := range podMetrics.Containers {
cpuUsage := container.Usage.Cpu().ScaledValue(resource.Milli)
memoryUsage := fmt.Sprintf("%dMi", container.Usage.Memory().ScaledValue(resource.Mega))
fmt.Printf("%-20s %-55s %-12d %-15s\n", pod.Namespace, pod.Name, cpuUsage, memoryUsage)
}
}
}
}
return nil
}Risks and Technical DebtCrossplane CLI is part of the crossplane repository and in the future will be moved to a separate repository. This might accumulate some technical debt in the command that will need to address in the future.
|
|
Great stuff @Piotr1215, I think that is a very reasonable proposal and a great place to get started in v1.15 ✅ A few thoughts to share:
|
|
Thank you for the comments @jbw976 🙏
|
|
cool @Piotr1215!!
|
|
Thank you for suggestions @jbw976. A number of warning events could be a nice early trigger warning that something is wrong with the cluster. Also, nr of reconciles would definitely affect performance. Definitely worth looking into. For now in the first small PR I'm targeting basic functionality mirroring kubectl top pods but only for crossplane pods. The follow-up PRs will add more functionality :). |
|
Initial implementation of this feature has been merged for v1.15: #5245 |
There is currently not a streamlined (easy) way to get quick observability and insight into Crossplane's resource usage and metrics. We could build a
crossplane topcommand that quickly exposes some key metrics to users of the crossplane CLI so they get a high level view of overall health of the control plane.Quick thoughts:
kubectl topand perhaps it can be leveragedThe text was updated successfully, but these errors were encountered: