trace-command.md

Trace Command

Overview

Applications deployed with Crossplane usually contains several Crossplane Custom Resources (CRs). Additionally Crossplane applies the Kubernetes pattern for Persistent Volume (PV) claims and classes which results in multiple resources which are closely related. While this enables separation of concern between app teams and cluster administrators, it also makes debugging and troubleshooting uneasy. A problem in the bottom of the resource hierarchy affects upper layers and finding actual root cause requires couple of consecutive commands both to see the status and to get the reference of the next resource.

For example, when a KubernetesApplicationResource reports a failure in its status indicating that there is something wrong with accessing Kubernetes cluster, user needs to first find the corresponding Resource Claim, check its status and may need to continue with the Managed Resource, Resource Class and Provider resources.

Trace command aims to ease debugging and troubleshooting process by providing a holistic view for the resource being traced.

Usage

The trace tool is part of the crossplane kubectl plugin and can be used as follows:

kubectl crossplane trace TYPE[.GROUP] NAME [-n| --namespace NAMESPACE] [--kubeconfig KUBECONFIG] [-o| --outputFormat dot]

Examples:

### Text output
# Trace a KubernetesApplication
kubectl crossplane trace KubernetesApplication wordpress-app-83f04457-0b1b-4532-9691-f55cf6c0da6e -n app-project1-dev

# Trace a MySQLInstance
kubectl crossplane trace MySQLInstance wordpress-mysql-83f04457-0b1b-4532-9691-f55cf6c0da6e -n app-project1-dev

### Graph output, which can be visualized with graphviz as follows.
kubectl crossplane trace KubernetesApplication wordpress-app-83f04457-0b1b-4532-9691-f55cf6c0da6e -n app-project1-dev -o dot | dot -Tpng > /tmp/output.png

State of a resource

Each object in the trace output categorized in one of the following states which is represented in text (state column of overview) and graph output (as node attributes) accordingly.

State	Text Representation	Graph Representation
Ready	✓	Solid - Black
NotReady	!	Solid - Orange
Missing	✕	Dotted - Red

Sample outputs

Text output

Graph output

How it works

The trace tool categorizes Crossplane resources based on a mapping between resource group/kinds and one of the following Crossplane resource types. Then it finds the related resources with the help of the references available in the resource declarations as shown below.

• Kubernetes Application

  • status.clusterRef -> KubernetesCluster
  • listing/filtering all KubernetesApplicationResources and ResourceClaims in the same namespace with labels at spec.resourceSelector.matchLabels

• Kubernetes Application Resource

  • status.clusterRef -> KubernetesCluster
  • spec.secrets -> Connection secrets

• Resource Claim

  • spec.resourceRef -> Managed Resource
  • spec.classRef -> Resource Class
  • spec.writeConnectionSecretToRef -> Connection secrets

• Managed Resource

  • spec.claimRef -> Resource Claim
  • spec.classRef -> Resource Class
  • spec.writeConnectionSecretToRef -> Connection secrets

• Resource Class

  • specTemplate.providerRef -> Provider

• Provider

  • spec.credentialsSecretRef -> Credentials Secret

Knowing related objects for a given one, the trace tool builds a graph representing the relations. The data of the graph is outputted either in text format or as graph definition dot format. Text output contains both an overview of all related types indicating their state)

Future Work

• Improve overview of text output to represent the hierarchy

• Support tracing resources referenced with cross resource referencing.

• Support dynamically defining new resources/relations (via a config file) without rebuilding the binary which can position the tool as a general purpose tracing tool for tracing any Kubernetes Resource. This could make supporting cross resource referencing much easier.

• Based on the pattern established for parent child relationships, support tracing starting with stack instances.