trace should understand the package resource model to help troubleshoot them also

[jbw976]

What problem are you facing?

Currently, crossplane beta trace helps you troubleshoot issues with unhealthy resources by traversing their relationships and displaying relevant status information and events. This focuses on resources like claims, XRs, and MRs, but has no special understanding of the resource model relevant to the package manager.

We should teach the trace command about the package manager, so it knows about the following resources and the relationships between them:

• Provider/ProviderRevision
• Configuration/ConfigurationRevision
• Function/FunctionRevision
• Lock
• others?

How could Crossplane help solve your problem?

Once crossplane CLI understands the package resource model, people could then use the trace command to troubleshoot issues with installing and upgrading packages just like they are now doing for infra resources.

[Piotr1215]

Maybe also adding ProviderConfig/secrets ? It could help to ensure that the right ProviderConfig was selected and the associated secret exists. This is one of the most common errors and always takes a moment to troubleshoot.

[jbw976]

I'm quite interested in taking this on 💪 🙏

[phisco]

Awesome, @jbw976! Feel free to ping me if you need any help!

[jbw976]

A working implementation of this functionality has been pushed to https://github.com/jbw976/crossplane/tree/trace-pkg. Folks are free to play around with it there, although there is some incomplete functionality and some areas to clean and polish still:

• versions of packages/revisions aren't being shown anywhere
• -o dot is not implemented yet, just default, wide, json
• no unit tests have been written at all
• refactor some inefficient code and long/ugly getPackageTree function

https://github.com/jbw976/crossplane/tree/trace-pkg

if you checkout that fork/branch, you can run it yourself to try out the experience with something like:

go run cmd/crank/main.go beta trace configuration.pkg.crossplane.io/upbound-platform-ref-aws

Example output of steady state happy path with everything healthy:

go run cmd/crank/main.go beta trace configuration.pkg.crossplane.io/upbound-platform-ref-aws
NAME                                                                               INSTALLED   HEALTHY   STATE    STATUS
Configuration/upbound-platform-ref-aws                                             True        True      -        HealthyPackageRevision
├─ ConfigurationRevision/upbound-platform-ref-aws-ad01153c1179                     -           True      Active   HealthyPackageRevision
├─ Configuration/upbound-configuration-aws-network                                 True        True      -        HealthyPackageRevision
│  ├─ ConfigurationRevision/upbound-configuration-aws-network-288fcd1b88dd         -           True      Active   HealthyPackageRevision
│  └─ Provider/upbound-provider-aws-ec2                                            True        True      -        HealthyPackageRevision
│     ├─ ProviderRevision/upbound-provider-aws-ec2-4bad1441a535                    -           True      Active   HealthyPackageRevision
│     └─ Provider/upbound-provider-family-aws                                      True        True      -        HealthyPackageRevision
│        └─ ProviderRevision/upbound-provider-family-aws-2f3cedd32390              -           True      Active   HealthyPackageRevision
├─ Configuration/upbound-configuration-aws-database                                True        True      -        HealthyPackageRevision
│  ├─ ConfigurationRevision/upbound-configuration-aws-database-b9498c9ae95c        -           True      Active   HealthyPackageRevision
│  └─ Provider/upbound-provider-aws-rds                                            True        True      -        HealthyPackageRevision
│     └─ ProviderRevision/upbound-provider-aws-rds-97388c2131f5                    -           True      Active   HealthyPackageRevision
├─ Configuration/upbound-configuration-aws-eks                                     True        True      -        HealthyPackageRevision
│  ├─ ConfigurationRevision/upbound-configuration-aws-eks-892fd2a71c6b             -           True      Active   HealthyPackageRevision
│  ├─ Provider/crossplane-contrib-provider-helm                                    True        True      -        HealthyPackageRevision
│  │  └─ ProviderRevision/crossplane-contrib-provider-helm-b4cc4c2c8db3            -           True      Active   HealthyPackageRevision
│  ├─ Provider/crossplane-contrib-provider-kubernetes                              True        True      -        HealthyPackageRevision
│  │  └─ ProviderRevision/crossplane-contrib-provider-kubernetes-fd7ab5be249e      -           True      Active   HealthyPackageRevision
│  ├─ Provider/upbound-provider-aws-eks                                            True        True      -        HealthyPackageRevision
│  │  └─ ProviderRevision/upbound-provider-aws-eks-879082c5f6c3                    -           True      Active   HealthyPackageRevision
│  └─ Provider/upbound-provider-aws-iam                                            True        True      -        HealthyPackageRevision
│     └─ ProviderRevision/upbound-provider-aws-iam-a5640230f7e7                    -           True      Active   HealthyPackageRevision
├─ Configuration/upbound-configuration-app                                         True        True      -        HealthyPackageRevision
│  └─ ConfigurationRevision/upbound-configuration-app-cf08a508be65                 -           True      Active   HealthyPackageRevision
├─ Configuration/upbound-configuration-observability-oss                           True        True      -        HealthyPackageRevision
│  ├─ ConfigurationRevision/upbound-configuration-observability-oss-06515e3a2c36   -           True      Active   HealthyPackageRevision
│  └─ Provider/grafana-provider-grafana                                            True        True      -        HealthyPackageRevision
│     └─ ProviderRevision/grafana-provider-grafana-ac529c8ce1c6                    -           True      Active   HealthyPackageRevision
└─ Configuration/upbound-configuration-gitops-flux                                 True        True      -        HealthyPackageRevision
   └─ ConfigurationRevision/upbound-configuration-gitops-flux-86b99b7c9ebe         -           True      Active   HealthyPackageRevision

Example of running trace in the middle of installation with lots of incomplete dependencies, unhealthy packages, etc.:

go run cmd/crank/main.go beta trace configuration.pkg.crossplane.io/upbound-platform-ref-aws
NAME                                                                               INSTALLED   HEALTHY   STATE    STATUS
Configuration/upbound-platform-ref-aws                                             True        Unknown   -        ActivePackageRevision
├─ ConfigurationRevision/upbound-platform-ref-aws-ad01153c1179                     -           Unknown   Active   UnknownPackageRevisionHealth: ...bound/provider-aws-eks xpkg.upbound.io/upbound/provider-aws-iam]
├─ Configuration/upbound-configuration-aws-network                                 True        Unknown   -        ActivePackageRevision
│  ├─ ConfigurationRevision/upbound-configuration-aws-network-288fcd1b88dd         -           Unknown   Active   UnknownPackageRevisionHealth: ...missing dependencies: [xpkg.upbound.io/upbound/provider-aws-ec2]
│  └─ Provider/upbound-provider-aws-ec2                                            -           -         -
├─ Configuration/upbound-configuration-aws-database                                True        Unknown   -        ActivePackageRevision
│  ├─ ConfigurationRevision/upbound-configuration-aws-database-b9498c9ae95c        -           Unknown   Active   UnknownPackageRevisionHealth: ...bound/provider-aws-ec2 xpkg.upbound.io/upbound/provider-aws-rds]
│  └─ Provider/upbound-provider-aws-rds                                            -           -         -        Error: providers.pkg.crossplane.io "upbound-provider-aws-rds" not found
├─ Configuration/upbound-configuration-aws-eks                                     True        Unknown   -        ActivePackageRevision
│  ├─ ConfigurationRevision/upbound-configuration-aws-eks-892fd2a71c6b             -           Unknown   Active   UnknownPackageRevisionHealth: ...bound/provider-aws-eks xpkg.upbound.io/upbound/provider-aws-iam]
│  ├─ Provider/crossplane-contrib-provider-helm                                    -           -         -        Error: ...s.pkg.crossplane.io "crossplane-contrib-provider-helm" not found
│  ├─ Provider/crossplane-contrib-provider-kubernetes                              -           -         -        Error: ...crossplane.io "crossplane-contrib-provider-kubernetes" not found
│  ├─ Provider/upbound-provider-aws-eks                                            -           -         -        Error: providers.pkg.crossplane.io "upbound-provider-aws-eks" not found
│  └─ Provider/upbound-provider-aws-iam                                            -           -         -        Error: providers.pkg.crossplane.io "upbound-provider-aws-iam" not found
├─ Configuration/upbound-configuration-app                                         True        Unknown   -        ActivePackageRevision
│  └─ ConfigurationRevision/upbound-configuration-app-cf08a508be65                 -           Unknown   Active   UnknownPackageRevisionHealth: ...dependencies: [xpkg.upbound.io/crossplane-contrib/provider-helm]
├─ Configuration/upbound-configuration-observability-oss                           True        Unknown   -        ActivePackageRevision
│  ├─ ConfigurationRevision/upbound-configuration-observability-oss-06515e3a2c36   -           Unknown   Active   UnknownPackageRevisionHealth: ...ib/provider-kubernetes xpkg.upbound.io/grafana/provider-grafana]
│  └─ Provider/grafana-provider-grafana                                            -           -         -        Error: providers.pkg.crossplane.io "grafana-provider-grafana" not found
└─ Configuration/upbound-configuration-gitops-flux                                 True        Unknown   -        ActivePackageRevision
   └─ ConfigurationRevision/upbound-configuration-gitops-flux-86b99b7c9ebe         -           Unknown   Active   UnknownPackageRevisionHealth: ...dependencies: [xpkg.upbound.io/crossplane-contrib/provider-helm]

Feedback and questions are welcome on this initial implementation! 🙇‍♂️

[jbasement]

Really nice functionality! One flag that might be useful is to allow setting a maximum depth. That might make sense for revisions as quite a few might accumulate. I think the flag can be specific to only revisions or something more generic that allows passing a pair of e.g. resource:depth.

[phisco]

+1 for the depth control, something similar to tree -L <N>.

[jbw976]

Thanks for taking a look @jbasement and @phisco! Not sure if you noticed, but there are already 2 flags that control output, that I think may work more specifically than a general "depth" option:

      --show-dependencies="unique"
                                   Show package dependencies in the output.
                                   One of: unique, all, none.
      --show-revisions="active"    Show package revisions in the output. One of:
                                   active, all, none.

By default, we'll only show:

• unique dependencies: if a dependency shows up multiple times in the tree, e.g. provider-aws-family, we'll just show the first instance of it
• active revisions: if there are multiple revisions for a package, only the active revision will be shown

If they want to see all of none for dependencies and revisions, that's OK too 😅

I like those options since they are more specific to the package types and the user can more consciously choose what they want to see versus a generic "depth" number. Let me know if I'm crazy with that idea though 🤪