OTA-1080: pkg/cli/admin/inspectalerts: New tech-preview inspect-alerts subcommand #1674
Conversation
|
@wking: This pull request references OTA-1080 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/valid-reference
|
@wking: This pull request references OTA-1080 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/cc |
| uris := make([]string, 0, len(route.Status.Ingress)) | ||
| for _, ingress := range route.Status.Ingress { | ||
| uri := &url.URL{ | ||
| Scheme: "https", |
There was a problem hiding this comment.
Will the certificate authority in kubeconfig be used?
There was a problem hiding this comment.
Nope, there's no CA handling at all, I'm just using a stock Go http.Client{}. Is there some kind of tooling/precedent for using the kubeconfig to figure out the CAs that should be trusted to connect to a particular Route?
|
|
||
| client := &http.Client{} | ||
| uris := make([]string, 0, len(route.Status.Ingress)) | ||
| for _, ingress := range route.Status.Ingress { |
There was a problem hiding this comment.
Do we always check the first one in the slice and return immediately?
There was a problem hiding this comment.
From my PR/commit message:
If I get a failure from one
status.ingress[].host, I don't bother attempting the others. Wiring up that kind of fallback with error aggregation when all hosts fail would be a useful future refinement.
If you have a preference on that or the Admitted=True filtering I floated in the PR/commit message, let me know about that too :)
| return nil, fmt.Errorf("no token is currently in use for this session") | ||
| } | ||
|
|
||
| route, err := getRoute(ctx, namespace, name, metav1.GetOptions{}) |
There was a problem hiding this comment.
What is the purpose of getRoute better than passing route object into GetWithBearer after retrieving it in Run
There was a problem hiding this comment.
Thanks to that, we wouldn't pass ctx which is only used by getRoute
There was a problem hiding this comment.
We're expecting to consume this in future oc adm upgrade status work, and I wanted to bundle as much as possible into the shared helper, just to make the call-site simpler on the consumer side. But having two separate call-and-catch-err in the status call-site would just be a bit less compact, and not hard to do, so I don't mind pivoting to passing route if you aren't buying my "easier on Go consumers" pitch here ;)
| return nil, err | ||
| } | ||
|
|
||
| req.Header.Add("Authorization", fmt.Sprintf("Bearer %s", bearerToken)) |
There was a problem hiding this comment.
As far as I understand we are sending request to another endpoint other than API server because this endpoint is not registered in API server?. This is not a common pattern which ~every request goes to API server?
There was a problem hiding this comment.
If /api/v2/alerts is registered to API server, why we don't use the builtin functionality, instead of manually adding bearer token in header?
There was a problem hiding this comment.
This is a non-Kube API, e.g. in a 4.16.0-ec.1 cluster, the only alert API resources are:
$ oc api-resources | grep -i alert
alertmanagerconfigs amcfg monitoring.coreos.com/v1beta1 true AlertmanagerConfig
alertmanagers am monitoring.coreos.com/v1 true Alertmanager
alertingrules monitoring.openshift.io/v1 true AlertingRule
alertrelabelconfigs monitoring.openshift.io/v1 true AlertRelabelConfigAnd none of those is this list of Alertmanager pending/firing alerts. An alternative access mechanism would be for the Alertmanager to register its alert type with Kube, e.g. as an aggregated API server. But that sort of pivot would be large, and in the meantime, we want some kind of convenient access to firing alerts for the status subcommand.
But narrowly, if there is Go/Kube/other tooling for Bearer injection on a shelf somewhere, I'm happy to pivot to using that when constructing these requests. I was unable to find it on a brief search yesterday, but I didn't sink too much time into hunting, because I wanted to limit the time invested in this tech-preview proof-of-concept.
There was a problem hiding this comment.
As you referred above in the source code, we could leverage github.com/prometheus/client_golang/v1 + github.com/prometheus/common for token injection but it can be deferred to a following PR.
@ardaguclu the monitoring stack exposes a few API services protected by kube-rbac-proxy which uses TokenReview + SubjectAccessReview to authenticate and authorizate requests. Hence the need to provide a bearer token.
There was a problem hiding this comment.
Thanks. But as I pointed out in #1674 (comment), bearer token may be empty and kubeconfig is still accessible to the API server. Any attempt discards the client-go may have such an issue. That results in user can access to API server but this command just doesn't work from user's perspective.
There was a problem hiding this comment.
Yes the request will be rejected. I'm not sure what you're advising for though?
There was a problem hiding this comment.
As you referred above in the source code, we could leverage github.com/prometheus/client_golang/v1 + github.com/prometheus/common for token injection but it can be deferred to a following PR.
I'm advising this ^ but it seems that it is not possible soon. I'd suggest to mention this case in the description of the command to warn user about this.
wking
force-pushed
the
alert-access
branch
3 times, most recently
from
a26bea5
to
675fdb4
Compare
|
Can we add an option to filter the alerts based on the severity e.g. |
All things are possible, but I was trying to avoid sinking time into aspects that $ OC_ENABLE_CMD_INSPECT_ALERTS=true ./oc adm inspect-alerts | jq -r '[.[] | select(.labels.severity == "warning") | .startsAt + " " + .endsAt + " " + .status.state + " " + (.labels | .severity + " " + .alertname) + " " + .annotations.summary] | sort[]'
2024-01-19T20:14:20.600Z 2024-02-01T00:31:50.600Z suppressed warning TechPreviewNoUpgrade Cluster has enabled tech preview features that will prevent upgrades.And since the main issue I personally have consuming the raw JSON is excluding all the properties I don't care about, I'm already using Anyhow, things in this space seem like they'd be nice-to-have, but I don't see them as proof-of-concept blockers. |
In future iterations, we could consider wiring up amtool to avoid implementing another client. |
| } | ||
|
|
||
| // GetWithBearer gets a Route by namespace/name, contructs a URI using | ||
| // status.ingress[].host and the path argument, and performs GETs on that |
There was a problem hiding this comment.
Maybe https://github.com/openshift/library-go/blob/2ad786549f07f264339f63b12b71a0fe5e77e832/pkg/route/routeapihelpers/routeapihelpers.go#L18 could help here, it also checks the conditions ;)
There was a problem hiding this comment.
oc vendors library-go, but not yet that sub-package^, and I wanted to avoid mucking with vendor/ and introduce thinking about iteration over ingress[] (IngressURI only returns the first match). But yes, we could pivot to using that. Or extend library-go to grow a callback-consuming function that supports iteration over multiiple matches, etc. Thoughts for this pull?
There was a problem hiding this comment.
We can leave that for future iterations for sure.
(If we could just add a TODO for this somewhere)
|
I'd rather use the Thanos Querier API endpoints than Alertmanager:
The downside is that you won't see which alerts are being silenced since it's only applicable to alerts from Alertmanager. |
Ah, docs around that here. I'd mistakenly thought it was Alertmanager converting Prometheus metrics into alerts, but looks like Prometheus calculates the alerts, and Alertmanager layers on silences and push notification. So yeah, I'll pivot to Thanos. |
I'm a bit worried about confused users who would expect to not see a silenced alert when they do have an alertmanager, but for the purpose of |
Ok, poking at this locally, looks like Thanos is only giving me some metric labels: $ curl -sH "Authorization: Bearer $(oc whoami -t)" "https://$(oc -n openshift-monitoring get route thanos-querier -o jsonpath='{.status.ingress[0].host}')/api/v1/query?query=ALERTS" | jq '.data.result[0]'
{
"metric": {
"__name__": "ALERTS",
"alertname": "ClusterAutoscalerUnschedulablePods",
"alertstate": "pending",
"container": "cluster-autoscaler",
"endpoint": "metrics",
"instance": "10.129.0.57:8085",
"job": "cluster-autoscaler-default",
"namespace": "openshift-machine-api",
"pod": "cluster-autoscaler-default-f8dd547c7-dg9t5",
"prometheus": "openshift-monitoring/k8s",
"service": "cluster-autoscaler-default",
"severity": "info"
},
"value": [
1706914546.182,
"1"
]
}While Alertmanager gives me But anyway, are folks ok if I stick with using Alertmanager for this initial proof-of-concept? |
|
Sorry I was a bit dense: Thanos also exposes a /api/v1/alerts endpoint which returns all active alerts including the annotations. |
As recommended by Simon [1]. By dropping one level down and querying Thanos, we lose Silence access (those only come in at the Alertmanager level), but we gain: * The ability to retrieve alerts when Alertmanager is disabled (especially true with ACM satellite clusters). * Access to pending alerts (not just firing alerts). * Access to both platform and user-defined alerts (the platform Alertmanager only has access to the former). Demo of the new functionality: $ OC_ENABLE_CMD_INSPECT_ALERTS=true oc adm inspect-alerts | jq -r '[.data.alerts[] | .activeAt + " " + .state + " " + (.labels | .severity + " " + .alertname) + " " + .annotations.summary] | sort[]' 2024-01-18T15:42:50Z firing warning TechPreviewNoUpgrade Cluster has enabled tech preview features that will prevent upgrades. 2024-01-18T15:45:13Z firing info ClusterNotUpgradeable One or more cluster operators have been blocking minor version cluster upgrades for at least an hour. 2024-01-19T20:20:58.083333925Z firing none Watchdog An alert that should always be firing to certify that Alertmanager is working properly. 2024-01-22T15:51:45.824704657Z firing info PodStartupStorageOperationsFailing Pods can't start because volume_mount of volume plugin kubernetes.io/configmap is permanently failing on node build0-gstfj-w-b-d6xzf.c.openshift-ci-build-farm.internal. 2024-01-22T15:51:45.824704657Z firing info PodStartupStorageOperationsFailing Pods can't start because volume_mount of volume plugin kubernetes.io/configmap is permanently failing on node build0-gstfj-w-c-q85c8.c.openshift-ci-build-farm.internal. 2024-02-05T20:09:16.894430475Z pending warning PodDisruptionBudgetAtLimit The pod disruption budget is preventing further disruption to pods. 2024-02-05T20:12:56.428212511Z pending info ClusterAutoscalerUnschedulablePods Cluster Autoscaler has 13 unschedulable pods 2024-02-05T20:15:43.59211931Z pending warning MachineNotYetDeleted machine build0-gstfj-ci-tests-worker-d-rrdl9 has been in Deleting phase for more than 6 hours ... [1]: openshift#1674 (comment)
|
Ok, I've pushed 41e5e82 -> da5669f with a pivot to Thanos and $ OC_ENABLE_CMD_INSPECT_ALERTS=true oc adm inspect-alerts | jq -r '[.data.alerts[] | .activeAt + " " + .state + " " + (.labels | .severity + " " + .alertname) + " " + .annotations.summary] | sort[]'
2024-01-18T15:42:50Z firing warning TechPreviewNoUpgrade Cluster has enabled tech preview features that will prevent upgrades.
2024-01-18T15:45:13Z firing info ClusterNotUpgradeable One or more cluster operators have been blocking minor version cluster upgrades for at least an hour.
2024-01-19T20:20:58.083333925Z firing none Watchdog An alert that should always be firing to certify that Alertmanager is working properly.
2024-01-22T15:51:45.824704657Z firing info PodStartupStorageOperationsFailing Pods can't start because volume_mount of volume plugin kubernetes.io/configmap is permanently failing on node build0-gstfj-w-b-d6xzf.c.openshift-ci-build-farm.internal.
2024-01-22T15:51:45.824704657Z firing info PodStartupStorageOperationsFailing Pods can't start because volume_mount of volume plugin kubernetes.io/configmap is permanently failing on node build0-gstfj-w-c-q85c8.c.openshift-ci-build-farm.internal.
2024-02-05T20:09:16.894430475Z pending warning PodDisruptionBudgetAtLimit The pod disruption budget is preventing further disruption to pods.
2024-02-05T20:12:56.428212511Z pending info ClusterAutoscalerUnschedulablePods Cluster Autoscaler has 13 unschedulable pods
2024-02-05T20:15:43.59211931Z pending warning MachineNotYetDeleted machine build0-gstfj-ci-tests-worker-d-rrdl9 has been in Deleting phase for more than 6 hours
... |
|
images hit build04 node capacity. /retest-required |
The new subcommand is gated by OC_ENABLE_CMD_INSPECT_ALERTS to avoid
customers accidentally using it while it's tech-preview:
$ OC_ENABLE_CMD_INSPECT_ALERTS=true oc adm inspect-alerts | jq -r '[.[] | .startsAt + " " + .endsAt + " " + .status.state + " " + (.labels | .severity + " " + .alertname) + " " + .annotations.summary] | sort[]'
2024-01-19T20:13:28.083Z 2024-01-31T05:23:28.083Z suppressed none Watchdog An alert that should always be firing to certify that Alertmanager is working properly.
2024-01-19T20:14:13.174Z 2024-01-31T05:24:13.174Z suppressed info ClusterNotUpgradeable One or more cluster operators have been blocking minor version cluster upgrades for at least an hour.
2024-01-19T20:14:20.600Z 2024-01-31T05:24:20.600Z suppressed warning TechPreviewNoUpgrade Cluster has enabled tech preview features that will prevent upgrades.
2024-01-22T15:56:45.824Z 2024-01-31T05:23:15.824Z suppressed info PodStartupStorageOperationsFailing Pods can't start because volume_mount of volume plugin kubernetes.io/configmap is permanently failing on node build0-gstfj-w-b-d6xzf.c.openshift-ci-build-farm.internal.
2024-01-22T15:56:45.824Z 2024-01-31T05:23:15.824Z suppressed info PodStartupStorageOperationsFailing Pods can't start because volume_mount of volume plugin kubernetes.io/configmap is permanently failing on node build0-gstfj-w-c-q85c8.c.openshift-ci-build-farm.internal.
2024-01-30T06:02:23.913Z 2024-01-31T05:24:23.913Z active warning KubeJobFailed Job failed to complete.
This provides a more convenient syntax based on the existing docs'
[1]:
$ oc login -u <username> -p <password>
$ host=$(oc -n openshift-monitoring get route alertmanager-main -ojsonpath={.spec.host})
$ token=$(oc whoami -t)
$ curl -H "Authorization: Bearer $token" -k "https://$host/api/v2/receivers"
But I'm using the /alerts path [2] instead of the /receivers path
given in the docs example. This provides convenient access when the
console isn't available (e.g. because the Console ClusterVersion
capability is not enabled) or for use in other tooling like the
tech-preview 'oc adm upgrade status', and has a long-standing (but
currently deferred) RFE [3].
I'm also dropping the -k/--insecure from the doc'ed curl in my Go
translation, because:
* I don't want to leak a sensitive token to an untrusted server.
* Clusters are supposed to have set up a well-known cert/CA for their
ingress [4]:
The Ingress Operator generates a default certificate for an
Ingress Controller to serve as a placeholder until you configure a
custom default certificate. Do not use Operator-generated default
certificates in production clusters.
so our client will likely trust the cluster.
and failing on "I don't trust that server" seems fine for this
tech-preview proof-of-concept, vs. trying to use the
kubeconfig-protected connections to automatically figure out which CA
I need to trust.
I'm iterating over status.ingress[].host entries instead of using the
doc'ed spec.host, because the spec property is optional [5], while the
status property is required [6]. It also seems safer to consume the
Route controller's output status instead of wiring up directly to
Route-admin written spec input.
I'm not currently filtering on RouteIngress.Conditions including an
Admitted=True entry, although something like that might be reasonable
to avoid wasting time poking known-to-be-broken hosts.
I haven't dug into Bearer token RFCs or the REST config structure to
know if I need to worry about any encoding issues (base64?) in this
handoff, but in testing it works for me. Please feel free to stiffen,
or at least research, that handoff before making this generally
available :).
If I get a failure from one status.ingress[].host, I don't bother
attempting the others. Wiring up that kind of fallback with error
aggregation when all hosts fail would be a useful future refinement.
We seem to have types in the vendored
github.com/prometheus/common/model that could be used to deserialize
this output (e.g. for table printing), but for now I'll just dump the
JSON to output.
[1]: https://docs.openshift.com/container-platform/4.14/monitoring/accessing-third-party-monitoring-apis.html
[2]: https://github.com/prometheus/alertmanager/blob/v0.26.0/api/v2/openapi.yaml#L134
[3]: https://issues.redhat.com/browse/RFE-928
[4]: https://docs.openshift.com/container-platform/4.14/security/certificate_types_descriptions/ingress-certificates.html#location
[5]: https://github.com/openshift/api/blob/4f00b4f16b634830bb16b432eec91f6e514f2981/route/v1/types.go#L95
[6]: https://github.com/openshift/api/blob/4f00b4f16b634830bb16b432eec91f6e514f2981/route/v1/types.go#L352-L353
As recommended by Simon [1]. By dropping one level down and querying Thanos, we lose Silence access (those only come in at the Alertmanager level), but we gain: * The ability to retrieve alerts when Alertmanager is disabled (especially true with ACM satellite clusters). * Access to pending alerts (not just firing alerts). * Access to both platform and user-defined alerts (the platform Alertmanager only has access to the former). Demo of the new functionality: $ OC_ENABLE_CMD_INSPECT_ALERTS=true oc adm inspect-alerts | jq -r '[.data.alerts[] | .activeAt + " " + .state + " " + (.labels | .severity + " " + .alertname) + " " + .annotations.summary] | sort[]' 2024-01-18T15:42:50Z firing warning TechPreviewNoUpgrade Cluster has enabled tech preview features that will prevent upgrades. 2024-01-18T15:45:13Z firing info ClusterNotUpgradeable One or more cluster operators have been blocking minor version cluster upgrades for at least an hour. 2024-01-19T20:20:58.083333925Z firing none Watchdog An alert that should always be firing to certify that Alertmanager is working properly. 2024-01-22T15:51:45.824704657Z firing info PodStartupStorageOperationsFailing Pods can't start because volume_mount of volume plugin kubernetes.io/configmap is permanently failing on node build0-gstfj-w-b-d6xzf.c.openshift-ci-build-farm.internal. 2024-01-22T15:51:45.824704657Z firing info PodStartupStorageOperationsFailing Pods can't start because volume_mount of volume plugin kubernetes.io/configmap is permanently failing on node build0-gstfj-w-c-q85c8.c.openshift-ci-build-farm.internal. 2024-02-05T20:09:16.894430475Z pending warning PodDisruptionBudgetAtLimit The pod disruption budget is preventing further disruption to pods. 2024-02-05T20:12:56.428212511Z pending info ClusterAutoscalerUnschedulablePods Cluster Autoscaler has 13 unschedulable pods 2024-02-05T20:15:43.59211931Z pending warning MachineNotYetDeleted machine build0-gstfj-ci-tests-worker-d-rrdl9 has been in Deleting phase for more than 6 hours ... [1]: openshift#1674 (comment)
Simon, internally: > ...let's go with the CMO owners... So this brings in the approver set from [1]. That has the same people as reviewers and approvers, so I'm only creating a single monitoring-approvers here to ease future membership-management. If, in the future, monitoring decides to have distinct sets for reviewers and approvers, it would be straightforward to add a new alias. [1]: https://github.com/openshift/cluster-monitoring-operator/blob/b7e3f50875f2bb1fed912b23fb80a101d3a786c0/OWNERS
|
👍 This looks good to me. |
| return alertBytes, err | ||
| } | ||
|
|
||
| // if we end up going this way, probably check and error on 'result' being an empty set (it should at least contain Watchdog) |
There was a problem hiding this comment.
I would return the list as-is. An empty list isn't expected currently but who knows if it doesn't change in the future.
| return nil, err | ||
| } | ||
|
|
||
| req.Header.Add("Authorization", fmt.Sprintf("Bearer %s", bearerToken)) |
There was a problem hiding this comment.
As you referred above in the source code, we could leverage github.com/prometheus/client_golang/v1 + github.com/prometheus/common for token injection but it can be deferred to a following PR.
@ardaguclu the monitoring stack exposes a few API services protected by kube-rbac-proxy which uses TokenReview + SubjectAccessReview to authenticate and authorizate requests. Hence the need to provide a bearer token.
| } | ||
|
|
||
| func (o *options) Run(ctx context.Context) error { | ||
| alertBytes, err := GetAlerts(ctx, o.getRoute, o.RESTConfig.BearerToken) |
There was a problem hiding this comment.
There are many ways to send authenticated request to API server in RESTConfig
oc/vendor/k8s.io/client-go/rest/config.go
Lines 69 to 89 in 4fe33f1
ExecProvider) that skips bearer token step.
There was a problem hiding this comment.
Bearer token is what we doc, so it should be good enough for a env-var-gated first attempt, right?
There was a problem hiding this comment.
Bearer token is not functional for users working with cluster-admin kubeconfig, so that might be a potential problem for a set of our users.
There was a problem hiding this comment.
The other problem I run into when playing with this was:
OC_ENABLE_CMD_INSPECT_ALERTS=true ./oc adm inspect-alerts
Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority
I'm guessing you'd need to do something similar to what we do in https://github.com/openshift/must-gather/blob/master/collection-scripts/gather_monitoring to verify that route, but I'm hesitant about that approach.
There was a problem hiding this comment.
Isn't there a stable API we could access? How our console is accessing that data? Can we re-use that same API?
There was a problem hiding this comment.
must-gather is using a ServiceAccount token and collecting the default ingress CA. Both of which we could use. But 🤷, also work to implement, and not clear to me that we need to sink that time in before landing a subcommand behind a OC_ENABLE_CMD_INSPECT_ALERTS=true gate. Is "if there isn't a bearer token or the request is rejected with an HTTP auth code, go see if we have the power to borrow a must-gather ServiceAccount token" the sort of thing we need to land the initial implementation? There's always work we could invest to make any implementation better, but where are the goal-posts for merging an initial, env-var-gated command?
There was a problem hiding this comment.
RE: stable API , we can take up the work post the merge. This should not block the merge as this PR is behind OC_ENABLE_CMD_INSPECT_ALERTS=true flag.
There was a problem hiding this comment.
Isn't there a stable API we could access? How our console is accessing that data? Can we re-use that same API?
The console uses the same API service to query and display the cluster alerts.
|
/approve |
openshift-ci
bot
added
the
approved
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ardaguclu, LalatenduMohanty, wking The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@wking: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
[ART PR BUILD NOTIFIER] This PR has been included in build openshift-enterprise-cli-container-v4.16.0-202402230438.p0.ga36a8e3.assembly.stream.el8 for distgit openshift-enterprise-cli. |
The new subcommand is gated by
OC_ENABLE_CMD_INSPECT_ALERTSto avoid customers accidentally using it while it's tech-preview:This provides a more convenient syntax based on the existing docs':
But I'm using the
/alertspath instead of the/receiverspath given in the docs example. This provides convenient access when the console isn't available (e.g. because theConsoleClusterVersion capability is not enabled) or for use in other tooling like the tech-previewoc adm upgrade status, and has a long-standing (but currently deferred) RFE.I'm also dropping the
-k/--insecurefrom the doc'ed curl in my Go translation, because:I don't want to leak a sensitive token to an untrusted server.
Clusters are supposed to have set up a well-known cert/CA for their ingress:
so our client will likely trust the cluster.
and failing on "I don't trust that server" seems fine for this tech-preview proof-of-concept, vs. trying to use the kubeconfig-protected connections to automatically figure out which CA I need to trust.
I'm iterating over
status.ingress[].hostentries instead of using the doc'edspec.host, because thespecproperty isoptional, while thestatusproperty isrequired. It also seems safer to consume the Route controller's outputstatusinstead of wiring up directly to Route-admin writtenspecinput.I'm not currently filtering on
RouteIngress.Conditionsincluding anAdmitted=Trueentry, although something like that might be reasonable to avoid wasting time poking known-to-be-broken hosts.I haven't dug into
Bearertoken RFCs or the REST config structure to know if I need to worry about any encoding issues (base64?) in this handoff, but in testing it works for me. Please feel free to stiffen, or at least research, that handoff before making this generally available :).If I get a failure from one
status.ingress[].host, I don't bother attempting the others. Wiring up that kind of fallback with error aggregation when all hosts fail would be a useful future refinement.We seem to have types in the vendored
github.com/prometheus/common/modelthat could be used to deserialize this output (e.g. for table printing), but for now I'll just dump the JSON to output.