Include OpenShift cluster version in the telemetry

[czeslavo]

Is there an existing issue for this?

• I have searched the existing issues

Problem Statement

Currently, it's not possible to tell what OpenShift cluster versions KIC is running on in the wild. We would like to know it in order to broaden our context on the usage of KIC with OpenShift that is needed to make a proper decision on support policy for OpenShift.

Proposed Solution

• Spin up an OpenShift cluster and ensure that the reported cluster version is not distinguishable indeed.
• Find out how we can detect the OpenShift cluster version in runtime (oc CLI, ClusterVersion custom resource, node names heuristics?)
• Implement the most appealing heuristic and report OpenShift version field in the telemetry

Additional information

Initial investigation of the existing telemetry data has shown that the telemetry does include a Kubernetes cluster version which is not special in any sense for OpenShift clusters. Probably some dedicated heuristic is going to be needed to detect the OpenShift version.

Acceptance Criteria

• As a KIC developer, based on the telemetry data I can tell what OpenShift cluster versions KIC is being run on

[czeslavo]

ClusterVersion CRD is worth looking into. Probably we could read it/watch it and inspect its status.desired field.

[czeslavo]

I didn't know about the library we have for telemetry (that we don't use in KIC yet). I think it might be worth considering incorporating OpenShift detection in there and using the library.

[pmalek]

I didn't know about the library we have for telemetry (that we don't use in KIC yet). I think it might be worth considering incorporating OpenShift detection in there and using the library.

That's evidently a miscommunication on my end. But nevertheless putting it in a library is one thing, implementing is another. If we figure out how to detect said version I'll happily help to incorporate that into the library and use it in KIC

[rainest]

tl;dr there are some things that should provide the correct version with high confidence, but we don't currently have permissions to them and I think we should avoid adding new permissions if possible. There's something we do have access to already that looks like it will provide the OpenShift version, but with less confidence. Do we want to go ahead and use the latter?

ClusterVersion is what the oc CLI uses to print the OpenShift version: https://github.com/openshift/oc/blob/bdc5bb5b66e06e77e05ed3290c15ebfff61fa23f/pkg/cli/version/version.go#L138C51-L146

However we would not normally have access to them, and OpenShift also seems weirdly cagey about granting access to them. Using a non-admin user to run oc -v9 version will indeed show

I0615 17:01:44.652805   30665 request.go:1171] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"clusterversions.config.openshift.io \"version\" is forbidden: User \"developer\" cannot get resource \"clusterversions\" in API group \"config.openshift.io\" at the cluster scope","reason":"Forbidden","details":{"name":"version","group":"config.openshift.io","kind":"clusterversions"},"code":403}
I0615 17:01:44.653172   30665 version.go:143] OpenShift Version not found (must be logged in to cluster as admin): clusterversions.config.openshift.io "version" is forbidden: User "developer" cannot get resource "clusterversions" in API group "config.openshift.io" at the cluster scope

shift.yaml.txt is a kubectl get all -A -oyaml dump of all (well, most, since "all" isn't actually all) resources on a fresh 4.13 install. I also reviewed CRDs (not included in all), but it doesn't look like they're an option:

17:08:57-0700 esenin $ kubectl get crd -oyaml | grep -C4 4.13 
                      enum:
                      - None
                      - v4.11
                      - v4.12
                      - v4.13
                      - vCurrent
                      type: string
                  type: object
                channel:

Several replication controllers (Deployments and similar) do have a release.openshift.io/version annotation, but we don't currently have permission to those. We do have access to Pods (though I forget exactly why--they don't seem like something we should need access to, at least outside our own namespace) already, and the various built-in operators have information that we can probably use:

17:19:10-0700 esenin $ kubectl get po -n openshift-apiserver-operator -ojson | jq .items[0].spec.containers[0].env
[
  {
    "name": "IMAGE",
    "value": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef9f729c0a5759a573009d0009cfe878fe4415b7bf6311aeda1cf3478df8d517"
  },
  {
    "name": "OPERATOR_IMAGE",
    "value": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e4653a8f77aa893fe69bacb94d1f83e76756d51e75000d25c7e946dbdc1a533"
  },
  {
    "name": "OPERATOR_IMAGE_VERSION",
    "value": "4.13.0"
  },
  {
    "name": "OPERAND_IMAGE_VERSION",
    "value": "4.13.0"
  },
  {
    "name": "KUBE_APISERVER_OPERATOR_IMAGE",
    "value": "quay.io/crcont/openshift-crc-cluster-kube-apiserver-operator@sha256:31118a93da9b01f37280b37276ec2396002f21945c54532254376d37c148e605"
  }
]

The OpenShift operators look like they all consistently use the same version as the OpenShift version they're bundled with, so that OPERATOR_IMAGE_VERSION variable is an option. That's only based on review of the actual values though--I don't have any guarantee that the versions matching is always the case, though it seems reasonable to expect that it is.

[rainest]

Kong/kubernetes-telemetry#168 and #4211 work to ascertain the cluster version on 4.13, which is the current version CRC installs.

My choice of operator to probe was a best-guess about things that are likely to be consistent across versions. The API server is important and the associated operator has been around for ~4y: https://github.com/openshift/cluster-openshift-apiserver-operator

Attempting to try and confirm signal viability across older versions, but tools for deploying older versions of OpenShift locally appear to be quite bitrotten (especially as far as docs go), so a bit poking around in the dark to see what works.