kubectl cannot get flowschemas

[MikeSpreitzer]

What happened:
I am testing API Priority and Fairness. I launched it like this:

RUNTIME_CONFIG=flowcontrol.apiserver.k8s.io/v1alpha1=true FEATURE_GATES=APIPriorityAndFairness=true LOG_LEVEL=7 hack/local-up-cluster.sh

After the logging said the cluster is running and told me how to set KUBECONFIG, I did so in another shell and went looking for FlowSchema objects. But kubectl erroneously told me there is no such thing.

$ kubectl get Pod -A
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   kube-dns-74b889989-jh8zn   3/3     Running   0          49s

$ kubectl get flowschemas
error: the server doesn't have a resource type "flowschemas"

$ kubectl get flowschemas.flowcontrol.apiserver.k8s.io
error: the server doesn't have a resource type "flowschemas"

$ kubectl get flowschemas.v1alpha1.flowcontrol.apiserver.k8s.io
error: the server doesn't have a resource type "flowschemas"

However, looking in the kube-apiserver log shows flowschemas and prioritylevelconfigurations being read and written; for example:

I0118 06:00:08.222289   75066 httplog.go:90] verb="GET" URI="/apis/flowcontrol.apiserver.k8s.io/v1alpha1/flowschemas/catch-all" latency=954.726µs resp=200 UserAgent="kube-apiserver/v1.18.0 (linux/amd64) kubernetes/916edd9" srcIP="[::1]:48830": 

Also, I am able to read them with curl against the insecure port.

Interestingly, the kube-apiserver log also includes this complaint:

I0118 05:58:05.655870   75066 aggregator.go:182] Skipping APIService creation for flowcontrol.apiserver.k8s.io/v1alpha1

I will attach the kube-apiserver log.

What you expected to happen:
I expected kubectl get flowschemas to list them.

How to reproduce it (as minimally and precisely as possible):
I think the report above is pretty minimal.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): this is the master branch, git commit 916edd9

• Cloud provider or hardware configuration: bare metal, intel, 16 "CPUs", 64 GB of memory

• OS (e.g: cat /etc/os-release): 18.04.2 LTS (Bionic Beaver)

• Kernel (e.g. uname -a): Linux mjs-dev-1 4.15.0-74-generic gofmt -s pkg/ cmd/ #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Install tools: hack/local-up-cluster.sh

• Network plugin and version (if this is a network-related bug):

• Others:
  Docker version 18.09.7
  go version go1.13.4 linux/amd64
  kube-apiserver.log

[MikeSpreitzer]

/sig api-machinery

[MikeSpreitzer]

/cc @yue9944882
/cc @lavalamp
/cc @deads2k
/cc @jktomer

[MikeSpreitzer]

/assign @yue9944882

[liggitt]

the "flowcontrol.apiserver.k8s.io" API group is not included in the list of known groups to register at startup:

kubernetes/cmd/kube-apiserver/app/aggregator.go

Lines 248 to 294 in 37d9c22

	var apiVersionPriorities = map[schema.GroupVersion]priority{
	{Group: "", Version: "v1"}: {group: 18000, version: 1},
	// extensions is above the rest for CLI compatibility, though the level of unqualified resource compatibility we
	// can reasonably expect seems questionable.
	{Group: "extensions", Version: "v1beta1"}: {group: 17900, version: 1},
	// to my knowledge, nothing below here collides
	{Group: "apps", Version: "v1"}: {group: 17800, version: 15},
	{Group: "events.k8s.io", Version: "v1beta1"}: {group: 17750, version: 5},
	{Group: "authentication.k8s.io", Version: "v1"}: {group: 17700, version: 15},
	{Group: "authentication.k8s.io", Version: "v1beta1"}: {group: 17700, version: 9},
	{Group: "authorization.k8s.io", Version: "v1"}: {group: 17600, version: 15},
	{Group: "authorization.k8s.io", Version: "v1beta1"}: {group: 17600, version: 9},
	{Group: "autoscaling", Version: "v1"}: {group: 17500, version: 15},
	{Group: "autoscaling", Version: "v2beta1"}: {group: 17500, version: 9},
	{Group: "autoscaling", Version: "v2beta2"}: {group: 17500, version: 1},
	{Group: "batch", Version: "v1"}: {group: 17400, version: 15},
	{Group: "batch", Version: "v1beta1"}: {group: 17400, version: 9},
	{Group: "batch", Version: "v2alpha1"}: {group: 17400, version: 9},
	{Group: "certificates.k8s.io", Version: "v1beta1"}: {group: 17300, version: 9},
	{Group: "networking.k8s.io", Version: "v1"}: {group: 17200, version: 15},
	{Group: "networking.k8s.io", Version: "v1beta1"}: {group: 17200, version: 9},
	{Group: "policy", Version: "v1beta1"}: {group: 17100, version: 9},
	{Group: "rbac.authorization.k8s.io", Version: "v1"}: {group: 17000, version: 15},
	{Group: "rbac.authorization.k8s.io", Version: "v1beta1"}: {group: 17000, version: 12},
	{Group: "rbac.authorization.k8s.io", Version: "v1alpha1"}: {group: 17000, version: 9},
	{Group: "settings.k8s.io", Version: "v1alpha1"}: {group: 16900, version: 9},
	{Group: "storage.k8s.io", Version: "v1"}: {group: 16800, version: 15},
	{Group: "storage.k8s.io", Version: "v1beta1"}: {group: 16800, version: 9},
	{Group: "storage.k8s.io", Version: "v1alpha1"}: {group: 16800, version: 1},
	{Group: "apiextensions.k8s.io", Version: "v1"}: {group: 16700, version: 15},
	{Group: "apiextensions.k8s.io", Version: "v1beta1"}: {group: 16700, version: 9},
	{Group: "admissionregistration.k8s.io", Version: "v1"}: {group: 16700, version: 15},
	{Group: "admissionregistration.k8s.io", Version: "v1beta1"}: {group: 16700, version: 12},
	{Group: "scheduling.k8s.io", Version: "v1"}: {group: 16600, version: 15},
	{Group: "scheduling.k8s.io", Version: "v1beta1"}: {group: 16600, version: 12},
	{Group: "scheduling.k8s.io", Version: "v1alpha1"}: {group: 16600, version: 9},
	{Group: "coordination.k8s.io", Version: "v1"}: {group: 16500, version: 15},
	{Group: "coordination.k8s.io", Version: "v1beta1"}: {group: 16500, version: 9},
	{Group: "auditregistration.k8s.io", Version: "v1alpha1"}: {group: 16400, version: 1},
	{Group: "node.k8s.io", Version: "v1alpha1"}: {group: 16300, version: 1},
	{Group: "node.k8s.io", Version: "v1beta1"}: {group: 16300, version: 9},
	{Group: "discovery.k8s.io", Version: "v1beta1"}: {group: 16200, version: 12},
	{Group: "discovery.k8s.io", Version: "v1alpha1"}: {group: 16200, version: 9},
	// Append a new group to the end of the list if unsure.
	// You can use min(existing group)-100 as the initial value for a group.
	// Version can be set to 9 (to have space around) for a new group.
	}

That means resources inside that group will not appear in discovery, and clients like kubectl will not know how to find them.

[liggitt]

You can see all resources visible via discovery by running kubectl api-resources

[yue9944882]

just as @liggitt said, for now we have to manually create an apiservice into the cluster to get kubectl work. i dont think we're adding this to the list until the beta stage, can we make an exception for the case? @kubernetes/sig-api-machinery-misc

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    kube-aggregator.kubernetes.io/automanaged: onstart
  name: v1alpha1.flowcontrol.apiserver.k8s.io
spec:
  group: flowcontrol.apiserver.k8s.io
  groupPriorityMinimum: 16000
  service: null
  version: v1alpha1
  versionPriority: 1

[liggitt]

It is fine to add alpha things to the list I linked so they get created automatically if needed. They only get registered if the relevant runtime-config is enabled.