app/v1 show as extensions/v1beta1 when `kubectl get xxx -oyaml`

[gyliu513]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:

kubectl version

[root@ib17b07 ~]# kubectl version
Client Version: version.Info{Major:"", Minor:"", GitVersion:"v1.9.0+icp-ee", GitCommit:"63bab888f1dbf8e3266be14305e84d9ea539bffb", GitTreeState:"clean", BuildDate:"2018-01-08T14:29:32Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"", Minor:"", GitVersion:"v1.9.0+icp-ee", GitCommit:"63bab888f1dbf8e3266be14305e84d9ea539bffb", GitTreeState:"clean", BuildDate:"2018-01-08T14:29:32Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

Deployment file with apps/v1 as apiVersion.

[root@ib17b07 ~]# cat nginx.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: test-nginx
  name: test-nginx
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      run: test-nginx
  template:
    metadata:
      labels:
        run: test-nginx
    spec:
      containers:
      - image: nginx:1.8.1
        imagePullPolicy: IfNotPresent
        name: test-nginx
        ports:
        - containerPort: 80
          protocol: TCP

Create the deployment

kubectl apply -f ./nginx.yaml

After deployment finished, check the deployment yaml output, it was still using

[root@ib17b07 ~]# kubectl get deploy -oyaml, the apiVersion was still `extensions/v1beta1`.
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    annotations:
      deployment.kubernetes.io/revision: "1"
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"run":"test-nginx"},"name":"test-nginx","namespace":"default"},"spec":{"replicas":1,"selector":{"matchLabels":{"run":"test-nginx"}},"template":{"metadata":{"labels":{"run":"test-nginx"}},"spec":{"containers":[{"image":"nginx:1.8.1","imagePullPolicy":"IfNotPresent","name":"test-nginx","ports":[{"containerPort":80,"protocol":"TCP"}]}]}}}}
    creationTimestamp: 2018-01-11T04:15:29Z
    generation: 1
    labels:
      run: test-nginx
    name: test-nginx
    namespace: default
    resourceVersion: "97558"
    selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/test-nginx
    uid: 0eb24b62-f686-11e7-9b10-3440b5c70c60
  spec:
    progressDeadlineSeconds: 600
    replicas: 1
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        run: test-nginx
    strategy:
      rollingUpdate:
        maxSurge: 25%
        maxUnavailable: 25%
      type: RollingUpdate
    template:
      metadata:
        creationTimestamp: null
        labels:
          run: test-nginx
      spec:
        containers:
        - image: nginx:1.8.1
          imagePullPolicy: IfNotPresent
          name: test-nginx
          ports:
          - containerPort: 80
            protocol: TCP
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
        dnsPolicy: ClusterFirst
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext: {}
        terminationGracePeriodSeconds: 30
  status:
    availableReplicas: 1
    conditions:
    - lastTransitionTime: 2018-01-11T04:15:32Z
      lastUpdateTime: 2018-01-11T04:15:32Z
      message: Deployment has minimum availability.
      reason: MinimumReplicasAvailable
      status: "True"
      type: Available
    - lastTransitionTime: 2018-01-11T04:15:29Z
      lastUpdateTime: 2018-01-11T04:15:32Z
      message: ReplicaSet "test-nginx-647c4777c8" has successfully progressed.
      reason: NewReplicaSetAvailable
      status: "True"
      type: Progressing
    observedGeneration: 1
    readyReplicas: 1
    replicas: 1
    updatedReplicas: 1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

/sig api-machinery
What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[gyliu513]

@liggitt @kow3ns

[liggitt]

kubectl get deployment foo is ambiguous, since the server has deployments in multiple api groups. When a resource exists in multiple api groups, kubectl uses the first group listed in discovery docs published by the server which contains the resource. For backwards compatibility, that is the extensions api group.

If you want to ensure you get a deployment in the apps api group, fully qualify the resource you are requesting by running kubectl get deployments.apps test-nginx

If you want a specific version, like v1, in the apps api group, include that as well: kubectl get deployments.v1.apps test-nginx

[inge4pres]

Hi @liggitt

I appreciate the explanation, at the same time we see the behaviour of kubectl get deployments.v1.apps test-nginx is showing deployment that is stored as extensions/v1beta1 in etcd.
We'd expect that specifying the schema version would not result in fetched objects with a different apiVersion, otherwise when performing migrations how are we supposed to understand what is running with the new version and what needs to be still migrated?

We are willing to help to address this if it's any relevant for anyone...

[liggitt]

we see the behaviour of kubectl get deployments.v1.apps test-nginx is showing deployment that is stored as extensions/v1beta1 in etcd.

correct. that is working properly.

We'd expect that specifying the schema version would not result in fetched objects with a different apiVersion, otherwise when performing migrations how are we supposed to understand what is running with the new version and what needs to be still migrated?

We are willing to help to address this if it's any relevant for anyone...

it is intentional that internal storage details are not visible via the external versioned APIs. if we wanted to expose storage details, that information would be communicated separately, but I don't actually think that is necessary.

see https://docs.google.com/document/d/1eoS1K40HLMl4zUyw5pnC05dEF3mzFLp5TPEEt4PFvsM/edit# for some discussion of the options here.

the simplest approach is to get/put every object after upgrades. objects that don't need migration will no-op (they won't even increment resourceVersion in etcd). objects that do need migration will persist in the new preferred storage version.

[redbaron]

This is an interesting topic, which doesn't have much coverage in Kubernetes community.

What stored version is used for? Does stored api version influence behaviour of controllers, in other words can they potentially interpret same values and fields differently depending on the stored object's api version? Is it possible find out stored API version of the object without querying etcd directly? When it is safe to disable api groups like apps/v1beta* on API server and why would one want to do so at all?

the simplest approach is to get/put every object after upgrades.

how exactly it can be done? is it enough to do something like kubectl get deployment --all-namespaces -o json | kubectl convert -f - -o json | kubectl replace -f - ?

[liggitt]

What stored version is used for?

It is the serialized version in etcd. Whenever you fetch an object, the api server reads it from etcd, converts it into an internal version, then converts it to the version you requested.

Does stored api version influence behaviour of controllers, in other words can they potentially interpret same values and fields differently depending on the stored object's api version?

Controllers always get the version they requested from the API. They are not exposed to the stored version.

Is it possible find out stored API version of the object without querying etcd directly?

No

When it is safe to disable api groups like apps/v1beta* on API server and why would one want to do so at all?

Disabling the groups from being served doesn't disable the ability of the apiserver to decode objects in those versions from etcd.

is it enough to do something like kubectl get deployment --all-namespaces -o json | kubectl convert -f - -o json | kubectl replace -f - ?

no conversion is necessary... a simple get/put of raw bytes is sufficient

something like kubectl get deployment --all-namespaces -o json | kubectl replace -f - would be close, though you'd probably want something that handled retry on conflicts and tolerated objects not existing any more to be robust.

[redbaron]

@liggitt , thanks for clarifying, it seems that internal version can affect users only when apiserver can't decode stored objects, which shouldn't happen unless conversion controller is implemented. It explains why this topic is not covered widely - there is no much user-visible effect, which just shows great work Kubernetes developers are doing.

For curious minds, I found an extensive doc describing internal mechanics here https://github.com/kubernetes/community/blob/master/contributors/devel/api_changes.md

[daxmc99]

For curious minds, I found an extensive doc describing internal mechanics here https://github.com/kubernetes/community/blob/master/contributors/devel/api_changes.md

Current link is here https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md