ScheduledJob kubectl

[soltysh]

@erictune the last part (kubectl) of the ScheduledJob, as usual builds on top of previous PRs, so only last 2 commits matter (the kubectl one and storage leftovers).

* Introducing ScheduledJobs as described in [the proposal](https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/scheduledjob.md) as part of `batch/v2alpha1` version (experimental feature).

[soltysh]

@lavalamp I'm having some problems with initiating storage for scheduledjobs, iow. the batch/v2alpha1 groups, see the following error I'm getting. What am I missing, got some ideas?

I0518 15:38:12.822156   25767 storage_factory.go:217] storing {batch scheduledjobs} in batch/v1, reading as batch/__internal from { {0xc82037b6c0 0xc82037b730} /registry [http://127.0.0.1:4001]    false 50000}
panic: storage codec doesn't seem to match given type: Internal type not encodable: no kind "ScheduledJob" is registered for version "batch/v1"

goroutine 1 [running]:
panic(0x1f0f1a0, 0xc820488a10)
        /usr/lib/golang/src/runtime/panic.go:481 +0x3e6
k8s.io/kubernetes/pkg/storage.NewCacherFromConfig(0x64, 0x7f524108c398, 0xc8201a3400, 0x7f524108c360, 0x41b9228, 0x2b03cc0, 0xc820183400, 0x2b5c730, 0xe, 0xc820570bc0, ...)
        /home/maszulik/workspace/k8s/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/storage/cacher.go:156 +0x817
k8s.io/kubernetes/pkg/storage.NewCacher(0x7f524108c398, 0xc8201a3400, 0x64, 0x7f524108c360, 0x41b9228, 0x7f52410888e0, 0xc820183400, 0x2b5c730, 0xe, 0x7f52410c37b8, ...)
        /home/maszulik/workspace/k8s/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/storage/cacher.go:142 +0x185
k8s.io/kubernetes/pkg/registry/generic/registry.StorageWithCacher(0x7f524108c398, 0xc8201a3400, 0x64, 0x7f52410888e0, 0xc820183400, 0x2b5c730, 0xe, 0x7f52410c37b8, 0xc820570ba0, 0x32911e8, ...)
        /home/maszulik/workspace/k8s/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/generic/registry/storage_factory.go:36 +0x101
k8s.io/kubernetes/pkg/registry/scheduledjob/etcd.NewREST(0x7f524108c398, 0xc8201a3400, 0x3290fa8, 0x1, 0x3290fa8, 0x1)
        /home/maszulik/workspace/k8s/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/scheduledjob/etcd/etcd.go:41 +0x1cc
k8s.io/kubernetes/pkg/master.(*Master).getBatchResources(0xc8203ad930, 0xc820332fa0, 0x2b4dd78, 0x5, 0x2b92c90, 0x8, 0xc82037b260)
        /home/maszulik/workspace/k8s/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/master/master.go:844 +0x733
k8s.io/kubernetes/pkg/master.(*Master).InstallAPIs(0xc8203ad930, 0xc820332fa0)
        /home/maszulik/workspace/k8s/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/master/master.go:331 +0x2751
k8s.io/kubernetes/pkg/master.New(0xc820332fa0, 0x0, 0x0, 0x0)
        /home/maszulik/workspace/k8s/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/master/master.go:173 +0x189
k8s.io/kubernetes/cmd/kube-apiserver/app.Run(0xc8202c7680, 0x0, 0x0)
        /home/maszulik/workspace/k8s/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:239 +0x2e3c
main.main()
        /home/maszulik/workspace/k8s/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver.go:50 +0x121

From my debugging it looks like the storagefactory is picking the wrong codec for ScheduledJob, do you have some thoughts where should I register one to be properly picked up?

[lavalamp]

I have a long queue right now so sorry for the cryptic answer-- you need to somehow override the storage version for just the ScheduledJob resource to be v2alpha1 since it doesn't exist in v1. Not sure if the existing flag is going to make that easy or not.

[soltysh]

@lavalamp thanks, will try to poke around and see what I can come up with.

[soltysh]

@lavalamp I've noticed there's a SetResourceEncoding which is not used currently but does exactly what you've mentioned. Iow. it allows specifying a storage version for specific resource, it's not exposed, though. Mind having a look and letting me know if it's more or less what you'd expect to look like?

[soltysh]

@erictune @lavalamp with the current state I am able to get & create scheduledjobs using curl, but trying to use kubectl fails everytime with error when creating "scheduledjob.yaml": no kind "ScheduledJob" is registered for version "batch/v1" even when using kubectl get scheduledjobs.v2alpha1.batch which should reach the right endpoint. Any ideas?

[lavalamp]

@soltysh What's the file you're creating look like? it needs to have batch/v2alpha1 as the apiVersion?

Also presumably you built an new kubectl?

[soltysh]

@lavalamp this is the resource I'm trying to create: https://gist.github.com/soltysh/a917d4aac4cbccbff690117c0fc99a6c, but I still hit error: error when creating "scheduledjob.yaml": no kind "ScheduledJob" is registered for version "batch/v1

[janetkuo]

just a heads up that #29187 is in now

[soltysh]

just a heads up that #29187 is in now

Yup, seen it. I'm currently updating this one. Should be ready in a few minutes.

[soltysh]

Rebased and addressed all your comments @janetkuo . The only problem I'm struggling with is hack/update-generated-protobuf.sh is failing for me locally, but hopefully that won't be a problem with this PR. ptal

[janetkuo]

Code looks good, but the panic: storage codec doesn't seem to match given type: Internal type not encodable: no kind "ScheduledJob" is registered for version "batch/v1" error is back again

[soltysh]

Code looks good, but the panic: storage codec doesn't seem to match given type: Internal type not encodable: no kind "ScheduledJob" is registered for version "batch/v1" error is back again

Of course it does... 😭 I'll try to figure it out later today.

[soltysh]

This is a problem with integration tests that I need to fix to address multiversion tests.

[soltysh]

@janetkuo the problem was that I've added batchapi.v2alpha1.SchemeGroupVersion in DefaultAPIResourceConfigSource but I should not. I've removed it, but I've extended our integration tests with batch/v2alpha1 version.

[soltysh]

There's still one problem, already filled in issue #29998, but I think we'll have to deal with it as a followup. I'd like to move forward with the existing SJ PRs.

[soltysh]

It looks like the rbac is not working as expected against multiversioned batch, I've created #30079 to track this one and I'm removing the last commit (adding batch/v2alpha1 to integration) from this PR.

[soltysh]

@erictune @janetkuo ptal

[k8s-bot]

GCE e2e build/test passed for commit b5c68a9.

• Test Results
• Build Log
• Test Artifacts
• Internal Jenkins Results

[k8s-github-robot]

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

[k8s-bot]

GCE e2e build/test passed for commit b5c68a9.

• Test Results
• Build Log
• Test Artifacts
• Internal Jenkins Results

[k8s-github-robot]

Automatic merge from submit-queue