diff --git a/README.md b/README.md index 669a9a0..74d9234 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,6 @@ [![In Progress](https://badge.waffle.io/nikhita/gsoc-meta-k8s.svg?label=In%20Progress&title=In%20Progress)](http://waffle.io/nikhita/gsoc-meta-k8s) - Issues created in this repository correspond to the tasks I'm working on. Please see the waffle board linked below for a better perspective of how the project is progressing. ## Student Info @@ -18,13 +17,17 @@ Issues created in this repository correspond to the tasks I'm working on. Please + [GSoC Proposal](notes/gsoc-proposal.pdf) + [Project on GSoC website](https://summerofcode.withgoogle.com/organizations/6540924424290304/#5982049109278720) -+ [TPR features issue](https://github.com/kubernetes/features/issues/95) -+ [Design Proposal: Validation for CustomResources](https://github.com/kubernetes/community/pull/708) ++ [TPR/CRD features issue](https://github.com/kubernetes/features/issues/95) + [Waffle board](https://waffle.io/nikhita/gsoc-meta-k8s) + [Daily update log](https://docs.google.com/document/d/1iCH03_jdyUsBfXKwgZtut7tXVbsl6lgusDt_lXLNNGo/edit?usp=sharing) + [cncf-soc github repo](https://github.com/cncf/soc) + [CNCF Blog post on GSoC](https://www.cncf.io/blog/2017/05/04/cncf-brings-kubernetes-coredns-opentracing-prometheus-google-summer-code-2017/) +## Design Proposals + ++ [Design Proposal: Validation for CustomResources](https://github.com/kubernetes/community/pull/708) ++ [Design Proposal: SubResources for CustomResources](https://github.com/kubernetes/community/pull/913) + ## Weekly Summary ### Community Bonding Period - May @@ -55,50 +58,41 @@ Issues created in this repository correspond to the tasks I'm working on. Please + [Week 15](weekly/week15.md) + [Week 16](weekly/week16.md) -## Blog posts - -A RSS feed for blog posts only related to GSoC is available [here](https://nikhita.github.io/feed-gsoc.xml). - -Note: I have decided not to write weekly updates as blog posts anymore but to include it as a quick weekly summary only. The blog posts will focus more on technical content. - -+ [GSoC with Kubernetes - Week 1](https://nikhita.github.io/google-summer-of-code-kubernetes-week-1) -+ [GSoC with Kubernetes - Week 2 and 3](https://nikhita.github.io/google-summer-of-code-kubernetes-week2-3-community-bonding) - - ## Pull Requests and Issues The following list is automatically generated using https://github.com/nikhita/github-contrib. **Repository: kubernetes** -Total Pull Requests Created: 27 -1. [kubernetes/kubernetes#50964](https://github.com/kubernetes/kubernetes/pull/50964) - Update Registry interface for deployment and endpoints -2. [kubernetes/kubernetes#50764](https://github.com/kubernetes/kubernetes/pull/50764) - apiextensions: update CRD strategy -3. [kubernetes/kubernetes#50638](https://github.com/kubernetes/kubernetes/pull/50638) - FeatureGate: update comments -4. [kubernetes/kubernetes#50444](https://github.com/kubernetes/kubernetes/pull/50444) - jsonpath: fix comments -5. [kubernetes/kubernetes#50250](https://github.com/kubernetes/kubernetes/pull/50250) - Automated cherry pick of #50098 -6. [kubernetes/kubernetes#50098](https://github.com/kubernetes/kubernetes/pull/50098) - apiextensions: fix data race in storage -7. [kubernetes/kubernetes#50085](https://github.com/kubernetes/kubernetes/pull/50085) - fix kube-openapi imports -8. [kubernetes/kubernetes#49747](https://github.com/kubernetes/kubernetes/pull/49747) - conversion-gen: support recursive types -9. [kubernetes/kubernetes#49307](https://github.com/kubernetes/kubernetes/pull/49307) - fuzzer: remove unreachable code -10. [kubernetes/kubernetes#48630](https://github.com/kubernetes/kubernetes/pull/48630) - update vendored gengo -11. [kubernetes/kubernetes#48389](https://github.com/kubernetes/kubernetes/pull/48389) - apiextensions: add cleanup section to client-go -12. [kubernetes/kubernetes#48114](https://github.com/kubernetes/kubernetes/pull/48114) - Automated cherry pick of #47748 -13. [kubernetes/kubernetes#48076](https://github.com/kubernetes/kubernetes/pull/48076) - Fix error in local-cluster-up -14. [kubernetes/kubernetes#47748](https://github.com/kubernetes/kubernetes/pull/47748) - Update custom-resources example in client-go -15. [kubernetes/kubernetes#47684](https://github.com/kubernetes/kubernetes/pull/47684) - Fix link to apiextensions client-go example -16. [kubernetes/kubernetes#47263](https://github.com/kubernetes/kubernetes/pull/47263) - apiextensions: validation for customresources -17. [kubernetes/kubernetes#46624](https://github.com/kubernetes/kubernetes/pull/46624) - Add test for advanced CRUD for apiextensions -18. [kubernetes/kubernetes#46585](https://github.com/kubernetes/kubernetes/pull/46585) - [WIP] apiextensions: add integration test for GC -19. [kubernetes/kubernetes#46200](https://github.com/kubernetes/kubernetes/pull/46200) - apiextensions: add integration test for name conflicts -20. [kubernetes/kubernetes#46059](https://github.com/kubernetes/kubernetes/pull/46059) - Integration test for kube-apiextensions-server: integers -21. [kubernetes/kubernetes#45793](https://github.com/kubernetes/kubernetes/pull/45793) - Add plural name for CustomResources example -22. [kubernetes/kubernetes#45721](https://github.com/kubernetes/kubernetes/pull/45721) - Add integration tests for kube-apiextensions-server -23. [kubernetes/kubernetes#44612](https://github.com/kubernetes/kubernetes/pull/44612) - Fix kube-apiserver crash when patching TPR data -24. [kubernetes/kubernetes#44026](https://github.com/kubernetes/kubernetes/pull/44026) - Preserve int data when unmarshalling for TPR -25. [kubernetes/kubernetes#43606](https://github.com/kubernetes/kubernetes/pull/43606) - Improve timeout error message for kubectl delete -26. [kubernetes/kubernetes#43591](https://github.com/kubernetes/kubernetes/pull/43591) - Update `kubectl help` descriptions and examples -27. [kubernetes/kubernetes#43573](https://github.com/kubernetes/kubernetes/pull/43573) - fixed formatting for examples README.md +Total Pull Requests Created: 28 +1. [kubernetes/kubernetes#51204](https://github.com/kubernetes/kubernetes/pull/51204) - roundtrip: fix error messages +2. [kubernetes/kubernetes#50964](https://github.com/kubernetes/kubernetes/pull/50964) - Update Registry interface for deployment and endpoints +3. [kubernetes/kubernetes#50764](https://github.com/kubernetes/kubernetes/pull/50764) - apiextensions: update CRD strategy +4. [kubernetes/kubernetes#50638](https://github.com/kubernetes/kubernetes/pull/50638) - FeatureGate: update comments +5. [kubernetes/kubernetes#50444](https://github.com/kubernetes/kubernetes/pull/50444) - jsonpath: fix comments +6. [kubernetes/kubernetes#50250](https://github.com/kubernetes/kubernetes/pull/50250) - Automated cherry pick of #50098 +7. [kubernetes/kubernetes#50098](https://github.com/kubernetes/kubernetes/pull/50098) - apiextensions: fix data race in storage +8. [kubernetes/kubernetes#50085](https://github.com/kubernetes/kubernetes/pull/50085) - fix kube-openapi imports +9. [kubernetes/kubernetes#49747](https://github.com/kubernetes/kubernetes/pull/49747) - conversion-gen: support recursive types +10. [kubernetes/kubernetes#49307](https://github.com/kubernetes/kubernetes/pull/49307) - fuzzer: remove unreachable code +11. [kubernetes/kubernetes#48630](https://github.com/kubernetes/kubernetes/pull/48630) - update vendored gengo +12. [kubernetes/kubernetes#48389](https://github.com/kubernetes/kubernetes/pull/48389) - apiextensions: add cleanup section to client-go +13. [kubernetes/kubernetes#48114](https://github.com/kubernetes/kubernetes/pull/48114) - Automated cherry pick of #47748 +14. [kubernetes/kubernetes#48076](https://github.com/kubernetes/kubernetes/pull/48076) - Fix error in local-cluster-up +15. [kubernetes/kubernetes#47748](https://github.com/kubernetes/kubernetes/pull/47748) - Update custom-resources example in client-go +16. [kubernetes/kubernetes#47684](https://github.com/kubernetes/kubernetes/pull/47684) - Fix link to apiextensions client-go example +17. [kubernetes/kubernetes#47263](https://github.com/kubernetes/kubernetes/pull/47263) - apiextensions: validation for customresources :star: +18. [kubernetes/kubernetes#46624](https://github.com/kubernetes/kubernetes/pull/46624) - Add test for advanced CRUD for apiextensions +19. [kubernetes/kubernetes#46585](https://github.com/kubernetes/kubernetes/pull/46585) - [WIP] apiextensions: add integration test for GC +20. [kubernetes/kubernetes#46200](https://github.com/kubernetes/kubernetes/pull/46200) - apiextensions: add integration test for name conflicts +21. [kubernetes/kubernetes#46059](https://github.com/kubernetes/kubernetes/pull/46059) - Integration test for kube-apiextensions-server: integers +22. [kubernetes/kubernetes#45793](https://github.com/kubernetes/kubernetes/pull/45793) - Add plural name for CustomResources example +23. [kubernetes/kubernetes#45721](https://github.com/kubernetes/kubernetes/pull/45721) - Add integration tests for kube-apiextensions-server +24. [kubernetes/kubernetes#44612](https://github.com/kubernetes/kubernetes/pull/44612) - Fix kube-apiserver crash when patching TPR data +25. [kubernetes/kubernetes#44026](https://github.com/kubernetes/kubernetes/pull/44026) - Preserve int data when unmarshalling for TPR +26. [kubernetes/kubernetes#43606](https://github.com/kubernetes/kubernetes/pull/43606) - Improve timeout error message for kubectl delete +27. [kubernetes/kubernetes#43591](https://github.com/kubernetes/kubernetes/pull/43591) - Update `kubectl help` descriptions and examples +28. [kubernetes/kubernetes#43573](https://github.com/kubernetes/kubernetes/pull/43573) - fixed formatting for examples README.md Total Issues Opened: 4 1. [kubernetes/kubernetes#49811](https://github.com/kubernetes/kubernetes/issues/49811) - api-reference docs don't contain types from staging repos diff --git a/weekly/week10.md b/weekly/week10.md index ebfb913..9a9d963 100644 --- a/weekly/week10.md +++ b/weekly/week10.md @@ -2,4 +2,5 @@ ## Delivered this week -TODO: Add content here! \ No newline at end of file ++ Couldn't work much this week due to GopherCon + jet lag. :( ++ Added polling while creating CRDs in `resources.go`. \ No newline at end of file diff --git a/weekly/week11.md b/weekly/week11.md index fce08a4..8b41049 100644 --- a/weekly/week11.md +++ b/weekly/week11.md @@ -2,4 +2,15 @@ ## Delivered this week -TODO: Add content here! \ No newline at end of file ++ Thought about adding a readOnly field but decided against it since it will go against kube's way of handling resources. + - this is present in `hyperschema` but we don't use it now. ++ Got rid of the `enqueue` for informers. Fixed the cache invalidation. ++ Added custom deepcopy functions because we use interfaces now. Or else, it gives this error: https://gist.github.com/nikhita/8cc04bfc68357695625da2e1ac92ba9a#file-deecopy-txt-L10. + - maybe we can have a special tag for deepcopy gen to avoid writing big deepcopy functions. + - used the one for `Table` in meta as a reference. ++ Added custom fuzzer functions for interface fields. Why we need this: interface can have invalid JSON values too. This breaks round trip tests. Also, https://github.com/kubernetes/kubernetes/pull/49307. ++ Proto doesn't work too because of the interface fiels. A similar issue was faced in `Table`. Decided to not support proto for now. + - there is a file which contains all packages where proto is supported. removed apiextensions/v1beta1 from there. ++ Changed the JSON tags on the `$ref` fields. ++ Had a bit of trouble with `update-staging-godeps` that was also reported with someone else (https://github.com/kubernetes/kubernetes/issues/49210) Fixed with a clean checkout. ++ Refractored some more code! \ No newline at end of file diff --git a/weekly/week12.md b/weekly/week12.md index 3eccf8b..650a370 100644 --- a/weekly/week12.md +++ b/weekly/week12.md @@ -2,4 +2,14 @@ ## Delivered this week -TODO: Add content here! \ No newline at end of file ++ `TestCustomResourceValidationErrors` was now constantly failing. + - after a LOT of debugging, turns out there was a problem upstream. + - after even more debugging, turns out we could fix it on our own. :smile: ++ Added validations for validating that the _schema_ is correct (not only the custom resource). ++ Fixed some nits. ++ Updated the PR to get all tests *green*! :tada: ++ Discussed about defaulting the `$schema` field. Changing the defaults would break the api. So, defaulting seemed like a gray area. ++ Started with the sub-resources proposal + a LOT of research on existing resources. ++ Added a new PR for fixing conversion. Improved some renaming from the gengo PR: https://github.com/kubernetes/kubernetes/pull/49747. ++ The api-reference doc was not updated for the apiextensions. Turned to be a deeper issue than that. Created https://github.com/kubernetes/kubernetes/issues/49811. ++ Since the GC fix was in place with tests, closed down the earlier GC intergration test. \ No newline at end of file diff --git a/weekly/week13.md b/weekly/week13.md index 9f089cb..13599de 100644 --- a/weekly/week13.md +++ b/weekly/week13.md @@ -2,4 +2,14 @@ ## Delivered this week -TODO: Add content here! \ No newline at end of file ++ Created the spreadsheet after lots of discussion about changing the JSONSchema struct: https://docs.google.com/spreadsheets/d/1Mkm9L7CXGvRorV0Cr4Vwfu0DH7XRi24YHPiDK1NZWo4/edit?usp=sharing + - went through the openAPI spec + - Additional discussion: https://github.com/kubernetes/kubernetes/issues/49879. ++ Discussed more about subresources (integration in kubectl, etc). ++ Created the WIP subresources proposal gdoc and iterated a LOT over it. Asked anthony for a review too. ++ Added feature gate for `CustomResourceValidation`. + - fixed feature gate godoc: https://github.com/kubernetes/kubernetes/pull/50638. ++ There was a data race for CRD `storageMap`. + - Fixed that: https://github.com/kubernetes/kubernetes/pull/50098 + - Cherry-picked it to 1.7: https://github.com/kubernetes/kubernetes/pull/50250 ++ Fixed the fuzzer for JSONSchemaProps. \ No newline at end of file diff --git a/weekly/week15.md b/weekly/week15.md index 0f0833b..ed5d3f8 100644 --- a/weekly/week15.md +++ b/weekly/week15.md @@ -2,4 +2,14 @@ ## Delivered this week -TODO: Add content here! \ No newline at end of file ++ Refractored the existing feature gates even more. ++ Enabled feature gate in integration tests. ++ Changed `CustomResourceValidation` to `*CustomResourceValidation`. ++ Published the proposal: https://github.com/kubernetes/community/pull/913. ++ Started work on the subresources prototype. + - added the status and scale subresources. + - updated docs for json path: https://github.com/kubernetes/kubernetes/pull/50444, https://github.com/kubernetes/kubernetes.github.io/pull/4986. + - todo: add tests. ++ Noticed that the spec and status for CRDs did not support incrementing `metadata.Generation`. Created: https://github.com/kubernetes/kubernetes/pull/50764. ++ Addressed reviews in the proposal + made some changes to it. ++ Added custom marshal functions. But round-tripping broke so needed to update fuzzers. diff --git a/weekly/week16.md b/weekly/week16.md index dac12c6..a4fe547 100644 --- a/weekly/week16.md +++ b/weekly/week16.md @@ -2,4 +2,14 @@ ## Delivered this week -TODO: Add content here! \ No newline at end of file ++ Continued work on fuzzers. + - this took a lot of time + a lot of debugging. ++ Submitted a talk for KubeCon. ++ Create a separate PR for round trip test error messages: https://github.com/kubernetes/kubernetes/pull/51204. ++ clayton pointed out that disabling protobuf is a breaking API change. ++ Discussed an idea/solution to implement proto for interface fields and got it working. + - essentially, proto is only for external types so we want to avoid interface fields in the external types. + - to do so, we can have a struct with a `Raw` field - basically, a stripped `runtime.Unknown`. + - Use this JSON blob and convert to the internal types (which is still an interface) + - added custom conversion functions + marshal functions + tests ++ Now the validation PR is ready! :rocket: \ No newline at end of file diff --git a/weekly/week8.md b/weekly/week8.md index a084211..ad4e352 100644 --- a/weekly/week8.md +++ b/weekly/week8.md @@ -2,7 +2,18 @@ ## Delivered this week -TODO: Add content here! ++ Fixed an error in `local-up-cluster.sh`. https://github.com/kubernetes/kubernetes/pull/48076 ++ The example in TPR client-go was outdated for release 1.7. Fixed it: https://github.com/kubernetes/kubernetes/pull/47748. ++ Created a cherry pick to port this to release-1.7 branch. + - came across a few minor errors in the cherry pic doc. Fixed them: https://github.com/kubernetes/community/pull/766. + - the cherry pick PR: https://github.com/kubernetes/kubernetes/pull/48114. + - note: for cherry-picking specific PRs, use the indivisual cherry-pick script. ++ Decided to use `GetURL` to go through a JSON string for the conversion to avoid accessing the internal fields. ++ The defaulting for go-openapi is on track. ++ The godoc for apiextensions was not generated because: + - a typo: https://github.com/kubernetes/kubernetes/commit/9ee991837c2ba22becee1fb324deb81de3327903 + - the bot had not synced since the last 5 days. ++ Addressed reviews on the proposal. The proposal now looks good to get merged! :tada: diff --git a/weekly/week9.md b/weekly/week9.md index f77526b..07c50c8 100644 --- a/weekly/week9.md +++ b/weekly/week9.md @@ -2,4 +2,39 @@ ## Delivered this week -TODO: Add content here! ++ For the proposal: only open question is about the order of validations. We would probably overengineer this if we define the order via a slice. This could be part of the webhook validation type. Though even multiple webhooks is probably too much. ++ Created a temp PR for conversion of JSONSchemaRef: https://github.com/nikhita/jsonreference/pull/1. Edit: decided not to use this, but the constructor which is already available upstream. ++ We probably don't need the 5 bools in `JSONSchemaRef` The parsing code only checks that it is a URL. everything after that is unconditionally, i.e. without added value for validation of the `JSONSchemaRef` value. ++ The [update-on-CRD case](https://github.com/kubernetes/kubernetes/pull/47263#discussion_r125323976) did not work because of no informers! :/ Invalidated the storage cache. ++ Also, added a check that the url created by `NewRef` is a valid url. ++ Discussed the non-trivial cases of `OneOf` + - one can construct json objects where the default values in a schema do not make sense. + - this `{allOf: [{minItems: 1, items:{default:7}}, {items:[{default:8}]}]}` will give either `[7]` or `[8]`, both are valid. ++ For default: + - default has no direct impact on validation. so it does not need to satisfy the schema. + - most validators dont take defaulting into account for this very reason. + - main insight: if defaulting does not influence validation, the defaulting of slices (in the sense of appending missing elements) cannot happen. :sparkle: ++ Updated the Ref field to a string. When I do that, it validates _every time_. Still figuring out why changing one field is affecting the whole validation to pass. ++ Updated the gengo PR! + - discussed with lavalamp and got the PR merged! :boom: + - simplified and refractored the existing code as well. the `buildTreeFromTypes` func is now very simple! ++ Plan to add Custom resources spec dynamically to main OpenAPI spec using the aggregator. (https://github.com/kubernetes/kubernetes/pull/46734) ++ Worked on updating the vendored gengo in k/k. And had trouble with godep...again! + - kept hitting this error: https://github.com/kubernetes/kubernetes/issues/35356#issuecomment-258054506, https://github.com/kubernetes/kubernetes/pull/42967#issuecomment-288593136. + - kept breaking on the bazel update. + + after godep-save, this lists as an untracked file: `vendor/github.com/docker/docker/project/`. + + then bazel panics in the next step. + - turns out `CONTRIBUTORS.md` from `docker/docker` has a symlink which keeps breaking bazel. https://github.com/moby/moby/issues/31825. + - adding a simple `rm` in godep-save with a good comment that it's a workaround would be good. (edit: done) + - The main problem was why godep was pulling down the `CONTRIBUTORS.md` file, there should no dependecies on this! + - Solution: + + godep pulls it down because it a "legal" file. https://github.com/tools/godep/pull/522. + + Related: https://github.com/moby/moby/pull/31847. + - continuing on the vendoring, get this huge change now: https://github.com/nikhita/kubernetes/pull/5. apimachinery is changed! :scream: + - Note: why `godep-save.sh` takes such a long time: + + The problem is that godep accesses k8s.io for every kube package (package, not repo!) + + k8s.io returns the meta info for "go get", i.e. where to find k8s.io/kubernetes on github there is no caching for that. + + in CI everything runs very near to the http://k8s.io server. So roundtrip is much much faster in CI => godep restore takes only 2 minutes or so in CI . + - The apimachinery problem is solved by making sure it does not exist in GOPATH before. Now there is a PR to make sure this does not happen anymore! :smile: + - After some more trouble with gengo (starting with a clean gopath + `make clean`), had the gengo PR out! https://github.com/kubernetes/kubernetes/pull/48630. ++ To top of everything, deepcopy generation broke. https://github.com/kubernetes/kubernetes/issues/48675.