Create fuzz testing for kubectl apply

[bgrant0607]

We discover bugs in kubectl apply on a field-by-field basis: #26229, #23551.

We don't have rigorous testing of the diff and patch code, nor of the API spec itself. There are many known problems, such as lack of ability to distinguish unset fields from Go default values for non-pointer primitive fields.

We should develop a random fuzz tester for kubectl apply.

cc @ghodss @kubernetes/kubectl

[dshulyak]

I would like to help with this bug.

[adohe-zz]

@dshulyak that would be great. Thanks :)

[dshulyak]

@adohe @bgrant0607 Am i right that the goal is to validate k8s objects using schema after applying ThreeWayMergePatch?
If so fuzz tester should be able to:

1. Generate k8s object (using existing helper)
2. Generate 2nd version of this object
3. Create and apply three way merge patch
4. Validate resulted object using schema, plus additional tests

[bgrant0607]

@dshulyak I think the main challenge is how to determine correctness of the result. That needs more thought before this will be worth doing.

[0xmichalis]

/sig cli

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[bgrant0607]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[bgrant0607]

cc @apelisse

[apelisse]

We don't have rigorous testing of the diff and patch code, nor of the API spec itself.
...
We should develop a random fuzz tester for kubectl apply.

I don't necessarily agree that the first statement should lead to the second. I'll keep in mind that we want solid unit-testing for diff/patch. I think "determining correctness" would indeed be an unnecessary challenge.

[sftim]

We have server-side apply now. Is this issue description still right?

[apelisse]

It's hard to tell for sure. We haven't really created fuzzing, I think we have a much stronger test suite now. We're also not bug-free (obviously), but I think this issue is somewhat irrelevant, let's close it.
/close

[k8s-ci-robot]

@apelisse: Closing this issue.

In response to this:

It's hard to tell for sure. We haven't really created fuzzing, I think we have a much stronger test suite now. We're also not bug-free (obviously), but I think this issue is somewhat irrelevant, let's close it.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.