Eliminate round-trip conversion of API objects in kubectl convert

[yujuhong]

kubectl decodes a json file and encodes it again before sending out the request to the API server, i.e.,
versioned json-> versioned api object -> internal api object -> versioned api object -> versioned json
The API server then decodes the versioned json and converts it to the internal API object.

It is debatable whether kubectl needs to perform the round-trip conversion, since the API server would perform the similar process immediately after receiving the request. Can we simplify this?

@bgrant0607 @smarterclayton

[smarterclayton]

Right now, the mutation we are doing on create/update is to set the namespace of the object, and during update we set the resource version. Ideally, we'd be able to do this opaquely, and with less conversions (perhaps by having a runtime.Object that supported deferred decoding by working a bit like runtime.Unknown). Right now we don't assume that it's safe to mutate the JSON directly because we'd have to write code for each object type that says what sort of mutation we need for metadata (we can read it from the scheme via the name conversion rules, but it's a bit tricky because we don't define what fields are metadata). The assumption that an apiversion can have rules defined on it is invalid (because eventually core resources will split and migrate at different times, so they may have different rules).

Once we nuke v1beta1/2, it would be a good time to go through the code and simplify a bit to say that there is a fast path for certain known object types (core resources) which can bypass direct conversion and do mutation directly on the JSON, and the rest can use the slower pass for mutation.

----- Original Message -----

kubectl decodes a json file and encodes it again before sending out the
request to the API server, i.e.,
versioned json-> versioned api object -> internal api object -> versioned api object -> versioned json
The API server then decodes the versioned json and converts it to the
internal API object.

It is debatable whether kubectl needs to perform the round-trip conversion,
since the API server would perform the similar process immediately after
receiving the request. Can we simplify this?

@bgrant0607 @smarterclayton

---

Reply to this email directly or view it on GitHub:
#3955

[bgrant0607]

Some observations:

• So long as defaults are set by kubectl, we'll need to capture user intent first in order to facilitate merging an updated configuration with the live desired state.
• Obvious but worth stating: apiserver can't rely on kubectl to apply defaults.
  1. The API has other clients.
  2. We need to retain the ability to add new fields without changing the api version.

[thockin]

Yeah, its dubious that clients should even be able to see internal structs
and conversion. The obvious exception is validation, with the caveat that
it could drift.

If validation is applied to versioned structs, then clients can stay
versioned. To get validation we will need per-version defaulting.
Luckily....
On Jan 30, 2015 7:39 PM, "Brian Grant" notifications@github.com wrote:

Some observations:

• So long as defaults are set by kubectl, we'll need to capture user
  intent first in order to facilitate merging an updated configuration with
  the live desired state.
• Obvious but worth stating: apiserver can't rely on kubectl to apply
  defaults.
  1. The API has other clients.
  2. We need to retain the ability to add new fields without changing
     the api version.

Reply to this email directly or view it on GitHub
#3955 (comment)
.

[bgrant0607]

We should use (potentially cached) swagger for basic (optional) client-side validation. That approach will work with newer releases, new versions, and new objects (plugins, etc.).

Detailed validation that can't be expressed via swagger can be performed by the server.

[derekwaynecarr]

Some of the work I did with RESTMapper last week could be used in the future where we introspect swagger and build a RESTMapper from it on the client.

Sent from my iPhone

On Jan 31, 2015, at 1:38 AM, Brian Grant notifications@github.com wrote:

We should use (potentially cached) swagger for basic (optional) client-side validation. That approach will work with newer releases, new versions, and new objects (plugins, etc.).

Detailed validation that can't be expressed via swagger can be performed by the server.

—
Reply to this email directly or view it on GitHub.

[yujuhong]

Agreed that clients should not have to perform the conversion. What's the main purpose of client-side validation?

[thockin]

Client-side validation can be lower-latency.

On Tue, Feb 3, 2015 at 11:58 AM, Yu-Ju Hong notifications@github.com
wrote:

Agreed that clients should not have to perform the conversion. What's the
main purpose of client-side validation?

Reply to this email directly or view it on GitHub
#3955 (comment)
.

[smarterclayton]

Clients can be about validation, which increases user frustration (go to web UI, set something, go to client, can't set it, hate product forever). I would argue client validation is always optional and as much as possible comes from source of truth (schema exposed a la swagger, which is a reasonable compromise).

----- Original Message -----

Client-side validation can be lower-latency.

On Tue, Feb 3, 2015 at 11:58 AM, Yu-Ju Hong notifications@github.com
wrote:

Agreed that clients should not have to perform the conversion. What's the
main purpose of client-side validation?

Reply to this email directly or view it on GitHub
#3955 (comment)
.

---

Reply to this email directly or view it on GitHub:
#3955 (comment)

[smarterclayton]

"can be very wrong"

----- Original Message -----

Clients can be about validation, which increases user frustration (go to web
UI, set something, go to client, can't set it, hate product forever). I
would argue client validation is always optional and as much as possible
comes from source of truth (schema exposed a la swagger, which is a
reasonable compromise).

----- Original Message -----

Client-side validation can be lower-latency.

On Tue, Feb 3, 2015 at 11:58 AM, Yu-Ju Hong notifications@github.com
wrote:

Agreed that clients should not have to perform the conversion. What's the
main purpose of client-side validation?

Reply to this email directly or view it on GitHub
#3955 (comment)
.

---

Reply to this email directly or view it on GitHub:
#3955 (comment)

[bgrant0607]

Agreed that we want the source of truth to come from the server.

[bgrant0607]

Found that the test for template-based printing in kubectl (resource_printer_test.go) failed when I removed the json tags on api/types.go for #3933.

[bgrant0607]

cc @vishh

[thockin]

w00t! that's huge.

On Thu, Aug 25, 2016 at 8:52 AM, Clayton Coleman notifications@github.com
wrote:

Phenomenal

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#3955 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVNxqTOM2P94CECuae7ERDH7ZuU11ks5qjbo2gaJpZM4DZOOC
.

[lavalamp]

@krousey is there more to do or can we close this issue?

[krousey]

@lavalamp Because of the way we do patching (relies on data being of the same version and also needs versioned struct tags for merge strategies), anything that computes a strategic merge patch still does client side conversion. I may have missed a few, but that's apply, edit, label, and annotate. I think label and annotate don't have any danger of introducing fields that have defaulting. apply and edit could potentially introduce new fields to an object that has some defaulted internals. It is possible, but I think most fields are defaulted before clients that apply or edit see the object.

We'd have to fix how we compute patches client-side (and/or handle them server-side) to change that. I don't know if we want to attempt that, and if we do, whether this is the place to track that effort.

[smarterclayton]

For operations not on the server we can either implement one patch calculation for each API version kubectl supports, or split defaulting into its own pass and allow conversion without defaulting (we are going to have to do that anyway for other reasons). The latter could still introduce patch artifacts. The "set" commands are going to end up having a lot of internal code that needs patch. I would regret the cut and paste, but one function per version isn't the end of the world.

[lavalamp]

Kicking this out of the milestone since we're not planning on doing more at the moment.

[smarterclayton]

This was mostly completed in 1.6. Some specific commands still use internal version.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[bgrant0607]

/remove-lifecycle stale
/lifecycle frozen

[krousey]

reassigning to @lavalamp as I haven't actually worked on this in a while.

[liggitt]

/unassign @lavalamp
/assign @seans3

I think kubectl convert is the last place this is being done

[thockin]

Is this still a real issue?

[liggitt]

This functionality got split out to kubectl-convert so kubectl no longer has a dependency on internal types. I think that's the final destination and this can be closed