Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eliminate round-trip conversion of API objects in kubectl #3955

Open
yujuhong opened this issue Jan 29, 2015 · 31 comments

Comments

@yujuhong
Copy link
Member

commented Jan 29, 2015

kubectl decodes a json file and encodes it again before sending out the request to the API server, i.e.,
versioned json-> versioned api object -> internal api object -> versioned api object -> versioned json
The API server then decodes the versioned json and converts it to the internal API object.

It is debatable whether kubectl needs to perform the round-trip conversion, since the API server would perform the similar process immediately after receiving the request. Can we simplify this?

@bgrant0607 @smarterclayton

@smarterclayton

This comment has been minimized.

Copy link
Contributor

commented Jan 29, 2015

Right now, the mutation we are doing on create/update is to set the namespace of the object, and during update we set the resource version. Ideally, we'd be able to do this opaquely, and with less conversions (perhaps by having a runtime.Object that supported deferred decoding by working a bit like runtime.Unknown). Right now we don't assume that it's safe to mutate the JSON directly because we'd have to write code for each object type that says what sort of mutation we need for metadata (we can read it from the scheme via the name conversion rules, but it's a bit tricky because we don't define what fields are metadata). The assumption that an apiversion can have rules defined on it is invalid (because eventually core resources will split and migrate at different times, so they may have different rules).

Once we nuke v1beta1/2, it would be a good time to go through the code and simplify a bit to say that there is a fast path for certain known object types (core resources) which can bypass direct conversion and do mutation directly on the JSON, and the rest can use the slower pass for mutation.

----- Original Message -----

kubectl decodes a json file and encodes it again before sending out the
request to the API server, i.e.,
versioned json-> versioned api object -> internal api object -> versioned api object -> versioned json
The API server then decodes the versioned json and converts it to the
internal API object.

It is debatable whether kubectl needs to perform the round-trip conversion,
since the API server would perform the similar process immediately after
receiving the request. Can we simplify this?

@bgrant0607 @smarterclayton


Reply to this email directly or view it on GitHub:
#3955

@bgrant0607

This comment has been minimized.

Copy link
Member

commented Jan 31, 2015

Some observations:

  • So long as defaults are set by kubectl, we'll need to capture user intent first in order to facilitate merging an updated configuration with the live desired state.
  • Obvious but worth stating: apiserver can't rely on kubectl to apply defaults.
    1. The API has other clients.
    2. We need to retain the ability to add new fields without changing the api version.
@thockin

This comment has been minimized.

Copy link
Member

commented Jan 31, 2015

Yeah, its dubious that clients should even be able to see internal structs
and conversion. The obvious exception is validation, with the caveat that
it could drift.

If validation is applied to versioned structs, then clients can stay
versioned. To get validation we will need per-version defaulting.
Luckily....
On Jan 30, 2015 7:39 PM, "Brian Grant" notifications@github.com wrote:

Some observations:

  • So long as defaults are set by kubectl, we'll need to capture user
    intent first in order to facilitate merging an updated configuration with
    the live desired state.
  • Obvious but worth stating: apiserver can't rely on kubectl to apply
    defaults.
    1. The API has other clients.
    2. We need to retain the ability to add new fields without changing
      the api version.

Reply to this email directly or view it on GitHub
#3955 (comment)
.

@bgrant0607

This comment has been minimized.

Copy link
Member

commented Jan 31, 2015

We should use (potentially cached) swagger for basic (optional) client-side validation. That approach will work with newer releases, new versions, and new objects (plugins, etc.).

Detailed validation that can't be expressed via swagger can be performed by the server.

@derekwaynecarr

This comment has been minimized.

Copy link
Member

commented Jan 31, 2015

Some of the work I did with RESTMapper last week could be used in the future where we introspect swagger and build a RESTMapper from it on the client.

Sent from my iPhone

On Jan 31, 2015, at 1:38 AM, Brian Grant notifications@github.com wrote:

We should use (potentially cached) swagger for basic (optional) client-side validation. That approach will work with newer releases, new versions, and new objects (plugins, etc.).

Detailed validation that can't be expressed via swagger can be performed by the server.


Reply to this email directly or view it on GitHub.

@yujuhong

This comment has been minimized.

Copy link
Member Author

commented Feb 3, 2015

Agreed that clients should not have to perform the conversion. What's the main purpose of client-side validation?

@thockin

This comment has been minimized.

Copy link
Member

commented Feb 3, 2015

Client-side validation can be lower-latency.

On Tue, Feb 3, 2015 at 11:58 AM, Yu-Ju Hong notifications@github.com
wrote:

Agreed that clients should not have to perform the conversion. What's the
main purpose of client-side validation?

Reply to this email directly or view it on GitHub
#3955 (comment)
.

@smarterclayton

This comment has been minimized.

Copy link
Contributor

commented Feb 3, 2015

Clients can be about validation, which increases user frustration (go to web UI, set something, go to client, can't set it, hate product forever). I would argue client validation is always optional and as much as possible comes from source of truth (schema exposed a la swagger, which is a reasonable compromise).

----- Original Message -----

Client-side validation can be lower-latency.

On Tue, Feb 3, 2015 at 11:58 AM, Yu-Ju Hong notifications@github.com
wrote:

Agreed that clients should not have to perform the conversion. What's the
main purpose of client-side validation?

Reply to this email directly or view it on GitHub
#3955 (comment)
.


Reply to this email directly or view it on GitHub:
#3955 (comment)

@smarterclayton

This comment has been minimized.

Copy link
Contributor

commented Feb 3, 2015

"can be very wrong"

----- Original Message -----

Clients can be about validation, which increases user frustration (go to web
UI, set something, go to client, can't set it, hate product forever). I
would argue client validation is always optional and as much as possible
comes from source of truth (schema exposed a la swagger, which is a
reasonable compromise).

----- Original Message -----

Client-side validation can be lower-latency.

On Tue, Feb 3, 2015 at 11:58 AM, Yu-Ju Hong notifications@github.com
wrote:

Agreed that clients should not have to perform the conversion. What's the
main purpose of client-side validation?

Reply to this email directly or view it on GitHub
#3955 (comment)
.


Reply to this email directly or view it on GitHub:
#3955 (comment)

@bgrant0607

This comment has been minimized.

Copy link
Member

commented Feb 4, 2015

Agreed that we want the source of truth to come from the server.

@bgrant0607 bgrant0607 changed the title Assess whether round-trip conversion of API objects is necessary in kubectl Eliminate round-trip conversion of API objects in kubectl Feb 4, 2015
@bgrant0607

This comment has been minimized.

Copy link
Member

commented Mar 19, 2015

Found that the test for template-based printing in kubectl (resource_printer_test.go) failed when I removed the json tags on api/types.go for #3933.

@bgrant0607

This comment has been minimized.

Copy link
Member

commented Apr 2, 2015

cc @vishh

juanvallejo pushed a commit to juanvallejo/kubernetes that referenced this issue Aug 17, 2016
Automatic merge from submit-queue

Change kubectl create to use dynamic client

kubernetes#16764 kubernetes#3955

This is a series of changes to allow kubectl create to use discovery-based REST mapping and dynamic clients.

cc @kubernetes/sig-api-machinery

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
kubectl will no longer do client-side defaulting on create and replace.
```
@krousey

This comment has been minimized.

Copy link
Member

commented Aug 25, 2016

With my most recent change, kubectl replace and create no longer do round trip conversions on the client side. I believe that covers most of the bugs we had with client-side conversion/defaulting.

@smarterclayton

This comment has been minimized.

Copy link
Contributor

commented Aug 25, 2016

@thockin

This comment has been minimized.

Copy link
Member

commented Aug 25, 2016

w00t! that's huge.

On Thu, Aug 25, 2016 at 8:52 AM, Clayton Coleman notifications@github.com
wrote:

Phenomenal


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#3955 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVNxqTOM2P94CECuae7ERDH7ZuU11ks5qjbo2gaJpZM4DZOOC
.

@lavalamp

This comment has been minimized.

Copy link
Member

commented Aug 25, 2016

@krousey is there more to do or can we close this issue?

@krousey

This comment has been minimized.

Copy link
Member

commented Aug 25, 2016

@lavalamp Because of the way we do patching (relies on data being of the same version and also needs versioned struct tags for merge strategies), anything that computes a strategic merge patch still does client side conversion. I may have missed a few, but that's apply, edit, label, and annotate. I think label and annotate don't have any danger of introducing fields that have defaulting. apply and edit could potentially introduce new fields to an object that has some defaulted internals. It is possible, but I think most fields are defaulted before clients that apply or edit see the object.

We'd have to fix how we compute patches client-side (and/or handle them server-side) to change that. I don't know if we want to attempt that, and if we do, whether this is the place to track that effort.

@smarterclayton

This comment has been minimized.

Copy link
Contributor

commented Aug 26, 2016

@lavalamp lavalamp removed this from the v1.4 milestone Aug 29, 2016
@lavalamp

This comment has been minimized.

Copy link
Member

commented Aug 29, 2016

Kicking this out of the milestone since we're not planning on doing more at the moment.

@smarterclayton

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2017

This was mostly completed in 1.6. Some specific commands still use internal version.

@fejta-bot

This comment has been minimized.

Copy link

commented Dec 25, 2017

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@bgrant0607

This comment has been minimized.

Copy link
Member

commented Jan 22, 2018

/remove-lifecycle stale
/lifecycle frozen

@krousey

This comment has been minimized.

Copy link
Member

commented Aug 5, 2019

reassigning to @lavalamp as I haven't actually worked on this in a while.

@liggitt

This comment has been minimized.

Copy link
Member

commented Aug 6, 2019

/unassign @lavalamp
/assign @seans3

I think kubectl convert is the last place this is being done

@k8s-ci-robot k8s-ci-robot assigned seans3 and unassigned lavalamp Aug 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.