Create YAML or JSON Clusterspec without creating the cluster #2954

chrislovecnm · 2017-07-15T20:08:09Z

Allowing a user to create a YAML or JSON cluster or instance group without creating the object. Some of the new methods will be used to fix the problems we are having with JSON output as well. Reading an array of JSON objects is not yet supported by kops create -f, but a single JSON object is supported.

This implements

kops create cluster --dry-run -oyaml

TODOs

init or create
update cobra docs
figure out what is going on with unit tests
figure out was issue(s) this closes

chrislovecnm · 2017-07-15T20:10:52Z

/assign @andrewsykim @sethpollack @geojaz @justinsb

We only need one, unless you are working on getting more PR reviews under your belt :)

k8s-ci-robot · 2017-07-15T20:10:53Z

@chrislovecnm: GitHub didn't allow me to assign the following users: sethpollack.

Note that only kubernetes members can be assigned.

In response to this:

/assign @andrewsykim @sethpollack @geojaz @justinsb

We only need one, unless you are working on getting more PR reviews under your belt :)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

JustEra · 2017-07-15T20:27:08Z

AsfFor me and said on slack :

init would make more sense as it define a state that it should not "run" or "create" the cluster yet but only define the structure.

create would imply to me that the command really create it and that the cluster is more than a struct definition in a yaml.

I think it would confuse more people to have it in the create or at least we have to say like a dry-create

Would vote for the init even if its an alias of create subfunction.

chrislovecnm · 2017-07-15T20:39:22Z

In terms of naming this on slack, we had two that liked init and one for create. kops create -o yaml is much easier to code, since I am pretty much reusing create.

chrislovecnm · 2017-07-15T20:41:50Z

cmd/kops/create_cluster.go

+	switch targetName {
+	case cloudup.TargetYAML:
+
+		var clusters []*api.Cluster


Can I inline the creation of this var? Could not figure out the syntax ...

chrislovecnm · 2017-07-15T20:46:39Z

Marking WIP, as I need to refactor a bit, and figure out if we want to have init or create.

justinsb · 2017-07-15T22:06:31Z

kubectl uses kubectl create cluster, and it uses --dryrun -oyaml for the functionality I think you're targeting about now.

--target is not really the same concept (it is the final model, rather than the input models), so I'm not sure they should be conflated.

Still not sure what we're going to do in kops server mode, where k8s doesn't really have a user-driven "apply changes" step (our kops update)

chrislovecnm · 2017-07-15T22:39:40Z

So kubectl does have a dry run, but is this a dry run? Dry run to me means, execute something that will do something, and test that it will do the correct actions, without doing anything. That is pretty much what helm dry run does, but you have to have a manifest first.

This is something that helm, kubectl, etc do not really have ...

Maybe init ... pondering

justinsb · 2017-07-16T01:31:02Z

Let's hold off on non-critical UX changes until we are building the UX for kops-server; I don't want to paint ourselves into a corner.

hubt · 2017-07-16T14:03:56Z

I like init. I agree with @chrislovecnm on dry run being different.

chrislovecnm · 2017-07-16T17:30:55Z

@justinsb this is the PR that I meant to comment on. This appears to be orthogonal to the kops-server. This is a utility that anyone would use to create yaml files to interact with the server.

This is something that kubectl, helm, frankly any component in k8s, that I know of, does not have.

Please exaplain how this would impact the kops-server.

chrislovecnm · 2017-10-31T23:56:20Z

@justinsb PTAL - this was asked for on office hours

justinsb · 2017-11-04T15:39:48Z

cmd/kops/create_cluster.go

 }

+var targetOptions = sets.NewString(cloudup.TargetDryRun, cloudup.TargetCloudformation, cloudup.TargetDirect, cloudup.TargetTerraform)


I don't think dryrun should be an explicit target here in the CLI.

correct ... will remove

justinsb · 2017-11-04T15:42:41Z

cmd/kops/create_cluster.go

+		case OutputYaml:
+
+			var clusters []*api.Cluster
+			cluster.ObjectMeta.CreationTimestamp = v1.NewTime(time.Now())


It looks like kubectl create secret generic dummy --dry-run -oyaml does not set CreationTimestamp.

I can try w/o, but I do not think kops create -f mycluster.yaml works w/o it.

justinsb · 2017-11-04T15:43:23Z

cmd/kops/create_cluster.go

+		switch c.Output {
+		case OutputYaml:
+
+			var clusters []*api.Cluster


Hoist the common code out of the switch I think

justinsb · 2017-11-04T15:45:38Z

cmd/kops/create_cluster.go

@@ -1008,6 +1065,7 @@ func RunCreateCluster(f *util.Factory, out io.Writer, c *CreateClusterOptions) e
 		}
 	}

+	// Can we acutally get to this if??


Do you mean that targetName is always != "" ?

Is targetName ever == "". I think it is always set, and the if statement is not needed

I think you're right, but separate PR probably

Can you pull this comment out of this PR?

justinsb · 2017-11-04T15:48:22Z

cmd/kops/get_cluster.go

@@ -215,6 +214,100 @@ func clusterOutputJson(clusters []*api.Cluster, out io.Writer) error {
 	return nil
 }

+// TODO refactor all output JSON methods to use this


Can we create two helpers:

func marshalMultipleToYaml(w io.Writer, obj runtime.Object...) error {

func marshalMultipleToJson(w io.Writer, obj runtime.Object...) error {

Should make this simpler

justinsb

Generally looks good, just a few small fixes

justinsb · 2017-11-04T20:04:42Z

cmd/kops/create_cluster.go

 }

+var targetOptions = sets.NewString(cloudup.TargetCloudformation, cloudup.TargetDirect, cloudup.TargetTerraform)
+var allTargetOptions = strings.Join(targetOptions.List(), ",")


targetOptions is unused? Can we put direct first, as it is the default?

I will remove the set then ...

justinsb · 2017-11-04T20:10:00Z

cmd/kops/create_cluster.go

@@ -987,6 +1013,39 @@ func RunCreateCluster(f *util.Factory, out io.Writer, c *CreateClusterOptions) e
 		return err
 	}

+	if c.DryRun {
+		var obj []runtime.Object
+		// if we do not set the ts we get creationTimestamp: null


That's what kubectl does, so can you add a comment on the consequence of creationTimestamp null

kubectl sets null, I think we should probably be doing the same

justinsb · 2017-11-04T20:10:55Z

cmd/kops/create_cluster.go

@@ -1008,6 +1065,7 @@ func RunCreateCluster(f *util.Factory, out io.Writer, c *CreateClusterOptions) e
 		}
 	}

+	// Can we acutally get to this if??


I think you're right, but separate PR probably

justinsb · 2017-11-04T20:11:19Z

cmd/kops/create_ig.go

@@ -39,6 +40,11 @@ import (
 type CreateInstanceGroupOptions struct {
 	Role    string
 	Subnets []string
+	// DryRun we output data


Comment is a little obscure

justinsb · 2017-11-04T20:13:42Z

cmd/kops/get.go

+			obj = append(obj, group)
+		}
+		if err := fullOutputJSON(out, obj...); err != nil {
+			return fmt.Errorf("error writing cluster yaml to stdout: %v", err)


justinsb · 2017-11-04T20:13:51Z

cmd/kops/get.go

-				return err
-			}*/
+
+		var obj []runtime.Object


Hoist out of the loop please

justinsb · 2017-11-04T20:17:52Z

cmd/kops/get_cluster.go


 	default:
-		return fmt.Errorf("Unknown output format: %q", options.output)
+		return fmt.Errorf("unknown output format: %q", options.output)


So the reason for these being upper case is because they look better when displayed, and this is a CLI command object. It's the same way the go rules conflict with all our comments in apimachinery types.

I guess we should format the error forcing the first character to upper-case? But in another PR. Please no more case-switches in this PR though.

I will back them out, they bug me

justinsb · 2017-11-04T20:18:53Z

cmd/kops/get_cluster.go

+func fullOutputJSON(out io.Writer, args ...runtime.Object) error {
+	argsLen := len(args)
+
+	if argsLen > 1 {


I think we should always write the array, even if it is empty. I don't think an empty string is valid JSON?

JSON object is not an array. For instance when we create an ig, we just get a single object

We can always write an array, but at least with a single JSON object create -f works, with a JSON array it bombs

We discussed this; you are right - better to only write the array if there are multiple members. This way a single object is an object

justinsb · 2017-11-04T20:19:15Z

cmd/kops/get_cluster.go

 	return nil
 }

-func clusterOutputYAML(clusters []*api.Cluster, out io.Writer) error {
-	for i, cluster := range clusters {
+// fullOutputJson outputs the marshalled YAML of a list of clusters and instance groups.  It will handle


justinsb · 2017-11-04T22:43:04Z

Looks good, we just need to figure out if creationTimestamp should be set... I think not?

Added --dry-run for create_ig and create_cluster

justinsb · 2017-11-04T23:08:56Z

/lgtm

k8s-github-robot · 2017-11-04T23:09:39Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justinsb

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~OWNERS~~ [justinsb]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-11-04T23:37:55Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2017-11-05T00:05:04Z

Automatic merge from submit-queue.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 15, 2017

k8s-ci-robot assigned andrewsykim, geojaz and justinsb Jul 15, 2017

chrislovecnm force-pushed the create-yaml-or-json branch from b625dc5 to 504d9a4 Compare July 15, 2017 20:37

chrislovecnm force-pushed the create-yaml-or-json branch from 504d9a4 to a486d28 Compare July 15, 2017 20:41

chrislovecnm commented Jul 15, 2017

View reviewed changes

chrislovecnm added the WIP: Work In Progress label Jul 15, 2017

chrislovecnm unassigned justinsb, geojaz and andrewsykim Jul 15, 2017

chrislovecnm force-pushed the create-yaml-or-json branch from a486d28 to dd96bed Compare July 15, 2017 21:08

justinsb added this to the kops-server milestone Jul 16, 2017

blakebarnett mentioned this pull request Jul 29, 2017

Configuration file instead of command line switches #267

Closed

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 1, 2017

k8s-github-robot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 13, 2017

justinsb modified the milestones: kops-server, backlog Sep 23, 2017

justinsb added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 4, 2017

justinsb removed the WIP: Work In Progress label Nov 4, 2017

chrislovecnm force-pushed the create-yaml-or-json branch from dd96bed to 285c7e8 Compare November 4, 2017 15:33

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 4, 2017

chrislovecnm removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 4, 2017

justinsb reviewed Nov 4, 2017

View reviewed changes

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 4, 2017

chrislovecnm force-pushed the create-yaml-or-json branch from 8133d21 to 13fc997 Compare November 4, 2017 20:01

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 4, 2017

chrislovecnm modified the milestones: backlog, 1.8.0 Nov 4, 2017

chrislovecnm force-pushed the create-yaml-or-json branch 3 times, most recently from 4185ca6 to 3ba6bfc Compare November 4, 2017 20:11

justinsb reviewed Nov 4, 2017

View reviewed changes

chrislovecnm force-pushed the create-yaml-or-json branch 2 times, most recently from 937d32f to 66e6f95 Compare November 4, 2017 22:36

chrislovecnm force-pushed the create-yaml-or-json branch from 66e6f95 to d7af014 Compare November 4, 2017 23:01

Fixing YAML and JSON output across multiple cli functions. Also

4c82a6d

Added --dry-run for create_ig and create_cluster

chrislovecnm force-pushed the create-yaml-or-json branch from d7af014 to 4c82a6d Compare November 4, 2017 23:04

k8s-ci-robot assigned justinsb Nov 4, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 4, 2017

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 4, 2017

k8s-github-robot merged commit 40a94a7 into kubernetes:master Nov 5, 2017

		}

		var targetOptions = sets.NewString(cloudup.TargetDryRun, cloudup.TargetCloudformation, cloudup.TargetDirect, cloudup.TargetTerraform)

Create YAML or JSON Clusterspec without creating the cluster #2954

Create YAML or JSON Clusterspec without creating the cluster #2954

Conversation

chrislovecnm commented Jul 15, 2017 • edited Loading

chrislovecnm commented Jul 15, 2017

k8s-ci-robot commented Jul 15, 2017

JustEra commented Jul 15, 2017

chrislovecnm commented Jul 15, 2017

Choose a reason for hiding this comment

chrislovecnm commented Jul 15, 2017

justinsb commented Jul 15, 2017

chrislovecnm commented Jul 15, 2017 • edited Loading

justinsb commented Jul 16, 2017

hubt commented Jul 16, 2017

chrislovecnm commented Jul 16, 2017

chrislovecnm commented Oct 31, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justinsb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justinsb commented Nov 4, 2017

justinsb commented Nov 4, 2017

k8s-github-robot commented Nov 4, 2017

k8s-github-robot commented Nov 4, 2017

k8s-github-robot commented Nov 5, 2017

chrislovecnm commented Jul 15, 2017 •

edited

Loading

chrislovecnm commented Jul 15, 2017 •

edited

Loading