Use ksonnet to generate examples #9

chuckha · 2017-07-30T21:35:40Z

Closes issue #5.

Adds jsonnet to generate the quickstart examples
Removes the YAML
Adds the generated JSON
Docs are updated to reflect JSON files vs YAML
Adds a Makefile target, generate-examples

Signed-off-by: Chuck Ha chuck@heptio.com

chuckha · 2017-07-30T21:43:48Z

Here is the process I took to ensure the generated json is equivalent to yaml:

Generate json with new make command
Convert generated json to yaml (https://github.com/redsymbol/json2yaml)
YAML diff (https://www.npmjs.com/package/koimo)

cha$ node_modules/.bin/koimo gen_00-rbac.yaml 00-rbac.yaml 
  
  ===================
   Koimo - YAML Diff 
  ===================
  
  [ gen_00-rbac.yaml ] vs. [ 00-rbac.yaml ]

  ✓ All good!

cha$ node_modules/.bin/koimo gen_10-configmaps.yaml 10-configmaps.yaml 
  
  ===================
   Koimo - YAML Diff 
  ===================
  
  [ gen_10-configmaps.yaml ] vs. [ 10-configmaps.yaml ]

  ✓ All good!

cha$ node_modules/.bin/koimo gen_20-pod.yaml 20-pod.yaml 
  
  ===================
   Koimo - YAML Diff 
  ===================
  
  [ gen_20-pod.yaml ] vs. [ 20-pod.yaml ]

  ✓ All good!

chuckha · 2017-07-31T14:10:41Z

I would love input on what to do with the YAML. I'm not convinced deleting it is the best path forward because in my experience people would prefer reading YAML. If they are looking for a way to get started, it would be nice to have the YAML available.

kensimon

Small fixes, plus maybe want @hausdorff to review the jsonnet (I'm a little out of my expertise here)

kensimon · 2017-07-31T22:41:51Z

Makefile

+	$(MAKE) -C build/systemd-logs clean
+	rm -f /examples/quickstart/*.json


Probably don't want to rm -r /examples, but ./examples?

kensimon · 2017-07-31T22:50:54Z

docs/building-examples.md

+
+The JSON files are generated with the `make generate-examples` command. To get this to work you must have ksonnet-lib installed in `/usr/local/lib/jsonnet/ksonnet-lib`. After that, `make generate-examples` will build the JSON files that are checked into this repository.
+
+0: https://ksonnet.heptio.com


HTTPS doesn't work on this site, link needs fixing

timothysc · 2017-08-01T03:06:21Z

examples/quickstart/00-rbac.json

+         }
+      },
+      {
+         "apiVersion": "rbac.authorization.k8s.io/v1beta1",


Can we somehow option wrap rbac with a default of true but allow for disabled generation per other issue filed?

It's not easy right now, but it's certainly on the roadmap for the CLI.

One way to do it right now is to use ext vars, which are passed in via command line. Your build process would involve interrogating the API server, getting capabilities, and passing a flag --ext-str with a value to indicate as such.

timothysc · 2017-08-01T03:07:33Z

examples/quickstart/10-configmaps.json

+      {
+         "apiVersion": "v1",
+         "data": {
+            "config.json": "{\"Description\": \"EXAMPLE\", \"Filters\": {\"LabelSelector\": \"\", \"Namespaces\": \".*\"}, \"PluginNamespace\": \"heptio-sonobuoy\", \"Plugins\": [{\"name\": \"systemd_logs\"}, {\"name\": \"e2e\"}], \"Resources\": [\"CertificateSigningRequests\", \"ClusterRoleBindings\", \"ClusterRoles\", \"ComponentStatuses\", \"Nodes\", \"PersistentVolumes\", \"PodSecurityPolicies\", \"ServerVersion\", \"StorageClasses\", \"ThirdPartyResources\", \"ConfigMaps\", \"DaemonSets\", \"Deployments\", \"Endpoints\", \"Events\", \"HorizontalPodAutoscalers\", \"Ingresses\", \"Jobs\", \"LimitRanges\", \"PersistentVolumeClaims\", \"Pods\", \"PodLogs\", \"PodDisruptionBudgets\", \"PodPresets\", \"PodTemplates\", \"ReplicaSets\", \"ReplicationControllers\", \"ResourceQuotas\", \"RoleBindings\", \"Roles\", \"ServiceAccounts\", \"Services\", \"StatefulSets\"], \"ResultsDir\": \"/tmp/sonobuoy\", \"Server\": {\"advertiseaddress\": \"sonobuoy-master:8080\", \"bindaddress\": \"0.0.0.0\", \"bindport\": 8080, \"timeoutseconds\": 3600}, \"Version\": \"v0.8.0\"}"


This almost makes it unreadable...

I would use kubecfg here, and generate YAML instead of JSON. kubecfg natively emits multi-line strings all prettified.

hausdorff · 2017-08-01T04:48:50Z

examples/quickstart/00-rbac.json

+         }
+      },
+      {
+         "apiVersion": "rbac.authorization.k8s.io/v1beta1",


It's not easy right now, but it's certainly on the roadmap for the CLI.

One way to do it right now is to use ext vars, which are passed in via command line. Your build process would involve interrogating the API server, getting capabilities, and passing a flag --ext-str with a value to indicate as such.

hausdorff · 2017-08-01T04:50:15Z

examples/quickstart/00-rbac.jsonnet

+
+};
+
+local namespace = local ns = k.core.v1.namespace;


Super tiny micro nit, when jsonnet fmt matures, this local will get indented below. AFAIK that's actively being worked on by Dave's intern.

hausdorff · 2017-08-01T04:51:48Z

examples/quickstart/10-configmaps.json

+      {
+         "apiVersion": "v1",
+         "data": {
+            "config.json": "{\"Description\": \"EXAMPLE\", \"Filters\": {\"LabelSelector\": \"\", \"Namespaces\": \".*\"}, \"PluginNamespace\": \"heptio-sonobuoy\", \"Plugins\": [{\"name\": \"systemd_logs\"}, {\"name\": \"e2e\"}], \"Resources\": [\"CertificateSigningRequests\", \"ClusterRoleBindings\", \"ClusterRoles\", \"ComponentStatuses\", \"Nodes\", \"PersistentVolumes\", \"PodSecurityPolicies\", \"ServerVersion\", \"StorageClasses\", \"ThirdPartyResources\", \"ConfigMaps\", \"DaemonSets\", \"Deployments\", \"Endpoints\", \"Events\", \"HorizontalPodAutoscalers\", \"Ingresses\", \"Jobs\", \"LimitRanges\", \"PersistentVolumeClaims\", \"Pods\", \"PodLogs\", \"PodDisruptionBudgets\", \"PodPresets\", \"PodTemplates\", \"ReplicaSets\", \"ReplicationControllers\", \"ResourceQuotas\", \"RoleBindings\", \"Roles\", \"ServiceAccounts\", \"Services\", \"StatefulSets\"], \"ResultsDir\": \"/tmp/sonobuoy\", \"Server\": {\"advertiseaddress\": \"sonobuoy-master:8080\", \"bindaddress\": \"0.0.0.0\", \"bindport\": 8080, \"timeoutseconds\": 3600}, \"Version\": \"v0.8.0\"}"


I would use kubecfg here, and generate YAML instead of JSON. kubecfg natively emits multi-line strings all prettified.

hausdorff · 2017-08-01T04:58:33Z

examples/quickstart/10-configmaps.jsonnet

+};
+
+local plugins = {
+  "systemdlogs.yaml": 'name: systemd_logs


Hmm, it seems to me you could write this as a normal JSON object and then call std.toString(systemdLogs) on it to get a string? Am I missing something crucial?

Unfortunately the plugin loader only knows how to read YAML.

I'll make an issue and this issue will depend on that issue.

@chuckha Is YAML not a superset of JSON, though? So it seems like we should be able to use JSON here and be ok.

If not, kubecfg allows you to import YAML files, as well manifestYAML, which pretty-prints an object as YAML.

Or perhaps I am missing something?

hausdorff · 2017-08-01T05:02:01Z

examples/quickstart/10-configmaps.jsonnet

+k.core.v1.list.new([
+    sonobuoyConfig,
+    pluginConfigs,
+])


Tiny, tiny microest of nits, you might consider putting newlines at the end so that diffs are cleaner.

timothysc · 2017-08-01T14:22:42Z

So I'm inclined to hold off on this unless it is strictly cleaner and more concise, that is the bar we should be holding here.

hausdorff · 2017-08-01T16:05:14Z

@timothysc Your comment here makes me wonder if there is anything beyond the comments listed you'd like to see changed? Changing to YAML will make it cleaner and more concise, but it's not a change to the Jsonnet code itself, which is where I'd expect the bulk of cleanliness changes to go.

timothysc · 2017-08-01T19:28:36Z

@hausdorff Right now I think the biggest detractor is the readability. I'd also like to get to the point where we can generate the plugin definitions, and perhaps use that as a mixin to the quickstart .yaml.

hausdorff · 2017-08-01T20:22:26Z

@timothysc Strongly agree about the plugin definitions, which I think are doable with std.toString, I think I mentioned this before.

I think my main question, though, is that it's not clear (to me, at least) what else needs to change to meet the readability bar. Just the string pretty printing?

timothysc · 2017-08-01T21:06:22Z

I think my main question, though, is that it's not clear (to me, at least) what else needs to change to meet the readability bar. Just the string pretty printing?

Eliminate the nested structure string escaping.

Closes issue #5. - Adds jsonnet to generate the quickstart examples - Removes the YAML - Adds the generated JSON - Docs are updated to reflect JSON files vs YAML - Adds a Makefile target, `generate-examples` Signed-off-by: Chuck Ha <chuck@heptio.com>

chuckha · 2017-08-17T15:01:30Z

I believe all issues have been addressed in #52

chuckha requested a review from timothysc July 30, 2017 21:35

kensimon suggested changes Jul 31, 2017

View reviewed changes

timothysc reviewed Aug 1, 2017

View reviewed changes

hausdorff reviewed Aug 1, 2017

View reviewed changes

chuckha mentioned this pull request Aug 10, 2017

Supports JSON configs #48

Merged

Chuck Ha added 3 commits August 14, 2017 12:24

Use ksonnet to generate examples

3512926

Closes issue #5. - Adds jsonnet to generate the quickstart examples - Removes the YAML - Adds the generated JSON - Docs are updated to reflect JSON files vs YAML - Adds a Makefile target, `generate-examples` Signed-off-by: Chuck Ha <chuck@heptio.com>

fixup some issues

15df1d9

testing

9f0fa1c

chuckha closed this Aug 17, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use ksonnet to generate examples #9

Use ksonnet to generate examples #9

chuckha commented Jul 30, 2017

chuckha commented Jul 30, 2017

chuckha commented Jul 31, 2017

kensimon left a comment

kensimon Jul 31, 2017

kensimon Jul 31, 2017

timothysc Aug 1, 2017

hausdorff Aug 1, 2017

timothysc Aug 1, 2017

hausdorff Aug 1, 2017

hausdorff Aug 1, 2017

hausdorff Aug 1, 2017

hausdorff Aug 1, 2017

hausdorff Aug 1, 2017

chuckha Aug 6, 2017

hausdorff Aug 7, 2017

hausdorff Aug 1, 2017

timothysc commented Aug 1, 2017

hausdorff commented Aug 1, 2017

timothysc commented Aug 1, 2017

hausdorff commented Aug 1, 2017

timothysc commented Aug 1, 2017

chuckha commented Aug 17, 2017

		$(MAKE) -C build/systemd-logs clean
		rm -f /examples/quickstart/*.json


		The JSON files are generated with the `make generate-examples` command. To get this to work you must have ksonnet-lib installed in `/usr/local/lib/jsonnet/ksonnet-lib`. After that, `make generate-examples` will build the JSON files that are checked into this repository.

		0: https://ksonnet.heptio.com

Use ksonnet to generate examples #9

Use ksonnet to generate examples #9

Conversation

chuckha commented Jul 30, 2017

chuckha commented Jul 30, 2017

chuckha commented Jul 31, 2017

kensimon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothysc commented Aug 1, 2017

hausdorff commented Aug 1, 2017

timothysc commented Aug 1, 2017

hausdorff commented Aug 1, 2017

timothysc commented Aug 1, 2017

chuckha commented Aug 17, 2017