Tnc bootstrap #3053

thorfour · 2018-03-01T00:42:26Z

No description provided.

coreosbot · 2018-03-01T00:42:27Z

Can one of the admins verify this patch?

thorfour · 2018-03-01T00:43:19Z

modules/bootkube-ut2/resources/manifests/tectonic-node-controller.yaml

+        - /bootstrap
+        - --config=/etc/cluster-config
+        - --port=45900
+        - --cert=/assets/tls/ca.crt


@enxebre where can I mount the CA assets from?

ca certs are dropped into https://github.com/coreos/tectonic-installer/blob/master/modules/ignition/ca_certs.tf#L19
Other kube certs a temporary dropped into /opt/tectonic/tls
This probably relates to https://github.com/coreos/tectonic-installer/pull/2972/files

enxebre

@thorfour I rebased this branch onto master and got this to a semi-working state in this commit enxebre@d6ff12c, which addresses the comments here and add/remove some missing parts.
Feel free to cherry pick it. Things TBD:

Add right permissions to the docker image so it can be downloaded at tectonic bootstrap time
Then:
Verify the ignition config provided by the TNC satisfy the nodes. E.g address this https://github.com/coreos-inc/tectonic-operators/pull/286/files#r171519139
Add support for rest like TNC paths https://github.com/thorfour/tectonic-operators/blob/templates/controller/node/pkg/ignition/server.go#L81
For now let's get this working for http. The first node bootstrapped gets the config through a cname pointing to s3 which does not support https (see INST-944), moreover the TNC will need a proper certificate trusted by ignition. Once we address the points above we should approach https

enxebre · 2018-03-02T13:51:07Z

config.tf

@@ -71,6 +71,7 @@ variable "tectonic_container_images" {
    awscli                               = "quay.io/coreos/awscli:025a357f05242fdad6a81e8a6b520098aa65a600"
    gcloudsdk                            = "google/cloud-sdk:178.0.0-alpine"
    bootkube                             = "quay.io/coreos/bootkube:v0.10.0"
+    tnc_bootstrap                        = "quay.io/coreos/tectonic-node-controller-dev:fad3a8e284e2c414fdf1713c7e0ae9d1e1e487ba"


can't download this image with my tectonic license

that's weird since I added your user to have read access on quay

enxebre · 2018-03-02T13:56:00Z

modules/aws/master-asg/master.tf

@@ -61,7 +61,7 @@ resource "aws_autoscaling_group" "masters" {

 data "ignition_config" "ncg_master" {
  append {
-    source = "http://${var.cluster_name}-ncg.${var.base_domain}/ignition?profile=master"
+    source = "http://${var.cluster_name}-ncg.${var.base_domain}/ign/v1/role/master"


The docker image seems to be created from a branch which does not support rest like, but querystrings https://github.com/thorfour/tectonic-operators/blob/templates/controller/node/pkg/ignition/server.go#L81

Also I tried to modify and bazel build //controller/node/cmd/bootstrap but got running external/local_config_cc/cc_wrapper.sh failed: exit status 1 ld: library not found for -lcrt0.o

Yea the image was old. I pushed a new dev image that uses the rest path

enxebre · 2018-03-02T13:58:24Z

modules/aws/worker-asg/worker.tf

@@ -27,7 +27,7 @@ data "aws_ami" "coreos_ami" {

 data "ignition_config" "ncg_worker" {
  append {
-    source = "http://${var.cluster_name}-ncg.${var.base_domain}/ignition?profile=worker"
+    source = "http://${var.cluster_name}-ncg.${var.base_domain}/ign/v1/role/worker"


The docker image seems to be created from a branch which does not support rest like, but querystrings https://github.com/thorfour/tectonic-operators/blob/templates/controller/node/pkg/ignition/server.go#L81

Fixed in image: cb561ffb2b7c747a037231fab07a64fe6ebc7322

enxebre · 2018-03-02T14:01:24Z

modules/bootkube-ut2/resources/manifests/cluster-config.yaml

@@ -13,3 +13,24 @@ data:
    networkProfile: ${tectonic_networking}
    calicoConfig:
      mtu: ${calico_mtu}
+  tnc-config: |


cluster-config.yaml is now generated at runtime by the cli, so this template is ignored, there's already a PR to delete it.
To get this config through the cluster-config will need to modify
https://github.com/coreos/tectonic-installer/blob/master/installer/pkg/config-generator/generator.go#L66
and
https://github.com/coreos/tectonic-config

enxebre · 2018-03-02T14:02:30Z

modules/bootkube-ut2/resources/manifests/tectonic-node-controller.yaml

+kind: DaemonSet
+metadata:
+  name: tectonic-node-controller
+  namespace: tectonic-system


this lives in tectonic-system but the config in kube-system so it's not accessible for the pod

enxebre · 2018-03-02T14:03:31Z

modules/bootkube-ut2/resources/manifests/tectonic-node-controller.yaml

+      containers:
+      - name: tectonic-node-controller
+        image: ${tnc_bootstrap_image}
+        command:


this does not like to the current image, docker inspect shows:

"Entrypoint": [ "/app/controller/node/cmd/bootstrap/bootstrap" ],

So command should be removed and need to add args:

args: - --config=/etc/cluster-config/tnc-config - --port=45900 - --cert=/opt/tectonic/tls/ca.crt - --key=/opt/tectonic/tls/ca.key

thorfour · 2018-03-02T19:01:40Z

@enxebre latest dev image e10b61c01c8b297da029d517922c72cbd71017f7 supports the rest paths, and has a new flag called debug which turns off tls

thorfour · 2018-03-02T22:05:14Z

current state of affairs is bootstrapping is stuck with 2018/03/02 13:51:08 Waiting for TNC to be running, this might take a while...

enxebre · 2018-03-05T09:35:49Z

Hey @thorfour just need to set the s3 key for the bootstrap node back to the rest like url enxebre@c331064
Now we can focus on templates correctness, first issue with current config for either masters and workers is k8s-node-bootstrap.service failing because of tectonic_torcx_image_url and tectonic_torcx_image_tag are missing
Also noticed other variables not being set. The config yaml gets properly populated so this seems like tnc templates interpolation not working

thorfour · 2018-03-05T19:28:52Z

🎉 It bootstraps a cluster! (kind of)

tectonic comes up just fine. But I ended up with a single node in the cluster

thorfour · 2018-03-05T21:33:46Z

I'm wondering if that has to do with the fact that I had to run the join step after tnc bootstrapping was already torn down?

coreosbot · 2018-03-05T21:36:47Z

Can one of the admins verify this patch?

enxebre · 2018-03-06T09:02:04Z

Hey @thorfour tectonic install bootstrap will always deploy a single node cluster independently from the TNC (the bootstrap node which gets config from s3).
The rest of the nodes will get the ign config from the TNC running on the bootstrap node when running tectonic install join
Or you can just run it all in one go with tectonic install

Currently when running tectonic install bootstrap and then sshing into the bootstrap node and curl http://0.0.0.0:49500/ign/v1/role/master or curl http://0.0.0.0:49500/ign/v1/role/worker I get unable to parse node config: failed to unmarshal systemd unit into struct: yaml: unmarshal errors: so no nodes will ever be able to join the cluster.

We know this pr already take cares of deploying the tnc successfully so what needs to be done now is to ensure the TNC serves the right config so other nodes can join the cluster successfully

enxebre · 2018-03-06T09:46:49Z

this fixes the issue mentioned above, please cherry pick it enxebre@d7141f1
And these are the issues with the current templates https://github.com/coreos-inc/tectonic-operators/pull/286#issuecomment-370751593. By fixing those and #3053 (comment), nodes are able to join the cluster

enxebre · 2018-03-06T11:22:49Z

steps/assets/tectonic.tf

+  https_proxy         = "${var.tectonic_https_proxy_address}"
+  image_re            = "${var.tectonic_image_re}"
+  kube_dns_service_ip = "${module.bootkube.kube_dns_service_ip}"
+  kubelet_node_label  = "node-role.kubernetes.io/master"


this is setting master label for both master and workers

thorfour · 2018-03-06T23:07:47Z

@enxebre all green. Let's do TLS and long-running switchover in a follow-up PR

enxebre · 2018-03-07T08:34:47Z

hey @thorfour thanks a lot! This looks a good start for fully transitioning to the TNC.
TBD in follow up PRs:

To run initial TNC as a static pod (config need to be passed as host volumes here. Also I reckon the image will need to be public as there's no state to get secrets at this point) and switchover to long running TNC
To get the config for the long running TNC through the cluster-config object. Will need to modify
https://github.com/coreos/tectonic-installer/blob/master/installer/pkg/config-generator/generator.go#L66 and https://github.com/coreos/tectonic-config
Add liveness/readiness probe for TNC objects
Enable TLS (related to INST-944)

enxebre · 2018-03-07T08:43:12Z

modules/bootkube-ut2/assets.tf

+    kubelet_image_url        = "${replace(var.container_images["hyperkube"],var.image_re,"$1")}"
+    kubelet_image_tag        = "${replace(var.container_images["hyperkube"],var.image_re,"$2")}"
+    iscsi_enabled            = "${var.iscsi_enabled}"
+    kubeconfig_fetch_cmd     = "${var.kubeconfig_fetch_cmd != "" ? "ExecStartPre=${var.kubeconfig_fetch_cmd}" : ""}"


wouldn't it be to have "ExecStartPre=" as part of the go template a bit less error prone?

In 251147e (TNC bootstrapping, 2018-02-13, coreos/tectonic-installer#3053), the resource was renamed from 'ncg' to 'tnc', but the name property wasn't updated to match.

thorfour added the do-not-merge label Mar 1, 2018

thorfour requested a review from enxebre March 1, 2018 00:42

thorfour commented Mar 1, 2018

View reviewed changes

thorfour force-pushed the tnc_bootstrap branch from 7ef7ba2 to 2bb8d90 Compare March 1, 2018 17:42

enxebre suggested changes Mar 2, 2018

View reviewed changes

thorfour force-pushed the tnc_bootstrap branch from 2bb8d90 to 948de27 Compare March 2, 2018 18:29

thorfour force-pushed the tnc_bootstrap branch from be5b90b to 398fd34 Compare March 5, 2018 17:10

thorfour added platform/aws run-smoke-tests and removed do-not-merge labels Mar 5, 2018

thorfour closed this Mar 5, 2018

thorfour reopened this Mar 5, 2018

thorfour force-pushed the tnc_bootstrap branch from 3fd8a34 to 9a7eaf5 Compare March 5, 2018 23:46

enxebre reviewed Mar 6, 2018

View reviewed changes

thorfour force-pushed the tnc_bootstrap branch from 9a7eaf5 to a9025f2 Compare March 6, 2018 22:00

thorfour changed the title ~~Tnc bootstrap WIP~~ Tnc bootstrap Mar 6, 2018

TNC bootstrapping

251147e

thorfour force-pushed the tnc_bootstrap branch from a9025f2 to 251147e Compare March 6, 2018 22:11

enxebre reviewed Mar 7, 2018

View reviewed changes

enxebre approved these changes Mar 7, 2018

View reviewed changes

enxebre merged commit f4bca81 into coreos:master Mar 7, 2018

wking mentioned this pull request Aug 21, 2018

modules/aws/vpc/sg-elb: Fix name for TNC security group openshift/installer#155

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tnc bootstrap #3053

Tnc bootstrap #3053

thorfour commented Mar 1, 2018

coreosbot commented Mar 1, 2018

thorfour Mar 1, 2018

enxebre Mar 1, 2018

enxebre left a comment •

edited

Loading

enxebre Mar 2, 2018

thorfour Mar 2, 2018

enxebre Mar 2, 2018

enxebre Mar 2, 2018

thorfour Mar 2, 2018

enxebre Mar 2, 2018

thorfour Mar 2, 2018

enxebre Mar 2, 2018

enxebre Mar 2, 2018

enxebre Mar 2, 2018 •

edited

Loading

thorfour commented Mar 2, 2018

thorfour commented Mar 2, 2018

enxebre commented Mar 5, 2018 •

edited

Loading

thorfour commented Mar 5, 2018

thorfour commented Mar 5, 2018

coreosbot commented Mar 5, 2018

enxebre commented Mar 6, 2018 •

edited

Loading

enxebre commented Mar 6, 2018 •

edited

Loading

enxebre Mar 6, 2018

thorfour commented Mar 6, 2018

enxebre commented Mar 7, 2018 •

edited

Loading

enxebre Mar 7, 2018 •

edited

Loading

Tnc bootstrap #3053

Tnc bootstrap #3053

Conversation

thorfour commented Mar 1, 2018

coreosbot commented Mar 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enxebre left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enxebre Mar 2, 2018 • edited Loading

Choose a reason for hiding this comment

thorfour commented Mar 2, 2018

thorfour commented Mar 2, 2018

enxebre commented Mar 5, 2018 • edited Loading

thorfour commented Mar 5, 2018

thorfour commented Mar 5, 2018

coreosbot commented Mar 5, 2018

enxebre commented Mar 6, 2018 • edited Loading

enxebre commented Mar 6, 2018 • edited Loading

Choose a reason for hiding this comment

thorfour commented Mar 6, 2018

enxebre commented Mar 7, 2018 • edited Loading

enxebre Mar 7, 2018 • edited Loading

Choose a reason for hiding this comment

enxebre left a comment •

edited

Loading

enxebre Mar 2, 2018 •

edited

Loading

enxebre commented Mar 5, 2018 •

edited

Loading

enxebre commented Mar 6, 2018 •

edited

Loading

enxebre commented Mar 6, 2018 •

edited

Loading

enxebre commented Mar 7, 2018 •

edited

Loading

enxebre Mar 7, 2018 •

edited

Loading