Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kops update cluster stuck on nat gateway update #1917

Closed
hollowimage opened this issue Feb 15, 2017 · 21 comments
Closed

kops update cluster stuck on nat gateway update #1917

hollowimage opened this issue Feb 15, 2017 · 21 comments

Comments

@hollowimage
Copy link

@hollowimage hollowimage commented Feb 15, 2017

so im hitting a weird issue with using kops to do any sort of cluster updates post creation

[15:26:36][admin@domain.com]$ kops update cluster
Using cluster from kubectl context: prod-domain.com

I0215 15:26:55.140339   24888 dns.go:89] Private DNS: skipping DNS validation
I0215 15:26:55.147283   24888 executor.go:91] Tasks: 0 done / 96 total; 31 can run
I0215 15:27:01.612480   24888 executor.go:91] Tasks: 31 done / 96 total; 21 can run
I0215 15:27:02.418030   24888 executor.go:91] Tasks: 52 done / 96 total; 30 can run
I0215 15:27:09.089526   24888 executor.go:91] Tasks: 82 done / 96 total; 8 can run
I0215 15:27:15.795805   24888 dnsname.go:107] AliasTarget for "api.prod-domain.com." is "prod-domain.us-east-1.elb.amazonaws.com."
W0215 15:27:15.903281   24888 executor.go:109] error running task "NatGateway/us-east-1dprod-domain.com" (9m53s remaining to succeed): Field cannot be changed: ElasticIp
W0215 15:27:15.903313   24888 executor.go:109] error running task "NatGateway/us-east-1c.prod-domain.com" (9m53s remaining to succeed): Field cannot be changed: ElasticIp
W0215 15:27:15.903324   24888 executor.go:109] error running task "NatGateway/us-east-1b.prod-domain.com" (9m53s remaining to succeed): Field cannot be changed: ElasticIp
I0215 15:27:15.903342   24888 executor.go:91] Tasks: 87 done / 96 total; 6 can run
W0215 15:27:16.040721   24888 executor.go:109] error running task "NatGateway/us-east-1d.prod-domain.com" (9m53s remaining to succeed): Field cannot be changed: ElasticIp
W0215 15:27:16.040746   24888 executor.go:109] error running task "NatGateway/us-east-1c.prod-domain.com" (9m53s remaining to succeed): Field cannot be changed: ElasticIp
W0215 15:27:16.040757   24888 executor.go:109] error running task "NatGateway/us-east-1b.prod-domain.com" (9m53s remaining to succeed): Field cannot be changed: ElasticIp
I0215 15:27:16.040774   24888 executor.go:91] Tasks: 90 done / 96 total; 3 can run
W0215 15:27:16.141043   24888 executor.go:109] error running task "NatGateway/us-east-1d.prod-domain.com" (9m52s remaining to succeed): Field cannot be changed: ElasticIp
W0215 15:27:16.141069   24888 executor.go:109] error running task "NatGateway/us-east-1c.prod-domain.com" (9m52s remaining to succeed): Field cannot be changed: ElasticIp
W0215 15:27:16.141079   24888 executor.go:109] error running task "NatGateway/us-east-1b.prod-domain.com" (9m52s remaining to succeed): Field cannot be changed: ElasticIp
I0215 15:27:16.141089   24888 executor.go:124] No progress made, sleeping before retrying 3 failed task(s)

i am using Version 1.5.1 (git-01deca8) and

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

creation summary:

  1. using a shared vpc
  2. everything else was created by kops
  3. private cluster topology with weave
kops create cluster --cloud aws --name prod.domain.com --dns private --dns-zone [ZONE] --vpc=vpc-xxxxxx --topology private --master-size=m4.large --master-zones=us-east-1b,us-east-1c,us-east-1d --network-cidr =172.x.x.x/16 --node-count=3 --node-size=m4.large --zones=us-east-1b,us-east-1c,us-east-1d --networking weave --channel alpha --associate-public-ip=false

I made the following changes to the cluster post-initial creation:

  1. change egress in the route tables for utility subnets to point to custom firewall solution we have in place
  2. change security group rules to allow inbound traffic to the public subnets from 0.0.0.0/0 to x.x.x.x where the latter is the proxy source IP that handles our VPC egress and ingress
  3. i left NGWs it created with their EIPs as is. nothing was detached or deleted.

in summary the only real change was the subnet egress routing and the security group for inbound traffic to block public access.

@kwent
Copy link

@kwent kwent commented Apr 5, 2017

Same error here.

@chrislovecnm
Copy link
Member

@chrislovecnm chrislovecnm commented Apr 23, 2017

Can we get debug logging? -v=10. @kwent any other details? Is someone able to provide aws cli steps, or ui level steps on creating the vpc/networking, so that we can reproduce?

@caarlos0
Copy link
Contributor

@caarlos0 caarlos0 commented Apr 26, 2017

Same issue here. Logs -v10:

❯ kops update cluster $NAME -v10
I0425 21:53:06.175875   19569 s3context.go:114] Found bucket "kubernetes.prod.contaazul.local" in region "us-east-1"
I0425 21:53:06.175940   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/config"
I0425 21:53:06.397091   19569 s3fs.go:199] Listing objects in S3 bucket "kubernetes.prod.contaazul.local" with prefix "kubernetes.prod.us-east-1.contaazul.local/instancegroup/"
I0425 21:53:06.773255   19569 s3fs.go:225] Listed files in s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/instancegroup: [s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/instancegroup/master-us-east-1a s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/instancegroup/nodes]
I0425 21:53:06.773303   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/instancegroup/master-us-east-1a"
I0425 21:53:06.948231   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/instancegroup/nodes"
I0425 21:53:07.221431   19569 channel.go:92] resolving "stable" against default channel location "https://raw.githubusercontent.com/kubernetes/kops/master/channels/"
I0425 21:53:07.221473   19569 channel.go:97] Loading channel from "https://raw.githubusercontent.com/kubernetes/kops/master/channels/stable"
I0425 21:53:07.221501   19569 context.go:126] Performing HTTP request: GET https://raw.githubusercontent.com/kubernetes/kops/master/channels/stable
I0425 21:53:07.924616   19569 channel.go:106] Channel contents: spec:
  images:
    # We put the "legacy" version first, for kops versions that don't support versions ( < 1.5.0 )
    - name: kope.io/k8s-1.4-debian-jessie-amd64-hvm-ebs-2016-10-21
      providerID: aws
      kubernetesVersion: ">=1.4.0 <1.5.0"
    - name: kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09
      providerID: aws
      kubernetesVersion: ">=1.5.0"
  cluster:
    kubernetesVersion: v1.4.8
    networking:
      kubenet: {}
  kubernetesVersions:
  - range: ">=1.5.0"
    recommendedVersion: 1.5.2
    requiredVersion: 1.5.1
  - range: "<1.5.0"
    recommendedVersion: 1.4.8
    requiredVersion: 1.4.2
  kopsVersions:
  - range: ">=1.5.0-alpha1"
    recommendedVersion: 1.5.1
    #requiredVersion: 1.5.1
    kubernetesVersion: 1.5.2
  - range: "<1.5.0"
    recommendedVersion: 1.4.4
    #requiredVersion: 1.4.4
    kubernetesVersion: 1.4.8
I0425 21:53:07.925797   19569 populate_cluster_spec.go:343] Defaulted KubeControllerManager.ClusterCIDR to 100.96.0.0/11
I0425 21:53:07.925822   19569 populate_cluster_spec.go:350] Defaulted ServiceClusterIPRange to 100.64.0.0/13
I0425 21:53:07.925868   19569 aws_utils.go:38] Querying EC2 for all valid regions
I0425 21:53:08.562629   19569 aws_cloud.go:631] Querying EC2 for all valid zones in region "us-east-1"
I0425 21:53:08.563019   19569 request_logger.go:45] AWS request: ec2/DescribeAvailabilityZones
I0425 21:53:08.731429   19569 subnets.go:48] All subnets have CIDRs; skipping asssignment logic
I0425 21:53:08.731502   19569 aws_cloud.go:631] Querying EC2 for all valid zones in region "us-east-1"
I0425 21:53:08.731698   19569 request_logger.go:45] AWS request: ec2/DescribeAvailabilityZones
I0425 21:53:08.898286   19569 utils.go:140] Querying for all DNS zones to find match for "kubernetes.prod.us-east-1.contaazul.local"
I0425 21:53:09.890192   19569 populate_cluster_spec.go:236] Defaulting DNS zone to: ZCMPK68VV5UY9
I0425 21:53:09.890234   19569 tagbuilder.go:96] tags: [_aws _k8s_1_5 _networking_cni]
I0425 21:53:09.890299   19569 tree_walker.go:97] visit "config/_aws"
I0425 21:53:09.890323   19569 tree_walker.go:124] Descending into directory, as tag is present: "config/_aws"
I0425 21:53:09.890333   19569 tree_walker.go:97] visit "config/_aws/defaults.options"
I0425 21:53:09.890573   19569 tree_walker.go:97] visit "config/_gce"
I0425 21:53:09.890605   19569 tree_walker.go:120] Skipping directory "config/_gce" as tag "_gce" not present
I0425 21:53:09.890620   19569 tree_walker.go:97] visit "config/components"
I0425 21:53:09.890647   19569 tree_walker.go:97] visit "config/components/docker"
I0425 21:53:09.890673   19569 tree_walker.go:97] visit "config/components/docker/_networking_kubenet"
I0425 21:53:09.890698   19569 tree_walker.go:120] Skipping directory "config/components/docker/_networking_kubenet" as tag "_networking_kubenet" not present
I0425 21:53:09.890716   19569 tree_walker.go:97] visit "config/components/docker/docker.options"
I0425 21:53:09.890873   19569 tree_walker.go:97] visit "config/components/docker/_e2e_storage_test_environment"
I0425 21:53:09.890907   19569 tree_walker.go:120] Skipping directory "config/components/docker/_e2e_storage_test_environment" as tag "_e2e_storage_test_environment" not present
I0425 21:53:09.890926   19569 tree_walker.go:97] visit "config/components/docker/_networking_cni"
I0425 21:53:09.890949   19569 tree_walker.go:124] Descending into directory, as tag is present: "config/components/docker/_networking_cni"
I0425 21:53:09.890969   19569 tree_walker.go:97] visit "config/components/docker/_networking_cni/cni.options"
I0425 21:53:09.891072   19569 tree_walker.go:97] visit "config/components/kube-apiserver"
I0425 21:53:09.891122   19569 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_3"
I0425 21:53:09.891148   19569 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_k8s_1_3" as tag "_k8s_1_3" not present
I0425 21:53:09.891163   19569 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_4"
I0425 21:53:09.891185   19569 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_k8s_1_4" as tag "_k8s_1_4" not present
I0425 21:53:09.891200   19569 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_5"
I0425 21:53:09.891219   19569 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-apiserver/_k8s_1_5"
I0425 21:53:09.891237   19569 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_5/kube-apiserver.options"
I0425 21:53:09.891380   19569 tree_walker.go:97] visit "config/components/kube-apiserver/kube-apiserver.options"
I0425 21:53:09.891559   19569 tree_walker.go:97] visit "config/components/kube-apiserver/_aws"
I0425 21:53:09.891592   19569 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-apiserver/_aws"
I0425 21:53:09.891609   19569 tree_walker.go:97] visit "config/components/kube-apiserver/_aws/kube-apiserver.aws.options"
I0425 21:53:09.891701   19569 tree_walker.go:97] visit "config/components/kube-apiserver/_gce"
I0425 21:53:09.891730   19569 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_gce" as tag "_gce" not present
I0425 21:53:09.891747   19569 tree_walker.go:97] visit "config/components/kube-proxy"
I0425 21:53:09.891769   19569 tree_walker.go:97] visit "config/components/kube-proxy/kube-proxy.options"
I0425 21:53:09.891982   19569 tree_walker.go:97] visit "config/components/kube-scheduler"
I0425 21:53:09.892016   19569 tree_walker.go:97] visit "config/components/kube-scheduler/kube-scheduler.options"
I0425 21:53:09.892475   19569 options_loader.go:101] executing template components/docker/docker.options (tags=[])
I0425 21:53:09.892664   19569 options_loader.go:101] executing template components/kube-apiserver/kube-apiserver.options (tags=[])
I0425 21:53:09.893033   19569 options_loader.go:101] executing template components/kube-proxy/kube-proxy.options (tags=[])
I0425 21:53:09.893219   19569 options_loader.go:101] executing template components/kube-scheduler/kube-scheduler.options (tags=[])
I0425 21:53:09.893423   19569 options_loader.go:101] executing template _aws/defaults.options (tags=[_aws])
I0425 21:53:09.893480   19569 options_loader.go:101] executing template components/docker/_networking_cni/cni.options (tags=[_networking_cni])
I0425 21:53:09.893555   19569 options_loader.go:101] executing template components/kube-apiserver/_aws/kube-apiserver.aws.options (tags=[_aws])
I0425 21:53:09.893660   19569 options_loader.go:101] executing template components/kube-apiserver/_k8s_1_5/kube-apiserver.options (tags=[_k8s_1_5])
I0425 21:53:09.893834   19569 options_loader.go:128] executing builder *components.DefaultsOptionsBuilder
I0425 21:53:09.893848   19569 options_loader.go:128] executing builder *components.KubeAPIServerOptionsBuilder
I0425 21:53:09.893861   19569 options_loader.go:128] executing builder *components.DockerOptionsBuilder
I0425 21:53:09.893874   19569 options_loader.go:128] executing builder *components.NetworkingOptionsBuilder
I0425 21:53:09.893881   19569 options_loader.go:128] executing builder *components.KubeDnsOptionsBuilder
I0425 21:53:09.893900   19569 options_loader.go:128] executing builder *components.KubeletOptionsBuilder
I0425 21:53:09.893924   19569 options_loader.go:128] executing builder *components.KubeControllerManagerOptionsBuilder
I0425 21:53:09.893949   19569 kubecontrollermanager.go:74] Kubernetes version "1.5.4" supports AttachDetachReconcileSyncPeriod; will configure
I0425 21:53:09.893979   19569 kubecontrollermanager.go:79] AttachDetachReconcileSyncPeriod is not set; will set to default 1m0s
I0425 21:53:09.894488   19569 options_loader.go:101] executing template components/docker/docker.options (tags=[])
I0425 21:53:09.894646   19569 options_loader.go:101] executing template components/kube-apiserver/kube-apiserver.options (tags=[])
I0425 21:53:09.894956   19569 options_loader.go:101] executing template components/kube-proxy/kube-proxy.options (tags=[])
I0425 21:53:09.895148   19569 options_loader.go:101] executing template components/kube-scheduler/kube-scheduler.options (tags=[])
I0425 21:53:09.895300   19569 options_loader.go:101] executing template _aws/defaults.options (tags=[_aws])
I0425 21:53:09.895365   19569 options_loader.go:101] executing template components/docker/_networking_cni/cni.options (tags=[_networking_cni])
I0425 21:53:09.895463   19569 options_loader.go:101] executing template components/kube-apiserver/_aws/kube-apiserver.aws.options (tags=[_aws])
I0425 21:53:09.895560   19569 options_loader.go:101] executing template components/kube-apiserver/_k8s_1_5/kube-apiserver.options (tags=[_k8s_1_5])
I0425 21:53:09.895742   19569 options_loader.go:128] executing builder *components.DefaultsOptionsBuilder
I0425 21:53:09.895755   19569 options_loader.go:128] executing builder *components.KubeAPIServerOptionsBuilder
I0425 21:53:09.895764   19569 options_loader.go:128] executing builder *components.DockerOptionsBuilder
I0425 21:53:09.895770   19569 options_loader.go:128] executing builder *components.NetworkingOptionsBuilder
I0425 21:53:09.895777   19569 options_loader.go:128] executing builder *components.KubeDnsOptionsBuilder
I0425 21:53:09.895786   19569 options_loader.go:128] executing builder *components.KubeletOptionsBuilder
I0425 21:53:09.895810   19569 options_loader.go:128] executing builder *components.KubeControllerManagerOptionsBuilder
I0425 21:53:09.895834   19569 kubecontrollermanager.go:74] Kubernetes version "1.5.4" supports AttachDetachReconcileSyncPeriod; will configure
I0425 21:53:09.896424   19569 spec_builder.go:68] options: {
  "channel": "stable",
  "configBase": "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local",
  "cloudProvider": "aws",
  "kubernetesVersion": "1.5.4",
  "subnets": [
    {
      "name": "us-east-1a",
      "zone": "us-east-1a",
      "cidr": "10.110.240.0/24",
      "type": "Private"
    },
    {
      "name": "us-east-1e",
      "zone": "us-east-1e",
      "cidr": "10.110.241.0/24",
      "type": "Private"
    },
    {
      "name": "utility-us-east-1a",
      "zone": "us-east-1a",
      "cidr": "10.110.200.0/21",
      "type": "Utility"
    },
    {
      "name": "utility-us-east-1e",
      "zone": "us-east-1e",
      "cidr": "10.110.208.0/21",
      "type": "Utility"
    }
  ],
  "masterPublicName": "api.kubernetes.prod.us-east-1.contaazul.local",
  "masterInternalName": "api.internal.kubernetes.prod.us-east-1.contaazul.local",
  "networkCIDR": "10.110.0.0/16",
  "networkID": "vpc-66673502",
  "topology": {
    "masters": "private",
    "nodes": "private",
    "dns": {
      "type": "Private"
    }
  },
  "secretStore": "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/secrets",
  "keyStore": "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki",
  "configStore": "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local",
  "dnsZone": "ZCMPK68VV5UY9",
  "clusterDNSDomain": "cluster.local",
  "serviceClusterIPRange": "100.64.0.0/13",
  "nonMasqueradeCIDR": "100.64.0.0/10",
  "sshAccess": [
    "0.0.0.0/0"
  ],
  "kubernetesApiAccess": [
    "0.0.0.0/0"
  ],
  "etcdClusters": [
    {
      "name": "main",
      "etcdMembers": [
        {
          "name": "a",
          "instanceGroup": "master-us-east-1a"
        }
      ]
    },
    {
      "name": "events",
      "etcdMembers": [
        {
          "name": "a",
          "instanceGroup": "master-us-east-1a"
        }
      ]
    }
  ],
  "docker": {
    "bridge": "",
    "logLevel": "warn",
    "ipTables": false,
    "ipMasq": false,
    "storage": "overlay,aufs",
    "version": "1.12.3"
  },
  "kubeDNS": {
    "image": "gcr.io/google_containers/kubedns-amd64:1.3",
    "replicas": 2,
    "domain": "cluster.local",
    "serverIP": "100.64.0.10"
  },
  "kubeAPIServer": {
    "pathSrvKubernetes": "/srv/kubernetes",
    "pathSrvSshproxy": "/srv/sshproxy",
    "image": "gcr.io/google_containers/kube-apiserver:v1.5.4",
    "logLevel": 2,
    "cloudProvider": "aws",
    "securePort": 443,
    "address": "127.0.0.1",
    "etcdServers": [
      "http://127.0.0.1:4001"
    ],
    "etcdServersOverrides": [
      "/events#http://127.0.0.1:4002"
    ],
    "admissionControl": [
      "NamespaceLifecycle",
      "LimitRanger",
      "ServiceAccount",
      "PersistentVolumeLabel",
      "DefaultStorageClass",
      "ResourceQuota"
    ],
    "serviceClusterIPRange": "100.64.0.0/13",
    "clientCAFile": "/srv/kubernetes/ca.crt",
    "basicAuthFile": "/srv/kubernetes/basic_auth.csv",
    "tlsCertFile": "/srv/kubernetes/server.cert",
    "tlsPrivateKeyFile": "/srv/kubernetes/server.key",
    "tokenAuthFile": "/srv/kubernetes/known_tokens.csv",
    "allowPrivileged": true,
    "apiServerCount": 1,
    "anonymousAuth": false,
    "kubeletPreferredAddressTypes": [
      "InternalIP",
      "Hostname",
      "ExternalIP",
      "LegacyHostIP"
    ],
    "storageBackend": "etcd2"
  },
  "kubeControllerManager": {
    "master": "127.0.0.1:8080",
    "logLevel": 2,
    "serviceAccountPrivateKeyFile": "/srv/kubernetes/server.key",
    "image": "gcr.io/google_containers/kube-controller-manager:v1.5.4",
    "pathSrvKubernetes": "/srv/kubernetes",
    "cloudProvider": "aws",
    "clusterName": "kubernetes.prod.us-east-1.contaazul.local",
    "clusterCIDR": "100.96.0.0/11",
    "allocateNodeCIDRs": true,
    "configureCloudRoutes": false,
    "rootCAFile": "/srv/kubernetes/ca.crt",
    "leaderElection": {
      "leaderElect": true
    },
    "attachDetachReconcileSyncPeriod": "1m0s"
  },
  "kubeScheduler": {
    "master": "127.0.0.1:8080",
    "logLevel": 2,
    "image": "gcr.io/google_containers/kube-scheduler:v1.5.4",
    "leaderElection": {
      "leaderElect": true
    }
  },
  "kubeProxy": {
    "image": "gcr.io/google_containers/kube-proxy:v1.5.4",
    "cpuRequest": "100m",
    "logLevel": 2,
    "master": "https://api.internal.kubernetes.prod.us-east-1.contaazul.local"
  },
  "kubelet": {
    "apiServers": "https://api.internal.kubernetes.prod.us-east-1.contaazul.local",
    "logLevel": 2,
    "podManifestPath": "/etc/kubernetes/manifests",
    "hostnameOverride": "@aws",
    "allowPrivileged": true,
    "enableDebuggingHandlers": true,
    "clusterDomain": "cluster.local",
    "clusterDNS": "100.64.0.10",
    "networkPluginName": "cni",
    "cloudProvider": "aws",
    "cgroupRoot": "docker",
    "babysitDaemons": true,
    "nonMasqueradeCIDR": "100.64.0.0/10",
    "networkPluginMTU": null,
    "evictionHard": "memory.available\u003c100Mi,nodefs.available\u003c10%,nodefs.inodesFree\u003c5%,imagefs.available\u003c10%,imagefs.inodesFree\u003c5%"
  },
  "masterKubelet": {
    "apiServers": "http://127.0.0.1:8080",
    "logLevel": 2,
    "podManifestPath": "/etc/kubernetes/manifests",
    "hostnameOverride": "@aws",
    "allowPrivileged": true,
    "enableDebuggingHandlers": true,
    "clusterDomain": "cluster.local",
    "clusterDNS": "100.64.0.10",
    "networkPluginName": "cni",
    "cloudProvider": "aws",
    "cgroupRoot": "docker",
    "babysitDaemons": true,
    "registerSchedulable": false,
    "nonMasqueradeCIDR": "100.64.0.0/10",
    "networkPluginMTU": null,
    "evictionHard": "memory.available\u003c100Mi,nodefs.available\u003c10%,nodefs.inodesFree\u003c5%,imagefs.available\u003c10%,imagefs.inodesFree\u003c5%"
  },
  "networking": {
    "cni": {}
  },
  "api": {
    "loadBalancer": {
      "type": "Public"
    }
  }
}
I0425 21:53:09.897286   19569 channel.go:157] RecommendedVersion="1.5.1", Have="1.5.3".  No upgrade needed.
I0425 21:53:09.897312   19569 channel.go:185] VersionRecommendationSpec does not specify RequiredVersion
I0425 21:53:09.897333   19569 channel.go:137] RecommendedVersion="1.5.2", Have="1.5.4".  No upgrade needed.
I0425 21:53:09.897349   19569 channel.go:177] RequiredVersion="1.5.1", Have="1.5.4".  No upgrade needed.
I0425 21:53:09.897444   19569 apply_cluster.go:204] Adding default kubelet release asset: https://storage.googleapis.com/kubernetes-release/release/v1.5.4/bin/linux/amd64/kubelet
I0425 21:53:09.897463   19569 context.go:126] Performing HTTP request: GET https://storage.googleapis.com/kubernetes-release/release/v1.5.4/bin/linux/amd64/kubelet.sha1
I0425 21:53:10.554284   19569 apply_cluster.go:705] Found hash "24ed2e036c53877174eb34efdb0249b81fcea467" for "https://storage.googleapis.com/kubernetes-release/release/v1.5.4/bin/linux/amd64/kubelet"
I0425 21:53:10.554363   19569 apply_cluster.go:215] Adding default kubectl release asset: https://storage.googleapis.com/kubernetes-release/release/v1.5.4/bin/linux/amd64/kubectl
I0425 21:53:10.554386   19569 context.go:126] Performing HTTP request: GET https://storage.googleapis.com/kubernetes-release/release/v1.5.4/bin/linux/amd64/kubectl.sha1
I0425 21:53:10.717324   19569 apply_cluster.go:705] Found hash "15d8430dc52b1f3772b88bc6a236c8fa58e07c0d" for "https://storage.googleapis.com/kubernetes-release/release/v1.5.4/bin/linux/amd64/kubectl"
I0425 21:53:10.717376   19569 networking.go:94] Adding default CNI asset: https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz
I0425 21:53:10.717386   19569 urls.go:40] Using default base url: "https://kubeupv2.s3.amazonaws.com/kops/1.5.3/"
I0425 21:53:10.717397   19569 apply_cluster.go:231] Using default utils.tar.gz location: "https://kubeupv2.s3.amazonaws.com/kops/1.5.3/linux/amd64/utils.tar.gz"
I0425 21:53:10.717414   19569 context.go:126] Performing HTTP request: GET https://kubeupv2.s3.amazonaws.com/kops/1.5.3/linux/amd64/utils.tar.gz.sha1
I0425 21:53:11.608053   19569 apply_cluster.go:705] Found hash "661289336f118e82194000adc63f2b72f449636e" for "https://kubeupv2.s3.amazonaws.com/kops/1.5.3/linux/amd64/utils.tar.gz"
I0425 21:53:11.608099   19569 urls.go:64] Using default nodeup location: "https://kubeupv2.s3.amazonaws.com/kops/1.5.3/linux/amd64/nodeup"
I0425 21:53:11.608141   19569 aws_cloud.go:631] Querying EC2 for all valid zones in region "us-east-1"
I0425 21:53:11.608315   19569 request_logger.go:45] AWS request: ec2/DescribeAvailabilityZones
I0425 21:53:11.777689   19569 s3fs.go:199] Listing objects in S3 bucket "kubernetes.prod.contaazul.local" with prefix "kubernetes.prod.us-east-1.contaazul.local/pki/ssh/public/admin/"
I0425 21:53:11.946497   19569 s3fs.go:225] Listed files in s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/ssh/public/admin: [s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/ssh/public/admin/11fed5e49f6d77f0e01072067833a14b]
I0425 21:53:11.946535   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/ssh/public/admin/11fed5e49f6d77f0e01072067833a14b"
I0425 21:53:12.135336   19569 dns.go:90] Private DNS: skipping DNS validation
I0425 21:53:12.363028   19569 tagbuilder.go:96] tags: [_aws _k8s_1_5 _networking_cni]
I0425 21:53:12.363105   19569 tree_walker.go:97] visit "config/_aws"
I0425 21:53:12.363124   19569 tree_walker.go:124] Descending into directory, as tag is present: "config/_aws"
I0425 21:53:12.363134   19569 tree_walker.go:97] visit "config/_aws/defaults.options"
I0425 21:53:12.363146   19569 tree_walker.go:97] visit "config/_gce"
I0425 21:53:12.363160   19569 tree_walker.go:120] Skipping directory "config/_gce" as tag "_gce" not present
I0425 21:53:12.363168   19569 tree_walker.go:97] visit "config/components"
I0425 21:53:12.363226   19569 tree_walker.go:97] visit "config/components/docker"
I0425 21:53:12.363267   19569 tree_walker.go:97] visit "config/components/docker/_e2e_storage_test_environment"
I0425 21:53:12.363297   19569 tree_walker.go:120] Skipping directory "config/components/docker/_e2e_storage_test_environment" as tag "_e2e_storage_test_environment" not present
I0425 21:53:12.363315   19569 tree_walker.go:97] visit "config/components/docker/_networking_cni"
I0425 21:53:12.363337   19569 tree_walker.go:124] Descending into directory, as tag is present: "config/components/docker/_networking_cni"
I0425 21:53:12.363355   19569 tree_walker.go:97] visit "config/components/docker/_networking_cni/cni.options"
I0425 21:53:12.363380   19569 tree_walker.go:97] visit "config/components/docker/_networking_kubenet"
I0425 21:53:12.363405   19569 tree_walker.go:120] Skipping directory "config/components/docker/_networking_kubenet" as tag "_networking_kubenet" not present
I0425 21:53:12.363421   19569 tree_walker.go:97] visit "config/components/docker/docker.options"
I0425 21:53:12.363455   19569 tree_walker.go:97] visit "config/components/kube-apiserver"
I0425 21:53:12.363489   19569 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_3"
I0425 21:53:12.363517   19569 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_k8s_1_3" as tag "_k8s_1_3" not present
I0425 21:53:12.363533   19569 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_4"
I0425 21:53:12.363557   19569 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_k8s_1_4" as tag "_k8s_1_4" not present
I0425 21:53:12.363572   19569 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_5"
I0425 21:53:12.363593   19569 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-apiserver/_k8s_1_5"
I0425 21:53:12.363611   19569 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_5/kube-apiserver.options"
I0425 21:53:12.363635   19569 tree_walker.go:97] visit "config/components/kube-apiserver/kube-apiserver.options"
I0425 21:53:12.363728   19569 tree_walker.go:97] visit "config/components/kube-apiserver/_aws"
I0425 21:53:12.363752   19569 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-apiserver/_aws"
I0425 21:53:12.363769   19569 tree_walker.go:97] visit "config/components/kube-apiserver/_aws/kube-apiserver.aws.options"
I0425 21:53:12.363794   19569 tree_walker.go:97] visit "config/components/kube-apiserver/_gce"
I0425 21:53:12.363819   19569 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_gce" as tag "_gce" not present
I0425 21:53:12.363834   19569 tree_walker.go:97] visit "config/components/kube-proxy"
I0425 21:53:12.363856   19569 tree_walker.go:97] visit "config/components/kube-proxy/kube-proxy.options"
I0425 21:53:12.363889   19569 tree_walker.go:97] visit "config/components/kube-scheduler"
I0425 21:53:12.363913   19569 tree_walker.go:97] visit "config/components/kube-scheduler/kube-scheduler.options"
I0425 21:53:12.363939   19569 tree_walker.go:97] visit "cloudup/resources"
I0425 21:53:12.363961   19569 tree_walker.go:97] visit "cloudup/resources/addons"
I0425 21:53:12.364000   19569 tree_walker.go:97] visit "cloudup/resources/addons/networking.projectcalico.org.canal"
I0425 21:53:12.364028   19569 tree_walker.go:97] visit "cloudup/resources/addons/networking.projectcalico.org.canal/v1.0.yaml.template"
I0425 21:53:12.364254   19569 loader.go:282] loading (templated) resource "addons/networking.projectcalico.org.canal/v1.0.yaml"
I0425 21:53:12.364285   19569 tree_walker.go:97] visit "cloudup/resources/addons/networking.weave"
I0425 21:53:12.364306   19569 tree_walker.go:97] visit "cloudup/resources/addons/networking.weave/v1.9.0.yaml"
I0425 21:53:12.364437   19569 loader.go:290] loading resource "addons/networking.weave/v1.9.0.yaml"
I0425 21:53:12.364458   19569 tree_walker.go:97] visit "cloudup/resources/addons/networking.weave/v1.9.2.yaml"
I0425 21:53:12.364570   19569 loader.go:290] loading resource "addons/networking.weave/v1.9.2.yaml"
I0425 21:53:12.364588   19569 tree_walker.go:97] visit "cloudup/resources/addons/networking.weave/v1.9.3.yaml"
I0425 21:53:12.364762   19569 loader.go:290] loading resource "addons/networking.weave/v1.9.3.yaml"
I0425 21:53:12.364782   19569 tree_walker.go:97] visit "cloudup/resources/addons/storage-aws.addons.k8s.io"
I0425 21:53:12.364803   19569 tree_walker.go:97] visit "cloudup/resources/addons/storage-aws.addons.k8s.io/v1.5.0.yaml"
I0425 21:53:12.364904   19569 loader.go:290] loading resource "addons/storage-aws.addons.k8s.io/v1.5.0.yaml"
I0425 21:53:12.364937   19569 tree_walker.go:97] visit "cloudup/resources/addons/core.addons.k8s.io"
I0425 21:53:12.364968   19569 tree_walker.go:97] visit "cloudup/resources/addons/core.addons.k8s.io/addon.yaml"
I0425 21:53:12.365034   19569 loader.go:290] loading resource "addons/core.addons.k8s.io/addon.yaml"
I0425 21:53:12.365051   19569 tree_walker.go:97] visit "cloudup/resources/addons/core.addons.k8s.io/v1.4.0.yaml"
I0425 21:53:12.365119   19569 loader.go:290] loading resource "addons/core.addons.k8s.io/v1.4.0.yaml"
I0425 21:53:12.365135   19569 tree_walker.go:97] visit "cloudup/resources/addons/limit-range.addons.k8s.io"
I0425 21:53:12.365167   19569 tree_walker.go:97] visit "cloudup/resources/addons/limit-range.addons.k8s.io/addon.yaml"
I0425 21:53:12.365259   19569 loader.go:290] loading resource "addons/limit-range.addons.k8s.io/addon.yaml"
I0425 21:53:12.365277   19569 tree_walker.go:97] visit "cloudup/resources/addons/limit-range.addons.k8s.io/v1.5.0.yaml"
I0425 21:53:12.365372   19569 loader.go:290] loading resource "addons/limit-range.addons.k8s.io/v1.5.0.yaml"
I0425 21:53:12.365393   19569 tree_walker.go:97] visit "cloudup/resources/addons/networking.kope.io"
I0425 21:53:12.365417   19569 tree_walker.go:97] visit "cloudup/resources/addons/networking.kope.io/v1.0.20161116.yaml"
I0425 21:53:12.365531   19569 loader.go:290] loading resource "addons/networking.kope.io/v1.0.20161116.yaml"
I0425 21:53:12.365550   19569 tree_walker.go:97] visit "cloudup/resources/addons/networking.projectcalico.org"
I0425 21:53:12.365575   19569 tree_walker.go:97] visit "cloudup/resources/addons/networking.projectcalico.org/v2.0.2.yaml.template"
I0425 21:53:12.365802   19569 loader.go:282] loading (templated) resource "addons/networking.projectcalico.org/v2.0.2.yaml"
I0425 21:53:12.365830   19569 tree_walker.go:97] visit "cloudup/resources/addons/dns-controller.addons.k8s.io"
I0425 21:53:12.365862   19569 tree_walker.go:97] visit "cloudup/resources/addons/dns-controller.addons.k8s.io/addon.yaml"
I0425 21:53:12.365932   19569 loader.go:290] loading resource "addons/dns-controller.addons.k8s.io/addon.yaml"
I0425 21:53:12.365948   19569 tree_walker.go:97] visit "cloudup/resources/addons/dns-controller.addons.k8s.io/v1.5.2.yaml.template"
I0425 21:53:12.366078   19569 loader.go:282] loading (templated) resource "addons/dns-controller.addons.k8s.io/v1.5.2.yaml"
I0425 21:53:12.366095   19569 tree_walker.go:97] visit "cloudup/resources/addons/kube-dns.addons.k8s.io"
I0425 21:53:12.366114   19569 tree_walker.go:97] visit "cloudup/resources/addons/kube-dns.addons.k8s.io/v1.4.0.yaml.template"
I0425 21:53:12.366751   19569 loader.go:282] loading (templated) resource "addons/kube-dns.addons.k8s.io/v1.4.0.yaml"
I0425 21:53:12.366807   19569 tree_walker.go:97] visit "cloudup/resources/addons/kube-dns.addons.k8s.io/v1.5.1.yaml.template"
I0425 21:53:12.367037   19569 loader.go:282] loading (templated) resource "addons/kube-dns.addons.k8s.io/v1.5.1.yaml"
I0425 21:53:12.367067   19569 tree_walker.go:97] visit "cloudup/resources/addons/kube-dns.addons.k8s.io/v1.6.0.yaml.template"
I0425 21:53:12.367339   19569 loader.go:282] loading (templated) resource "addons/kube-dns.addons.k8s.io/v1.6.0.yaml"
I0425 21:53:12.367379   19569 tree_walker.go:97] visit "cloudup/resources/addons/networking.flannel"
I0425 21:53:12.367417   19569 tree_walker.go:97] visit "cloudup/resources/addons/networking.flannel/v0.7.0.yaml"
I0425 21:53:12.367633   19569 loader.go:290] loading resource "addons/networking.flannel/v0.7.0.yaml"
I0425 21:53:12.367656   19569 tree_walker.go:97] visit "cloudup/tokens"
I0425 21:53:12.367695   19569 tree_walker.go:97] visit "cloudup/tokens/tokens.yaml"
I0425 21:53:12.367717   19569 loader.go:300] Reading cloudup/tokens/tokens.yaml
I0425 21:53:12.368905   19569 loader.go:396] Built tokens/tokens.yaml:secret/system-dns => *fitasks.Secret {"Name":"system:dns"}
I0425 21:53:12.369047   19569 loader.go:396] Built tokens/tokens.yaml:secret/system-logging => *fitasks.Secret {"Name":"system:logging"}
I0425 21:53:12.369173   19569 loader.go:396] Built tokens/tokens.yaml:secret/system-monitoring => *fitasks.Secret {"Name":"system:monitoring"}
I0425 21:53:12.369222   19569 loader.go:396] Built tokens/tokens.yaml:secret/system-scheduler => *fitasks.Secret {"Name":"system:scheduler"}
I0425 21:53:12.369248   19569 loader.go:396] Built tokens/tokens.yaml:secret/admin => *fitasks.Secret {"Name":null}
I0425 21:53:12.369284   19569 loader.go:396] Built tokens/tokens.yaml:secret/kube => *fitasks.Secret {"Name":null}
I0425 21:53:12.369317   19569 loader.go:396] Built tokens/tokens.yaml:secret/system-controller_manager => *fitasks.Secret {"Name":"system:controller_manager"}
I0425 21:53:12.369342   19569 loader.go:396] Built tokens/tokens.yaml:secret/kube-proxy => *fitasks.Secret {"Name":null}
I0425 21:53:12.369365   19569 loader.go:396] Built tokens/tokens.yaml:secret/kubelet => *fitasks.Secret {"Name":null}
I0425 21:53:12.370430   19569 task.go:71] EnsureTask ignoring identical
I0425 21:53:12.370581   19569 iam_builder.go:226] Ignoring location "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/" because found parent "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/"
I0425 21:53:12.370604   19569 iam_builder.go:226] Ignoring location "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/secrets/" because found parent "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/"
I0425 21:53:12.370617   19569 iam_builder.go:231] Found root location "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/"
I0425 21:53:12.370893   19569 iam_builder.go:226] Ignoring location "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/" because found parent "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/"
I0425 21:53:12.370915   19569 iam_builder.go:226] Ignoring location "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/secrets/" because found parent "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/"
I0425 21:53:12.370937   19569 iam_builder.go:231] Found root location "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/"
I0425 21:53:12.371117   19569 network.go:55] Kubernetes version "1.5.4"; skipping EnableDNSHostnames requirement on VPC
I0425 21:53:12.372295   19569 topological_sort.go:62] Dependencies:
I0425 21:53:12.372309   19569 topological_sort.go:64] 	LaunchConfiguration/nodes.kubernetes.prod.us-east-1.contaazul.local:	[SSHKey/kubernetes.kubernetes.prod.us-east-1.contaazul.local-11:fe:d5:e4:9f:6d:77:f0:e0:10:72:06:78:33:a1:4b SecurityGroup/nodes.kubernetes.prod.us-east-1.contaazul.local IAMInstanceProfile/nodes.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372322   19569 topological_sort.go:64] 	IAMRolePolicy/nodes.kubernetes.prod.us-east-1.contaazul.local:	[IAMRole/nodes.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372330   19569 topological_sort.go:64] 	kubernetes.prod.us-east-1.contaazul.local-addons-storage-aws.addons.k8s.io:	[]
I0425 21:53:12.372335   19569 topological_sort.go:64] 	Subnet/utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local:	[VPC/kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372341   19569 topological_sort.go:64] 	secret/system-controller_manager:	[]
I0425 21:53:12.372347   19569 topological_sort.go:64] 	RouteTable/kubernetes.prod.us-east-1.contaazul.local:	[VPC/kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372370   19569 topological_sort.go:64] 	Subnet/us-east-1a.kubernetes.prod.us-east-1.contaazul.local:	[VPC/kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372377   19569 topological_sort.go:64] 	DNSZone/ZCMPK68VV5UY9:	[VPC/kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372383   19569 topological_sort.go:64] 	Keypair/kubelet:	[]
I0425 21:53:12.372388   19569 topological_sort.go:64] 	SecurityGroup/masters.kubernetes.prod.us-east-1.contaazul.local:	[VPC/kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372409   19569 topological_sort.go:64] 	AutoscalingGroup/nodes.kubernetes.prod.us-east-1.contaazul.local:	[Subnet/us-east-1a.kubernetes.prod.us-east-1.contaazul.local Subnet/us-east-1e.kubernetes.prod.us-east-1.contaazul.local LaunchConfiguration/nodes.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372435   19569 topological_sort.go:64] 	secret/system-monitoring:	[]
I0425 21:53:12.372466   19569 topological_sort.go:64] 	Keypair/master:	[]
I0425 21:53:12.372477   19569 topological_sort.go:64] 	RouteTableAssociation/utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local:	[RouteTable/kubernetes.prod.us-east-1.contaazul.local Subnet/utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372489   19569 topological_sort.go:64] 	ElasticIP/us-east-1a.kubernetes.prod.us-east-1.contaazul.local:	[RouteTable/private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372501   19569 topological_sort.go:64] 	IAMInstanceProfile/masters.kubernetes.prod.us-east-1.contaazul.local:	[]
I0425 21:53:12.372511   19569 topological_sort.go:64] 	SecurityGroupRule/https-api-elb-0.0.0.0/0:	[SecurityGroup/api-elb.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372521   19569 topological_sort.go:64] 	Route/private-us-east-1e-0.0.0.0/0:	[RouteTable/private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local NatGateway/us-east-1e.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372555   19569 topological_sort.go:64] 	SecurityGroupRule/node-to-master-tcp-1-4000:	[SecurityGroup/masters.kubernetes.prod.us-east-1.contaazul.local SecurityGroup/nodes.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372567   19569 topological_sort.go:64] 	SecurityGroup/api-elb.kubernetes.prod.us-east-1.contaazul.local:	[VPC/kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372578   19569 topological_sort.go:64] 	kubernetes.prod.us-east-1.contaazul.local-addons-bootstrap:	[]
I0425 21:53:12.372588   19569 topological_sort.go:64] 	kubernetes.prod.us-east-1.contaazul.local-addons-core.addons.k8s.io:	[]
I0425 21:53:12.372598   19569 topological_sort.go:64] 	NatGateway/us-east-1e.kubernetes.prod.us-east-1.contaazul.local:	[ElasticIP/us-east-1e.kubernetes.prod.us-east-1.contaazul.local Subnet/utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local RouteTable/private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372645   19569 topological_sort.go:64] 	LoadBalancer/api.kubernetes.prod.us-east-1.contaazul.local:	[Subnet/utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local Subnet/utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local SecurityGroup/api-elb.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372660   19569 topological_sort.go:64] 	LoadBalancerAttachment/api-master-us-east-1a:	[LoadBalancer/api.kubernetes.prod.us-east-1.contaazul.local AutoscalingGroup/master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372674   19569 topological_sort.go:64] 	Route/0.0.0.0/0:	[RouteTable/kubernetes.prod.us-east-1.contaazul.local InternetGateway/kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372685   19569 topological_sort.go:64] 	IAMRolePolicy/additional.nodes.kubernetes.prod.us-east-1.contaazul.local:	[IAMRole/nodes.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372696   19569 topological_sort.go:64] 	secret/system-dns:	[]
I0425 21:53:12.372719   19569 topological_sort.go:64] 	Route/private-us-east-1a-0.0.0.0/0:	[RouteTable/private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local NatGateway/us-east-1a.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372731   19569 topological_sort.go:64] 	Subnet/us-east-1e.kubernetes.prod.us-east-1.contaazul.local:	[VPC/kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372742   19569 topological_sort.go:64] 	kubernetes.prod.us-east-1.contaazul.local-addons-limit-range.addons.k8s.io:	[]
I0425 21:53:12.372752   19569 topological_sort.go:64] 	kubernetes.prod.us-east-1.contaazul.local-addons-dns-controller.addons.k8s.io:	[]
I0425 21:53:12.372762   19569 topological_sort.go:64] 	NatGateway/us-east-1a.kubernetes.prod.us-east-1.contaazul.local:	[ElasticIP/us-east-1a.kubernetes.prod.us-east-1.contaazul.local Subnet/utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local RouteTable/private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372776   19569 topological_sort.go:64] 	InternetGateway/kubernetes.prod.us-east-1.contaazul.local:	[VPC/kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372788   19569 topological_sort.go:64] 	IAMRolePolicy/masters.kubernetes.prod.us-east-1.contaazul.local:	[IAMRole/masters.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372799   19569 topological_sort.go:64] 	IAMRolePolicy/additional.masters.kubernetes.prod.us-east-1.contaazul.local:	[IAMRole/masters.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372809   19569 topological_sort.go:64] 	secret/kube-proxy:	[]
I0425 21:53:12.372819   19569 topological_sort.go:64] 	RouteTableAssociation/private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local:	[RouteTable/private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local Subnet/us-east-1a.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372831   19569 topological_sort.go:64] 	SecurityGroupRule/all-master-to-node:	[SecurityGroup/nodes.kubernetes.prod.us-east-1.contaazul.local SecurityGroup/masters.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372874   19569 topological_sort.go:64] 	SecurityGroupRule/https-elb-to-master:	[SecurityGroup/masters.kubernetes.prod.us-east-1.contaazul.local SecurityGroup/api-elb.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372888   19569 topological_sort.go:64] 	RouteTable/private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local:	[VPC/kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372899   19569 topological_sort.go:64] 	secret/system-scheduler:	[]
I0425 21:53:12.372908   19569 topological_sort.go:64] 	ElasticIP/us-east-1e.kubernetes.prod.us-east-1.contaazul.local:	[RouteTable/private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372919   19569 topological_sort.go:64] 	secret/kubelet:	[]
I0425 21:53:12.372928   19569 topological_sort.go:64] 	SecurityGroupRule/node-egress:	[SecurityGroup/nodes.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372938   19569 topological_sort.go:64] 	SecurityGroupRule/master-egress:	[SecurityGroup/masters.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372969   19569 topological_sort.go:64] 	SecurityGroupRule/all-master-to-master:	[SecurityGroup/masters.kubernetes.prod.us-east-1.contaazul.local SecurityGroup/masters.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.372984   19569 topological_sort.go:64] 	IAMRole/masters.kubernetes.prod.us-east-1.contaazul.local:	[]
I0425 21:53:12.372994   19569 topological_sort.go:64] 	AutoscalingGroup/master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local:	[Subnet/us-east-1a.kubernetes.prod.us-east-1.contaazul.local LaunchConfiguration/master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373006   19569 topological_sort.go:64] 	IAMRole/nodes.kubernetes.prod.us-east-1.contaazul.local:	[]
I0425 21:53:12.373020   19569 topological_sort.go:64] 	EBSVolume/a.etcd-main.kubernetes.prod.us-east-1.contaazul.local:	[]
I0425 21:53:12.373030   19569 topological_sort.go:64] 	SecurityGroupRule/ssh-external-to-node-0.0.0.0/0:	[SecurityGroup/nodes.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373041   19569 topological_sort.go:64] 	secret/system-logging:	[]
I0425 21:53:12.373051   19569 topological_sort.go:64] 	Subnet/utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local:	[VPC/kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373062   19569 topological_sort.go:64] 	secret/admin:	[]
I0425 21:53:12.373072   19569 topological_sort.go:64] 	IAMInstanceProfileRole/nodes.kubernetes.prod.us-east-1.contaazul.local:	[IAMInstanceProfile/nodes.kubernetes.prod.us-east-1.contaazul.local IAMRole/nodes.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373086   19569 topological_sort.go:64] 	IAMInstanceProfileRole/masters.kubernetes.prod.us-east-1.contaazul.local:	[IAMInstanceProfile/masters.kubernetes.prod.us-east-1.contaazul.local IAMRole/masters.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373098   19569 topological_sort.go:64] 	RouteTableAssociation/private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local:	[RouteTable/private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local Subnet/us-east-1e.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373111   19569 topological_sort.go:64] 	Keypair/kubecfg:	[]
I0425 21:53:12.373124   19569 topological_sort.go:64] 	kubernetes.prod.us-east-1.contaazul.local-addons-kube-dns.addons.k8s.io:	[]
I0425 21:53:12.373134   19569 topological_sort.go:64] 	LaunchConfiguration/master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local:	[SSHKey/kubernetes.kubernetes.prod.us-east-1.contaazul.local-11:fe:d5:e4:9f:6d:77:f0:e0:10:72:06:78:33:a1:4b SecurityGroup/masters.kubernetes.prod.us-east-1.contaazul.local IAMInstanceProfile/masters.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373161   19569 topological_sort.go:64] 	SecurityGroupRule/node-to-master-udp-1-65535:	[SecurityGroup/masters.kubernetes.prod.us-east-1.contaazul.local SecurityGroup/nodes.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373174   19569 topological_sort.go:64] 	SecurityGroupRule/all-node-to-node:	[SecurityGroup/nodes.kubernetes.prod.us-east-1.contaazul.local SecurityGroup/nodes.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373181   19569 topological_sort.go:64] 	RouteTable/private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local:	[VPC/kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373188   19569 topological_sort.go:64] 	SSHKey/kubernetes.kubernetes.prod.us-east-1.contaazul.local-11:fe:d5:e4:9f:6d:77:f0:e0:10:72:06:78:33:a1:4b:	[]
I0425 21:53:12.373279   19569 topological_sort.go:64] 	SecurityGroupRule/ssh-external-to-master-0.0.0.0/0:	[SecurityGroup/masters.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373293   19569 topological_sort.go:64] 	VPC/kubernetes.prod.us-east-1.contaazul.local:	[]
I0425 21:53:12.373303   19569 topological_sort.go:64] 	secret/kube:	[]
I0425 21:53:12.373312   19569 topological_sort.go:64] 	SecurityGroupRule/api-elb-egress:	[SecurityGroup/api-elb.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373323   19569 topological_sort.go:64] 	IAMInstanceProfile/nodes.kubernetes.prod.us-east-1.contaazul.local:	[]
I0425 21:53:12.373333   19569 topological_sort.go:64] 	SecurityGroup/nodes.kubernetes.prod.us-east-1.contaazul.local:	[VPC/kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373343   19569 topological_sort.go:64] 	RouteTableAssociation/utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local:	[RouteTable/kubernetes.prod.us-east-1.contaazul.local Subnet/utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373355   19569 topological_sort.go:64] 	DNSName/api.kubernetes.prod.us-east-1.contaazul.local:	[DNSZone/ZCMPK68VV5UY9 LoadBalancer/api.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373367   19569 topological_sort.go:64] 	SecurityGroupRule/node-to-master-tcp-4003-65535:	[SecurityGroup/masters.kubernetes.prod.us-east-1.contaazul.local SecurityGroup/nodes.kubernetes.prod.us-east-1.contaazul.local]
I0425 21:53:12.373378   19569 topological_sort.go:64] 	EBSVolume/a.etcd-events.kubernetes.prod.us-east-1.contaazul.local:	[]
I0425 21:53:12.373467   19569 executor.go:91] Tasks: 0 done / 75 total; 26 can run
I0425 21:53:12.373521   19569 executor.go:157] Executing task "secret/system-dns": *fitasks.Secret {"Name":"system:dns"}
I0425 21:53:12.373674   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/secrets/system:dns"
I0425 21:53:12.373521   19569 executor.go:157] Executing task "kubernetes.prod.us-east-1.contaazul.local-addons-bootstrap": *fitasks.ManagedFile {"Name":"kubernetes.prod.us-east-1.contaazul.local-addons-bootstrap","Location":"addons/bootstrap-channel.yaml","Contents":{"Name":"","Resource":{}}}
I0425 21:53:12.373634   19569 executor.go:157] Executing task "EBSVolume/a.etcd-main.kubernetes.prod.us-east-1.contaazul.local": *awstasks.EBSVolume {"Name":"a.etcd-main.kubernetes.prod.us-east-1.contaazul.local","ID":null,"AvailabilityZone":"us-east-1a","VolumeType":"gp2","SizeGB":20,"KmsKeyId":null,"Encrypted":false,"Tags":{"k8s.io/etcd/main":"a/a","k8s.io/role/master":"1"}}
I0425 21:53:12.373742   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/addons/bootstrap-channel.yaml"
I0425 21:53:12.374240   19569 executor.go:157] Executing task "kubernetes.prod.us-east-1.contaazul.local-addons-dns-controller.addons.k8s.io": *fitasks.ManagedFile {"Name":"kubernetes.prod.us-east-1.contaazul.local-addons-dns-controller.addons.k8s.io","Location":"addons/dns-controller.addons.k8s.io/v1.5.2.yaml","Contents":{"Name":"addons/dns-controller.addons.k8s.io/v1.5.2.yaml","Resource":{}}}
I0425 21:53:12.374288   19569 executor.go:157] Executing task "secret/system-monitoring": *fitasks.Secret {"Name":"system:monitoring"}
I0425 21:53:12.374309   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/addons/dns-controller.addons.k8s.io/v1.5.2.yaml"
I0425 21:53:12.374338   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/secrets/system:monitoring"
I0425 21:53:12.374281   19569 executor.go:157] Executing task "SSHKey/kubernetes.kubernetes.prod.us-east-1.contaazul.local-11:fe:d5:e4:9f:6d:77:f0:e0:10:72:06:78:33:a1:4b": *awstasks.SSHKey {"Name":"kubernetes.kubernetes.prod.us-east-1.contaazul.local-11:fe:d5:e4:9f:6d:77:f0:e0:10:72:06:78:33:a1:4b","PublicKey":{"Name":"","Resource":{}},"KeyFingerprint":null}
I0425 21:53:12.373559   19569 executor.go:157] Executing task "Keypair/master": *fitasks.Keypair {"Name":"master","subject":"cn=kubernetes-master","type":"server","alternateNames":["kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster.local","api.kubernetes.prod.us-east-1.contaazul.local","api.internal.kubernetes.prod.us-east-1.contaazul.local","100.64.0.1"],"alternateNameTasks":null}
I0425 21:53:12.374744   19569 s3fs.go:199] Listing objects in S3 bucket "kubernetes.prod.contaazul.local" with prefix "kubernetes.prod.us-east-1.contaazul.local/pki/issued/master/"
I0425 21:53:12.374706   19569 executor.go:157] Executing task "kubernetes.prod.us-east-1.contaazul.local-addons-storage-aws.addons.k8s.io": *fitasks.ManagedFile {"Name":"kubernetes.prod.us-east-1.contaazul.local-addons-storage-aws.addons.k8s.io","Location":"addons/storage-aws.addons.k8s.io/v1.5.0.yaml","Contents":{"Name":"addons/storage-aws.addons.k8s.io/v1.5.0.yaml","Resource":{}}}
I0425 21:53:12.374798   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/addons/storage-aws.addons.k8s.io/v1.5.0.yaml"
I0425 21:53:12.374994   19569 sshkey.go:205] Computed SSH key fingerprint as "05:51:03:76:b8:4b:cd:08:00:d0:61:f4:76:8a:84:9c"
I0425 21:53:12.373779   19569 executor.go:157] Executing task "IAMRole/masters.kubernetes.prod.us-east-1.contaazul.local": *awstasks.IAMRole {"ID":null,"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","RolePolicyDocument":{"Name":"","Resource":{}}}
I0425 21:53:12.374271   19569 executor.go:157] Executing task "Keypair/kubelet": *fitasks.Keypair {"Name":"kubelet","subject":"cn=kubelet","type":"client","alternateNames":null,"alternateNameTasks":null}
I0425 21:53:12.374605   19569 executor.go:157] Executing task "Keypair/kubecfg": *fitasks.Keypair {"Name":"kubecfg","subject":"cn=kubecfg","type":"client","alternateNames":null,"alternateNameTasks":null}
I0425 21:53:12.375155   19569 s3fs.go:199] Listing objects in S3 bucket "kubernetes.prod.contaazul.local" with prefix "kubernetes.prod.us-east-1.contaazul.local/pki/issued/kubecfg/"
I0425 21:53:12.375174   19569 request_logger.go:45] AWS request: ec2/DescribeKeyPairs
I0425 21:53:12.375407   19569 request_logger.go:45] AWS request: iam/GetRole
I0425 21:53:12.373947   19569 request_logger.go:45] AWS request: ec2/DescribeVolumes
I0425 21:53:12.373907   19569 executor.go:157] Executing task "EBSVolume/a.etcd-events.kubernetes.prod.us-east-1.contaazul.local": *awstasks.EBSVolume {"Name":"a.etcd-events.kubernetes.prod.us-east-1.contaazul.local","ID":null,"AvailabilityZone":"us-east-1a","VolumeType":"gp2","SizeGB":20,"KmsKeyId":null,"Encrypted":false,"Tags":{"k8s.io/etcd/events":"a/a","k8s.io/role/master":"1"}}
I0425 21:53:12.375919   19569 request_logger.go:45] AWS request: ec2/DescribeVolumes
I0425 21:53:12.373964   19569 executor.go:157] Executing task "secret/kube": *fitasks.Secret {"Name":"kube"}
I0425 21:53:12.376094   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/secrets/kube"
I0425 21:53:12.373971   19569 executor.go:157] Executing task "secret/kube-proxy": *fitasks.Secret {"Name":"kube-proxy"}
I0425 21:53:12.376488   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/secrets/kube-proxy"
I0425 21:53:12.373987   19569 executor.go:157] Executing task "secret/system-logging": *fitasks.Secret {"Name":"system:logging"}
I0425 21:53:12.376784   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/secrets/system:logging"
I0425 21:53:12.373992   19569 executor.go:157] Executing task "VPC/kubernetes.prod.us-east-1.contaazul.local": *awstasks.VPC {"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}
I0425 21:53:12.374089   19569 executor.go:157] Executing task "kubernetes.prod.us-east-1.contaazul.local-addons-core.addons.k8s.io": *fitasks.ManagedFile {"Name":"kubernetes.prod.us-east-1.contaazul.local-addons-core.addons.k8s.io","Location":"addons/core.addons.k8s.io/v1.4.0.yaml","Contents":{"Name":"addons/core.addons.k8s.io/v1.4.0.yaml","Resource":{}}}
I0425 21:53:12.377114   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/addons/core.addons.k8s.io/v1.4.0.yaml"
I0425 21:53:12.374196   19569 executor.go:157] Executing task "secret/system-controller_manager": *fitasks.Secret {"Name":"system:controller_manager"}
I0425 21:53:12.374178   19569 executor.go:157] Executing task "IAMRole/nodes.kubernetes.prod.us-east-1.contaazul.local": *awstasks.IAMRole {"ID":null,"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","RolePolicyDocument":{"Name":"","Resource":{}}}
I0425 21:53:12.374219   19569 executor.go:157] Executing task "secret/system-scheduler": *fitasks.Secret {"Name":"system:scheduler"}
I0425 21:53:12.374007   19569 executor.go:157] Executing task "kubernetes.prod.us-east-1.contaazul.local-addons-kube-dns.addons.k8s.io": *fitasks.ManagedFile {"Name":"kubernetes.prod.us-east-1.contaazul.local-addons-kube-dns.addons.k8s.io","Location":"addons/kube-dns.addons.k8s.io/v1.5.1.yaml","Contents":{"Name":"addons/kube-dns.addons.k8s.io/v1.5.1.yaml","Resource":{}}}
I0425 21:53:12.374098   19569 executor.go:157] Executing task "IAMInstanceProfile/nodes.kubernetes.prod.us-east-1.contaazul.local": *awstasks.IAMInstanceProfile {"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":null}
I0425 21:53:12.374226   19569 executor.go:157] Executing task "kubernetes.prod.us-east-1.contaazul.local-addons-limit-range.addons.k8s.io": *fitasks.ManagedFile {"Name":"kubernetes.prod.us-east-1.contaazul.local-addons-limit-range.addons.k8s.io","Location":"addons/limit-range.addons.k8s.io/v1.5.0.yaml","Contents":{"Name":"addons/limit-range.addons.k8s.io/v1.5.0.yaml","Resource":{}}}
I0425 21:53:12.374623   19569 executor.go:157] Executing task "IAMInstanceProfile/masters.kubernetes.prod.us-east-1.contaazul.local": *awstasks.IAMInstanceProfile {"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":null}
I0425 21:53:12.374645   19569 executor.go:157] Executing task "secret/kubelet": *fitasks.Secret {"Name":"kubelet"}
I0425 21:53:12.377749   19569 request_logger.go:45] AWS request: ec2/DescribeVpcs
I0425 21:53:12.377888   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/secrets/system:controller_manager"
I0425 21:53:12.378296   19569 request_logger.go:45] AWS request: iam/GetRole
I0425 21:53:12.378417   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/secrets/system:scheduler"
I0425 21:53:12.373949   19569 executor.go:157] Executing task "secret/admin": *fitasks.Secret {"Name":"admin"}
I0425 21:53:12.378757   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/secrets/admin"
I0425 21:53:12.379207   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/addons/limit-range.addons.k8s.io/v1.5.0.yaml"
I0425 21:53:12.379412   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/addons/kube-dns.addons.k8s.io/v1.5.1.yaml"
I0425 21:53:12.379755   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/secrets/kubelet"
I0425 21:53:12.379769   19569 s3fs.go:199] Listing objects in S3 bucket "kubernetes.prod.contaazul.local" with prefix "kubernetes.prod.us-east-1.contaazul.local/pki/issued/kubelet/"
I0425 21:53:12.380365   19569 request_logger.go:45] AWS request: iam/GetInstanceProfile
I0425 21:53:12.380407   19569 request_logger.go:45] AWS request: iam/GetInstanceProfile
I0425 21:53:12.585706   19569 sshkey.go:94] SSH key fingerprints match; assuming public keys match
I0425 21:53:12.793931   19569 ebsvolume.go:91] found existing volume
I0425 21:53:12.794027   19569 changes.go:80] Field changed "Tags" actual="map[k8s.io/role/master:1 CA_SERVICE:ContaAzul k8s.io/etcd/main:a/a KubernetesCluster:kubernetes.prod.us-east-1.contaazul.local CA_TEAM:All Name:a.etcd-main.kubernetes.prod.us-east-1.contaazul.local]" expected="map[k8s.io/etcd/main:a/a k8s.io/role/master:1 Name:a.etcd-main.kubernetes.prod.us-east-1.contaazul.local KubernetesCluster:kubernetes.prod.us-east-1.contaazul.local]"
I0425 21:53:13.038947   19569 s3fs.go:225] Listed files in s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/issued/master: [s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/issued/master/6397454486979108753981432530.crt]
I0425 21:53:13.038985   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/issued/master/6397454486979108753981432530.crt"
I0425 21:53:13.041957   19569 ebsvolume.go:91] found existing volume
I0425 21:53:13.042039   19569 changes.go:80] Field changed "Tags" actual="map[CA_SERVICE:ContaAzul KubernetesCluster:kubernetes.prod.us-east-1.contaazul.local Name:a.etcd-events.kubernetes.prod.us-east-1.contaazul.local k8s.io/etcd/events:a/a k8s.io/role/master:1 CA_TEAM:All]" expected="map[Name:a.etcd-events.kubernetes.prod.us-east-1.contaazul.local KubernetesCluster:kubernetes.prod.us-east-1.contaazul.local k8s.io/etcd/events:a/a k8s.io/role/master:1]"
I0425 21:53:13.050179   19569 s3fs.go:225] Listed files in s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/issued/kubecfg: [s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/issued/kubecfg/6397454487389566686610908086.crt]
I0425 21:53:13.050209   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/issued/kubecfg/6397454487389566686610908086.crt"
I0425 21:53:13.055261   19569 s3fs.go:225] Listed files in s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/issued/kubelet: [s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/issued/kubelet/6397454486960000703716705841.crt]
I0425 21:53:13.055307   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/issued/kubelet/6397454486960000703716705841.crt"
I0425 21:53:13.083020   19569 vpc.go:79] found matching VPC *awstasks.VPC {"Name":"ContaAzul Virginia","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":null,"Shared":null}
I0425 21:53:13.083277   19569 request_logger.go:45] AWS request: ec2/DescribeVpcAttribute
I0425 21:53:13.223169   19569 ca.go:367] Parsing pem block: "CERTIFICATE"
I0425 21:53:13.223321   19569 s3fs.go:199] Listing objects in S3 bucket "kubernetes.prod.contaazul.local" with prefix "kubernetes.prod.us-east-1.contaazul.local/pki/private/kubecfg/"
I0425 21:53:13.234750   19569 ca.go:367] Parsing pem block: "CERTIFICATE"
I0425 21:53:13.235011   19569 s3fs.go:199] Listing objects in S3 bucket "kubernetes.prod.contaazul.local" with prefix "kubernetes.prod.us-east-1.contaazul.local/pki/private/kubelet/"
I0425 21:53:13.237288   19569 ca.go:367] Parsing pem block: "CERTIFICATE"
I0425 21:53:13.237676   19569 s3fs.go:199] Listing objects in S3 bucket "kubernetes.prod.contaazul.local" with prefix "kubernetes.prod.us-east-1.contaazul.local/pki/private/master/"
I0425 21:53:13.262556   19569 iamrole.go:95] actual RolePolicyDocument was json-equal to expected; returning expected value
I0425 21:53:13.262588   19569 iamrole.go:103] found matching IAMRole "AROAI2LWZ77H5JABFQXI4"
I0425 21:53:13.266832   19569 iamrole.go:95] actual RolePolicyDocument was json-equal to expected; returning expected value
I0425 21:53:13.266869   19569 iamrole.go:103] found matching IAMRole "AROAIILQUP6HVQUOM5HBI"
I0425 21:53:13.289166   19569 request_logger.go:45] AWS request: ec2/DescribeVpcAttribute
I0425 21:53:13.398233   19569 s3fs.go:225] Listed files in s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/private/kubelet: [s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/private/kubelet/6397454486960000703716705841.key]
I0425 21:53:13.398275   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/private/kubelet/6397454486960000703716705841.key"
I0425 21:53:13.403249   19569 s3fs.go:225] Listed files in s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/private/kubecfg: [s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/private/kubecfg/6397454487389566686610908086.key]
I0425 21:53:13.403286   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/private/kubecfg/6397454487389566686610908086.key"
I0425 21:53:13.412545   19569 s3fs.go:225] Listed files in s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/private/master: [s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/private/master/6397454486979108753981432530.key]
I0425 21:53:13.412576   19569 s3fs.go:162] Reading file "s3://kubernetes.prod.contaazul.local/kubernetes.prod.us-east-1.contaazul.local/pki/private/master/6397454486979108753981432530.key"
I0425 21:53:13.464701   19569 changes.go:80] Field changed "Name" actual="ContaAzul Virginia" expected="kubernetes.prod.us-east-1.contaazul.local"
I0425 21:53:13.577532   19569 ca.go:385] Parsing pem block: "RSA PRIVATE KEY"
I0425 21:53:13.619439   19569 ca.go:385] Parsing pem block: "RSA PRIVATE KEY"
I0425 21:53:13.774932   19569 ca.go:385] Parsing pem block: "RSA PRIVATE KEY"
I0425 21:53:13.775594   19569 changes.go:173] comparing slices: 0 100.64.0.1 100.64.0.1
I0425 21:53:13.775617   19569 changes.go:173] comparing slices: 1 api.internal.kubernetes.prod.us-east-1.contaazul.local api.internal.kubernetes.prod.us-east-1.contaazul.local
I0425 21:53:13.775629   19569 changes.go:173] comparing slices: 2 api.kubernetes.prod.us-east-1.contaazul.local api.kubernetes.prod.us-east-1.contaazul.local
I0425 21:53:13.775642   19569 changes.go:173] comparing slices: 3 kubernetes kubernetes
I0425 21:53:13.775655   19569 changes.go:173] comparing slices: 4 kubernetes.default kubernetes.default
I0425 21:53:13.775665   19569 changes.go:173] comparing slices: 5 kubernetes.default.svc kubernetes.default.svc
I0425 21:53:13.775711   19569 changes.go:173] comparing slices: 6 kubernetes.default.svc.cluster.local kubernetes.default.svc.cluster.local
I0425 21:53:13.775764   19569 executor.go:91] Tasks: 26 done / 75 total; 18 can run
I0425 21:53:13.775820   19569 executor.go:157] Executing task "IAMRolePolicy/masters.kubernetes.prod.us-east-1.contaazul.local": *awstasks.IAMRolePolicy {"ID":null,"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","Role":{"ID":"AROAI2LWZ77H5JABFQXI4","Name":"masters.kubernetes.prod.us-east-1.contaazul.local","RolePolicyDocument":{"Name":"","Resource":{}}},"PolicyDocument":{"Name":"","Resource":{}}}
I0425 21:53:13.775880   19569 executor.go:157] Executing task "RouteTable/private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local": *awstasks.RouteTable {"Name":"private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":null,"VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}
I0425 21:53:13.776042   19569 executor.go:157] Executing task "RouteTable/kubernetes.prod.us-east-1.contaazul.local": *awstasks.RouteTable {"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":null,"VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}
I0425 21:53:13.775988   19569 executor.go:157] Executing task "Subnet/us-east-1a.kubernetes.prod.us-east-1.contaazul.local": *awstasks.Subnet {"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":null,"VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.240.0/24","Shared":false}
I0425 21:53:13.776121   19569 executor.go:157] Executing task "IAMRolePolicy/additional.nodes.kubernetes.prod.us-east-1.contaazul.local": *awstasks.IAMRolePolicy {"ID":null,"Name":"additional.nodes.kubernetes.prod.us-east-1.contaazul.local","Role":{"ID":"AROAIILQUP6HVQUOM5HBI","Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","RolePolicyDocument":{"Name":"","Resource":{}}},"PolicyDocument":{"Name":"","Resource":{}}}
I0425 21:53:13.776745   19569 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I0425 21:53:13.776778   19569 request_logger.go:45] AWS request: ec2/DescribeSubnets
I0425 21:53:13.776869   19569 request_logger.go:45] AWS request: iam/GetRolePolicy
I0425 21:53:13.776203   19569 request_logger.go:45] AWS request: iam/GetRolePolicy
I0425 21:53:13.775820   19569 executor.go:157] Executing task "IAMInstanceProfileRole/masters.kubernetes.prod.us-east-1.contaazul.local": *awstasks.IAMInstanceProfileRole {"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","InstanceProfile":{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"AIPAJHZRGBUYXRJYOV4QI"},"Role":{"ID":"AROAI2LWZ77H5JABFQXI4","Name":"masters.kubernetes.prod.us-east-1.contaazul.local","RolePolicyDocument":{"Name":"","Resource":{}}}}
I0425 21:53:13.776230   19569 executor.go:157] Executing task "Subnet/us-east-1e.kubernetes.prod.us-east-1.contaazul.local": *awstasks.Subnet {"Name":"us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":null,"VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1e","CIDR":"10.110.241.0/24","Shared":false}
I0425 21:53:13.777168   19569 request_logger.go:45] AWS request: iam/GetInstanceProfile
I0425 21:53:13.776243   19569 executor.go:157] Executing task "IAMRolePolicy/additional.masters.kubernetes.prod.us-east-1.contaazul.local": *awstasks.IAMRolePolicy {"ID":null,"Name":"additional.masters.kubernetes.prod.us-east-1.contaazul.local","Role":{"ID":"AROAI2LWZ77H5JABFQXI4","Name":"masters.kubernetes.prod.us-east-1.contaazul.local","RolePolicyDocument":{"Name":"","Resource":{}}},"PolicyDocument":{"Name":"","Resource":{}}}
I0425 21:53:13.777302   19569 request_logger.go:45] AWS request: ec2/DescribeSubnets
I0425 21:53:13.777324   19569 request_logger.go:45] AWS request: iam/GetRolePolicy
I0425 21:53:13.776230   19569 executor.go:157] Executing task "IAMRolePolicy/nodes.kubernetes.prod.us-east-1.contaazul.local": *awstasks.IAMRolePolicy {"ID":null,"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","Role":{"ID":"AROAIILQUP6HVQUOM5HBI","Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","RolePolicyDocument":{"Name":"","Resource":{}}},"PolicyDocument":{"Name":"","Resource":{}}}
I0425 21:53:13.776308   19569 executor.go:157] Executing task "IAMInstanceProfileRole/nodes.kubernetes.prod.us-east-1.contaazul.local": *awstasks.IAMInstanceProfileRole {"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","InstanceProfile":{"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":"AIPAJRMALFINY2V5VPVKW"},"Role":{"ID":"AROAIILQUP6HVQUOM5HBI","Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","RolePolicyDocument":{"Name":"","Resource":{}}}}
I0425 21:53:13.776305   19569 executor.go:157] Executing task "Subnet/utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local": *awstasks.Subnet {"Name":"utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":null,"VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1e","CIDR":"10.110.208.0/21","Shared":false}
I0425 21:53:13.777737   19569 request_logger.go:45] AWS request: iam/GetInstanceProfile
I0425 21:53:13.777843   19569 request_logger.go:45] AWS request: ec2/DescribeSubnets
I0425 21:53:13.776277   19569 executor.go:157] Executing task "DNSZone/ZCMPK68VV5UY9": *awstasks.DNSZone {"Name":"ZCMPK68VV5UY9","DNSName":null,"ZoneID":"ZCMPK68VV5UY9","Private":true,"PrivateVPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}
I0425 21:53:13.776369   19569 executor.go:157] Executing task "Subnet/utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local": *awstasks.Subnet {"Name":"utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":null,"VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.200.0/21","Shared":false}
I0425 21:53:13.778187   19569 request_logger.go:45] AWS request: ec2/DescribeSubnets
I0425 21:53:13.776396   19569 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I0425 21:53:13.778260   19569 request_logger.go:45] AWS request: route53/GetHostedZone
I0425 21:53:13.776398   19569 executor.go:157] Executing task "RouteTable/private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local": *awstasks.RouteTable {"Name":"private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":null,"VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}
I0425 21:53:13.775857   19569 executor.go:157] Executing task "SecurityGroup/api-elb.kubernetes.prod.us-east-1.contaazul.local": *awstasks.SecurityGroup {"Name":"api-elb.kubernetes.prod.us-east-1.contaazul.local","ID":null,"Description":"Security group for api ELB","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=443"],"Shared":null}
I0425 21:53:13.776208   19569 executor.go:157] Executing task "SecurityGroup/nodes.kubernetes.prod.us-east-1.contaazul.local": *awstasks.SecurityGroup {"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":null,"Description":"Security group for nodes","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22"],"Shared":null}
I0425 21:53:13.778724   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:13.775960   19569 executor.go:157] Executing task "SecurityGroup/masters.kubernetes.prod.us-east-1.contaazul.local": *awstasks.SecurityGroup {"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":null,"Description":"Security group for masters","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null}
I0425 21:53:13.776379   19569 executor.go:157] Executing task "InternetGateway/kubernetes.prod.us-east-1.contaazul.local": *awstasks.InternetGateway {"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":null,"VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"Shared":true}
I0425 21:53:13.779131   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:13.779314   19569 request_logger.go:45] AWS request: ec2/DescribeInternetGateways
I0425 21:53:13.777560   19569 request_logger.go:45] AWS request: iam/GetRolePolicy
I0425 21:53:13.778459   19569 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I0425 21:53:13.778926   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:14.009441   19569 subnet.go:75] found matching subnet "subnet-552ec01d"
I0425 21:53:14.065359   19569 routetable.go:59] found matching RouteTable "rtb-a06104d9"
I0425 21:53:14.191404   19569 subnet.go:75] found matching subnet "subnet-45cdad79"
I0425 21:53:14.260471   19569 subnet.go:75] found matching subnet "subnet-44cdad78"
I0425 21:53:14.386695   19569 subnet.go:75] found matching subnet "subnet-542ec01c"
I0425 21:53:14.461207   19569 routetable.go:59] found matching RouteTable "rtb-a16104d8"
I0425 21:53:14.461453   19569 routetable.go:59] found matching RouteTable "rtb-a36104da"
I0425 21:53:14.481782   19569 internetgateway.go:92] found matching InternetGateway "igw-5bb56b3f"
I0425 21:53:14.481837   19569 changes.go:80] Field changed "Name" actual="<nil>" expected="kubernetes.prod.us-east-1.contaazul.local"
I0425 21:53:14.485597   19569 securitygroup.go:78] found matching SecurityGroup "sg-888f45f7"
I0425 21:53:14.485683   19569 changes.go:173] comparing slices: 0 port=22 port=22
I0425 21:53:14.485880   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:14.495314   19569 securitygroup.go:78] found matching SecurityGroup "sg-898f45f6"
I0425 21:53:14.495355   19569 changes.go:173] comparing slices: 0 port=22 port=22
I0425 21:53:14.495370   19569 changes.go:173] comparing slices: 1 port=443 port=443
I0425 21:53:14.495380   19569 changes.go:173] comparing slices: 2 port=4001 port=4001
I0425 21:53:14.495388   19569 changes.go:173] comparing slices: 3 port=4789 port=4789
I0425 21:53:14.495398   19569 changes.go:173] comparing slices: 4 port=179 port=179
I0425 21:53:14.495607   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:14.504010   19569 securitygroup.go:78] found matching SecurityGroup "sg-878f45f8"
I0425 21:53:14.504041   19569 changes.go:173] comparing slices: 0 port=443 port=443
I0425 21:53:14.504239   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:14.729651   19569 securitygroup.go:403] permission matches rule {"Port":443}: {
  FromPort: 443,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "52.1.24.103/32"
    }],
  ToPort: 443
}
I0425 21:53:14.732610   19569 securitygroup.go:403] permission matches rule {"Port":443}: {
  FromPort: 443,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "189.58.105.66/32"
    }],
  ToPort: 443
}
I0425 21:53:14.732671   19569 securitygroup.go:403] permission matches rule {"Port":443}: {
  FromPort: 443,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "177.67.91.106/32"
    }],
  ToPort: 443
}
I0425 21:53:14.732703   19569 securitygroup.go:403] permission matches rule {"Port":443}: {
  FromPort: 443,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "52.205.71.191/32"
    }],
  ToPort: 443
}
I0425 21:53:14.732732   19569 securitygroup.go:403] permission matches rule {"Port":443}: {
  FromPort: 443,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "52.205.109.226/32"
    }],
  ToPort: 443
}
I0425 21:53:14.732757   19569 securitygroup.go:403] permission matches rule {"Port":443}: {
  FromPort: 443,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "54.207.25.102/32"
    }],
  ToPort: 443
}
I0425 21:53:14.732782   19569 securitygroup.go:403] permission matches rule {"Port":443}: {
  FromPort: 443,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "52.206.42.157/32"
    }],
  ToPort: 443
}
I0425 21:53:15.070767   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-08487c77\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.070821   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-0913a276\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.070853   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-3b0da444\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.070890   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-4943ea36\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.070926   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-888f45f7\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.070983   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-898f45f6\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071021   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-8e1dadf1\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071057   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-c0e93fbf\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071095   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-d77cc8a8\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071134   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-d7ef39a8\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071171   19569 securitygroup.go:403] permission matches rule {"Port":22}: {
  FromPort: 22,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0"
    }],
  ToPort: 22
}
I0425 21:53:15.071236   19569 securitygroup.go:409] Ignoring security group permission "{\n  FromPort: 1,\n  IpProtocol: \"tcp\",\n  ToPort: 4000,\n  UserIdGroupPairs: [{\n      GroupId: \"sg-888f45f7\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071307   19569 securitygroup.go:409] Ignoring security group permission "{\n  FromPort: 4003,\n  IpProtocol: \"tcp\",\n  ToPort: 65535,\n  UserIdGroupPairs: [{\n      GroupId: \"sg-888f45f7\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071361   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-08487c77\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071402   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-0913a276\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071441   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-3b0da444\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071482   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-4943ea36\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071521   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-898f45f6\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071560   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-8e1dadf1\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071599   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-c0e93fbf\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071637   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-d77cc8a8\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071677   19569 securitygroup.go:409] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-d7ef39a8\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071713   19569 securitygroup.go:403] permission matches rule {"Port":22}: {
  FromPort: 22,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0"
    }],
  ToPort: 22
}
I0425 21:53:15.071761   19569 securitygroup.go:409] Ignoring security group permission "{\n  FromPort: 1,\n  IpProtocol: \"udp\",\n  ToPort: 65535,\n  UserIdGroupPairs: [{\n      GroupId: \"sg-888f45f7\",\n      UserId: \"858038232233\"\n    }]\n}" (did not match removal rules)
I0425 21:53:15.071804   19569 securitygroup.go:403] permission matches rule {"Port":443}: {
  FromPort: 443,
  IpProtocol: "tcp",
  ToPort: 443,
  UserIdGroupPairs: [{
      GroupId: "sg-878f45f8",
      UserId: "858038232233"
    }]
}
I0425 21:53:15.071885   19569 executor.go:91] Tasks: 44 done / 75 total; 23 can run
I0425 21:53:15.071938   19569 executor.go:157] Executing task "SecurityGroupRule/all-node-to-node": *awstasks.SecurityGroupRule {"Name":"all-node-to-node","SecurityGroup":{"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-888f45f7","Description":"Security group for nodes","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22"],"Shared":null},"CIDR":null,"Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":{"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-888f45f7","Description":"Security group for nodes","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22"],"Shared":null},"Egress":null}
I0425 21:53:15.072122   19569 executor.go:157] Executing task "SecurityGroupRule/ssh-external-to-master-0.0.0.0/0": *awstasks.SecurityGroupRule {"Name":"ssh-external-to-master-0.0.0.0/0","SecurityGroup":{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-898f45f6","Description":"Security group for masters","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"CIDR":"0.0.0.0/0","Protocol":"tcp","FromPort":22,"ToPort":22,"SourceGroup":null,"Egress":null}
I0425 21:53:15.072351   19569 executor.go:157] Executing task "LaunchConfiguration/master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local": *awstasks.LaunchConfiguration {"Name":"master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local","UserData":{"Name":"","Resource":{}},"ImageID":"kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09","InstanceType":"m3.medium","SSHKey":{"Name":"kubernetes.kubernetes.prod.us-east-1.contaazul.local-11:fe:d5:e4:9f:6d:77:f0:e0:10:72:06:78:33:a1:4b","PublicKey":{"Name":"","Resource":{}},"KeyFingerprint":"05:51:03:76:b8:4b:cd:08:00:d0:61:f4:76:8a:84:9c"},"SecurityGroups":[{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-898f45f6","Description":"Security group for masters","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null}],"AssociatePublicIP":false,"IAMInstanceProfile":{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"AIPAJHZRGBUYXRJYOV4QI"},"RootVolumeSize":20,"RootVolumeType":"gp2","SpotPrice":"","ID":null}
I0425 21:53:15.072529   19569 executor.go:157] Executing task "SecurityGroupRule/node-egress": *awstasks.SecurityGroupRule {"Name":"node-egress","SecurityGroup":{"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-888f45f7","Description":"Security group for nodes","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22"],"Shared":null},"CIDR":"0.0.0.0/0","Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":null,"Egress":true}
I0425 21:53:15.072849   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:15.072459   19569 executor.go:157] Executing task "LoadBalancer/api.kubernetes.prod.us-east-1.contaazul.local": *awstasks.LoadBalancer {"Name":"api.kubernetes.prod.us-east-1.contaazul.local","LoadBalancerName":"api-kubernetes-prod-us-ea-pa2kem","DNSName":null,"HostedZoneId":null,"Subnets":[{"Name":"utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-542ec01c","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.200.0/21","Shared":false},{"Name":"utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-44cdad78","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1e","CIDR":"10.110.208.0/21","Shared":false}],"SecurityGroups":[{"Name":"api-elb.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-878f45f8","Description":"Security group for api ELB","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=443"],"Shared":null}],"Listeners":{"443":{"InstancePort":443}},"Scheme":null,"HealthCheck":{"Target":"TCP:443","HealthyThreshold":2,"UnhealthyThreshold":2,"Interval":10,"Timeout":5},"AccessLog":null,"ConnectionDraining":null,"ConnectionSettings":{"IdleTimeout":300},"CrossZoneLoadBalancing":null}
I0425 21:53:15.073194   19569 executor.go:157] Executing task "SecurityGroupRule/https-elb-to-master": *awstasks.SecurityGroupRule {"Name":"https-elb-to-master","SecurityGroup":{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-898f45f6","Description":"Security group for masters","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"CIDR":null,"Protocol":"tcp","FromPort":443,"ToPort":443,"SourceGroup":{"Name":"api-elb.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-878f45f8","Description":"Security group for api ELB","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=443"],"Shared":null},"Egress":null}
I0425 21:53:15.072051   19569 executor.go:157] Executing task "RouteTableAssociation/private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local": *awstasks.RouteTableAssociation {"Name":"private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":null,"RouteTable":{"Name":"private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a36104da","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}},"Subnet":{"Name":"us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-45cdad79","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1e","CIDR":"10.110.241.0/24","Shared":false}}
I0425 21:53:15.073480   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:15.073600   19569 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I0425 21:53:15.072373   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:15.072031   19569 executor.go:157] Executing task "SecurityGroupRule/master-egress": *awstasks.SecurityGroupRule {"Name":"master-egress","SecurityGroup":{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-898f45f6","Description":"Security group for masters","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"CIDR":"0.0.0.0/0","Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":null,"Egress":true}
I0425 21:53:15.072378   19569 executor.go:157] Executing task "SecurityGroupRule/all-master-to-master": *awstasks.SecurityGroupRule {"Name":"all-master-to-master","SecurityGroup":{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-898f45f6","Description":"Security group for masters","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"CIDR":null,"Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-898f45f6","Description":"Security group for masters","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"Egress":null}
I0425 21:53:15.073946   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:15.072461   19569 executor.go:157] Executing task "SecurityGroupRule/api-elb-egress": *awstasks.SecurityGroupRule {"Name":"api-elb-egress","SecurityGroup":{"Name":"api-elb.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-878f45f8","Description":"Security group for api ELB","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=443"],"Shared":null},"CIDR":"0.0.0.0/0","Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":null,"Egress":true}
I0425 21:53:15.074194   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:15.074208   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:15.072909   19569 request_logger.go:45] AWS request: autoscaling/DescribeLaunchConfigurations
I0425 21:53:15.072211   19569 executor.go:157] Executing task "ElasticIP/us-east-1e.kubernetes.prod.us-east-1.contaazul.local": *awstasks.ElasticIP {"Name":"us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":null,"PublicIP":null,"TagOnSubnet":null,"AssociatedNatGatewayRouteTable":{"Name":"private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a36104da","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}}
I0425 21:53:15.072926   19569 executor.go:157] Executing task "SecurityGroupRule/node-to-master-tcp-4003-65535": *awstasks.SecurityGroupRule {"Name":"node-to-master-tcp-4003-65535","SecurityGroup":{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-898f45f6","Description":"Security group for masters","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"CIDR":null,"Protocol":"tcp","FromPort":4003,"ToPort":65535,"SourceGroup":{"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-888f45f7","Description":"Security group for nodes","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22"],"Shared":null},"Egress":null}
I0425 21:53:15.073019   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:15.071992   19569 executor.go:157] Executing task "Route/0.0.0.0/0": *awstasks.Route {"Name":"0.0.0.0/0","RouteTable":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a06104d9","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}},"Instance":null,"CIDR":"0.0.0.0/0","InternetGateway":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"igw-5bb56b3f","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"Shared":true},"NatGateway":null}
I0425 21:53:15.074762   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:15.074933   19569 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I0425 21:53:15.073024   19569 executor.go:157] Executing task "SecurityGroupRule/node-to-master-udp-1-65535": *awstasks.SecurityGroupRule {"Name":"node-to-master-udp-1-65535","SecurityGroup":{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-898f45f6","Description":"Security group for masters","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"CIDR":null,"Protocol":"udp","FromPort":1,"ToPort":65535,"SourceGroup":{"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-888f45f7","Description":"Security group for nodes","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22"],"Shared":null},"Egress":null}
I0425 21:53:15.073041   19569 executor.go:157] Executing task "LaunchConfiguration/nodes.kubernetes.prod.us-east-1.contaazul.local": *awstasks.LaunchConfiguration {"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","UserData":{"Name":"","Resource":{}},"ImageID":"kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09","InstanceType":"t2.xlarge","SSHKey":{"Name":"kubernetes.kubernetes.prod.us-east-1.contaazul.local-11:fe:d5:e4:9f:6d:77:f0:e0:10:72:06:78:33:a1:4b","PublicKey":{"Name":"","Resource":{}},"KeyFingerprint":"05:51:03:76:b8:4b:cd:08:00:d0:61:f4:76:8a:84:9c"},"SecurityGroups":[{"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-888f45f7","Description":"Security group for nodes","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22"],"Shared":null}],"AssociatePublicIP":false,"IAMInstanceProfile":{"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":"AIPAJRMALFINY2V5VPVKW"},"RootVolumeSize":100,"RootVolumeType":"gp2","SpotPrice":"","ID":null}
I0425 21:53:15.073118   19569 executor.go:157] Executing task "RouteTableAssociation/utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local": *awstasks.RouteTableAssociation {"Name":"utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":null,"RouteTable":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a06104d9","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}},"Subnet":{"Name":"utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-542ec01c","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.200.0/21","Shared":false}}
I0425 21:53:15.075957   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:15.076307   19569 request_logger.go:45] AWS request: autoscaling/DescribeLaunchConfigurations
I0425 21:53:15.076644   19569 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I0425 21:53:15.073140   19569 executor.go:157] Executing task "RouteTableAssociation/private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local": *awstasks.RouteTableAssociation {"Name":"private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":null,"RouteTable":{"Name":"private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a16104d8","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}},"Subnet":{"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-552ec01d","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.240.0/24","Shared":false}}
I0425 21:53:15.073191   19569 executor.go:157] Executing task "SecurityGroupRule/ssh-external-to-node-0.0.0.0/0": *awstasks.SecurityGroupRule {"Name":"ssh-external-to-node-0.0.0.0/0","SecurityGroup":{"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-888f45f7","Description":"Security group for nodes","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22"],"Shared":null},"CIDR":"0.0.0.0/0","Protocol":"tcp","FromPort":22,"ToPort":22,"SourceGroup":null,"Egress":null}
I0425 21:53:15.077586   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:15.071950   19569 executor.go:157] Executing task "SecurityGroupRule/all-master-to-node": *awstasks.SecurityGroupRule {"Name":"all-master-to-node","SecurityGroup":{"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-888f45f7","Description":"Security group for nodes","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22"],"Shared":null},"CIDR":null,"Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-898f45f6","Description":"Security group for masters","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"Egress":null}
I0425 21:53:15.078654   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:15.073254   19569 executor.go:157] Executing task "RouteTableAssociation/utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local": *awstasks.RouteTableAssociation {"Name":"utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":null,"RouteTable":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a06104d9","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}},"Subnet":{"Name":"utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-44cdad78","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1e","CIDR":"10.110.208.0/21","Shared":false}}
I0425 21:53:15.079271   19569 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I0425 21:53:15.072294   19569 executor.go:157] Executing task "ElasticIP/us-east-1a.kubernetes.prod.us-east-1.contaazul.local": *awstasks.ElasticIP {"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":null,"PublicIP":null,"TagOnSubnet":null,"AssociatedNatGatewayRouteTable":{"Name":"private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a16104d8","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}}
I0425 21:53:15.079544   19569 natgateway.go:193] trying to match NatGateway via RouteTable %!s(*string=0xc420d0da08)
I0425 21:53:15.073318   19569 load_balancer.go:173] Listing all ELBs for findLoadBalancerByNameTag
I0425 21:53:15.079787   19569 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I0425 21:53:15.074464   19569 natgateway.go:193] trying to match NatGateway via RouteTable %!s(*string=0xc420ca7168)
I0425 21:53:15.080591   19569 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I0425 21:53:15.072998   19569 executor.go:157] Executing task "SecurityGroupRule/https-api-elb-0.0.0.0/0": *awstasks.SecurityGroupRule {"Name":"https-api-elb-0.0.0.0/0","SecurityGroup":{"Name":"api-elb.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-878f45f8","Description":"Security group for api ELB","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=443"],"Shared":null},"CIDR":"0.0.0.0/0","Protocol":"tcp","FromPort":443,"ToPort":443,"SourceGroup":null,"Egress":null}
I0425 21:53:15.077363   19569 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I0425 21:53:15.071973   19569 executor.go:157] Executing task "SecurityGroupRule/node-to-master-tcp-1-4000": *awstasks.SecurityGroupRule {"Name":"node-to-master-tcp-1-4000","SecurityGroup":{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-898f45f6","Description":"Security group for masters","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"CIDR":null,"Protocol":"tcp","FromPort":1,"ToPort":4000,"SourceGroup":{"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-888f45f7","Description":"Security group for nodes","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22"],"Shared":null},"Egress":null}
I0425 21:53:15.080404   19569 request_logger.go:45] AWS request: elasticloadbalancing/DescribeLoadBalancers
I0425 21:53:15.081292   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:15.081098   19569 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0425 21:53:15.784777   19569 routetableassociation.go:79] found matching RouteTableAssociation "rtbassoc-96f4c7ee"
I0425 21:53:15.784935   19569 routetableassociation.go:79] found matching RouteTableAssociation "rtbassoc-95f4c7ed"
I0425 21:53:15.786888   19569 natgateway.go:208] no NatGateway found in route table rtb-a16104d8
I0425 21:53:15.786909   19569 elastic_ip.go:88] AssociatedNatGatewayRouteTable not found
I0425 21:53:15.790885   19569 routetableassociation.go:79] found matching RouteTableAssociation "rtbassoc-94f4c7ec"
I0425 21:53:15.791132   19569 route.go:98] found route matching cidr 0.0.0.0/0
I0425 21:53:15.794993   19569 natgateway.go:208] no NatGateway found in route table rtb-a36104da
I0425 21:53:15.795013   19569 elastic_ip.go:88] AssociatedNatGatewayRouteTable not found
I0425 21:53:15.797232   19569 routetableassociation.go:79] found matching RouteTableAssociation "rtbassoc-88f4c7f0"
I0425 21:53:15.990644   19569 launchconfiguration.go:102] found existing AutoscalingLaunchConfiguration: "master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local-20170314203052"
I0425 21:53:15.990737   19569 aws_cloud.go:581] Calling DescribeImages to resolve name "kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09"
I0425 21:53:15.990955   19569 request_logger.go:45] AWS request: ec2/DescribeImages
I0425 21:53:15.995410   19569 launchconfiguration.go:102] found existing AutoscalingLaunchConfiguration: "nodes.kubernetes.prod.us-east-1.contaazul.local-20170314203052"
I0425 21:53:15.995481   19569 aws_cloud.go:581] Calling DescribeImages to resolve name "kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09"
I0425 21:53:15.995674   19569 request_logger.go:45] AWS request: ec2/DescribeImages
I0425 21:53:16.017623   19569 load_balancer.go:258] Querying ELB tags for [Fortknox-ELB api-kubernetes-prod-us-ea-pa2kem a93d391bf09b611e78d8f0a00310c692 a86abcf6909b711e78d8f0a00310c692 a277e2ad80e5911e78d8f0a00310c692 a60c2f1e50e6211e78d8f0a00310c692 a3684d6550f3b11e78d8f0a00310c692 a6ff814220fce11e78d8f0a00310c692 a2fe8c9ec0ff211e78d8f0a00310c692 a1a19570a25f311e78d8f0a00310c692]
I0425 21:53:16.017889   19569 request_logger.go:45] AWS request: elasticloadbalancing/DescribeTags
I0425 21:53:16.208970   19569 request_logger.go:45] AWS request: elasticloadbalancing/DescribeLoadBalancerAttributes
I0425 21:53:16.221374   19569 aws_cloud.go:626] Resolved image "ami-5f1afc49"
I0425 21:53:16.221405   19569 launchconfiguration.go:147] Returning matching ImageId as expected name: "ami-5f1afc49" -> "kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09"
I0425 21:53:16.221491   19569 urls.go:85] Using default protokube location: "https://kubeupv2.s3.amazonaws.com/kops/1.5.3/images/protokube.tar.gz"
I0425 21:53:16.221532   19569 context.go:126] Performing HTTP request: GET https://kubeupv2.s3.amazonaws.com/kops/1.5.3/images/protokube.tar.gz.sha1
I0425 21:53:16.238279   19569 aws_cloud.go:626] Resolved image "ami-5f1afc49"
I0425 21:53:16.238310   19569 launchconfiguration.go:147] Returning matching ImageId as expected name: "ami-5f1afc49" -> "kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09"
I0425 21:53:16.238399   19569 context.go:126] Performing HTTP request: GET https://kubeupv2.s3.amazonaws.com/kops/1.5.3/images/protokube.tar.gz.sha1
I0425 21:53:16.379253   19569 apply_cluster.go:705] Found hash "1d73d1816cd7492ce1e5a8eb7346a302982b0db2" for "https://kubeupv2.s3.amazonaws.com/kops/1.5.3/images/protokube.tar.gz"
I0425 21:53:16.381703   19569 changes.go:80] Field changed "InstanceType" actual="c4.2xlarge" expected="t2.xlarge"
I0425 21:53:16.381751   19569 changes.go:173] comparing slices: 0 *awstasks.SecurityGroup {"Name":null,"ID":"sg-888f45f7","Description":null,"VPC":null,"RemoveExtraRules":null,"Shared":null} *awstasks.SecurityGroup {"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-888f45f7","Description":"Security group for nodes","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22"],"Shared":null}
I0425 21:53:16.381838   19569 changes.go:80] Field changed "RootVolumeSize" actual="200" expected="100"
I0425 21:53:16.383508   19569 changes.go:173] comparing slices: 0 *awstasks.Subnet {"Name":"utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-44cdad78","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1e","CIDR":"10.110.208.0/21","Shared":false} *awstasks.Subnet {"Name":"utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-44cdad78","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1e","CIDR":"10.110.208.0/21","Shared":false}
I0425 21:53:16.383583   19569 changes.go:173] comparing slices: 1 *awstasks.Subnet {"Name":"utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-542ec01c","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.200.0/21","Shared":false} *awstasks.Subnet {"Name":"utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-542ec01c","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.200.0/21","Shared":false}
I0425 21:53:16.383630   19569 changes.go:173] comparing slices: 0 *awstasks.SecurityGroup {"Name":null,"ID":"sg-878f45f8","Description":null,"VPC":null,"RemoveExtraRules":null,"Shared":null} *awstasks.SecurityGroup {"Name":"api-elb.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-878f45f8","Description":"Security group for api ELB","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=443"],"Shared":null}
I0425 21:53:16.383720   19569 changes.go:148] comparing maps: 443 &{443} &{443}
I0425 21:53:16.548681   19569 apply_cluster.go:705] Found hash "1d73d1816cd7492ce1e5a8eb7346a302982b0db2" for "https://kubeupv2.s3.amazonaws.com/kops/1.5.3/images/protokube.tar.gz"
I0425 21:53:16.549603   19569 changes.go:173] comparing slices: 0 *awstasks.SecurityGroup {"Name":null,"ID":"sg-898f45f6","Description":null,"VPC":null,"RemoveExtraRules":null,"Shared":null} *awstasks.SecurityGroup {"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-898f45f6","Description":"Security group for masters","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null}
I0425 21:53:16.549694   19569 executor.go:91] Tasks: 67 done / 75 total; 5 can run
I0425 21:53:16.549781   19569 executor.go:157] Executing task "NatGateway/us-east-1a.kubernetes.prod.us-east-1.contaazul.local": *awstasks.NatGateway {"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ElasticIP":{"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":null,"PublicIP":null,"TagOnSubnet":null,"AssociatedNatGatewayRouteTable":{"Name":"private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a16104d8","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}},"Subnet":{"Name":"utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-542ec01c","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.200.0/21","Shared":false},"ID":null,"EgressId":null,"Shared":null,"AssociatedRouteTable":{"Name":"private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a16104d8","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}}
I0425 21:53:16.549744   19569 executor.go:157] Executing task "AutoscalingGroup/master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local": *awstasks.AutoscalingGroup {"Name":"master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local","MinSize":1,"MaxSize":1,"Subnets":[{"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-552ec01d","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.240.0/24","Shared":false}],"Tags":{"k8s.io/role/master":"1"},"LaunchConfiguration":{"Name":"master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local","UserData":{"Name":"","Resource":{}},"ImageID":"kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09","InstanceType":"m3.medium","SSHKey":{"Name":"kubernetes.kubernetes.prod.us-east-1.contaazul.local-11:fe:d5:e4:9f:6d:77:f0:e0:10:72:06:78:33:a1:4b","PublicKey":{"Name":"","Resource":{}},"KeyFingerprint":"05:51:03:76:b8:4b:cd:08:00:d0:61:f4:76:8a:84:9c"},"SecurityGroups":[{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-898f45f6","Description":"Security group for masters","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null}],"AssociatePublicIP":false,"IAMInstanceProfile":{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"AIPAJHZRGBUYXRJYOV4QI"},"RootVolumeSize":20,"RootVolumeType":"gp2","SpotPrice":"","ID":"master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local-20170314203052"}}
I0425 21:53:16.549925   19569 executor.go:157] Executing task "NatGateway/us-east-1e.kubernetes.prod.us-east-1.contaazul.local": *awstasks.NatGateway {"Name":"us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ElasticIP":{"Name":"us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":null,"PublicIP":null,"TagOnSubnet":null,"AssociatedNatGatewayRouteTable":{"Name":"private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a36104da","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}},"Subnet":{"Name":"utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-44cdad78","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1e","CIDR":"10.110.208.0/21","Shared":false},"ID":null,"EgressId":null,"Shared":null,"AssociatedRouteTable":{"Name":"private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a36104da","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}}
I0425 21:53:16.550445   19569 natgateway.go:193] trying to match NatGateway via RouteTable %!s(*string=0xc420ca7168)
I0425 21:53:16.549877   19569 executor.go:157] Executing task "DNSName/api.kubernetes.prod.us-east-1.contaazul.local": *awstasks.DNSName {"Name":"api.kubernetes.prod.us-east-1.contaazul.local","ID":null,"Zone":{"Name":"ZCMPK68VV5UY9","DNSName":"kubernetes.prod.us-east-1.contaazul.local","ZoneID":"ZCMPK68VV5UY9","Private":true,"PrivateVPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}},"ResourceType":"A","TargetLoadBalancer":{"Name":"api.kubernetes.prod.us-east-1.contaazul.local","LoadBalancerName":"api-kubernetes-prod-us-ea-pa2kem","DNSName":"api-kubernetes-prod-us-ea-pa2kem-1155968168.us-east-1.elb.amazonaws.com","HostedZoneId":"Z35SXDOTRQ7X7K","Subnets":[{"Name":"utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-44cdad78","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1e","CIDR":"10.110.208.0/21","Shared":false},{"Name":"utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-542ec01c","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.200.0/21","Shared":false}],"SecurityGroups":[{"Name":"api-elb.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-878f45f8","Description":"Security group for api ELB","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=443"],"Shared":null}],"Listeners":{"443":{"InstancePort":443}},"Scheme":null,"HealthCheck":{"Target":"TCP:443","HealthyThreshold":2,"UnhealthyThreshold":2,"Interval":10,"Timeout":5},"AccessLog":null,"ConnectionDraining":null,"ConnectionSettings":{"IdleTimeout":300},"CrossZoneLoadBalancing":null}}
I0425 21:53:16.549748   19569 executor.go:157] Executing task "AutoscalingGroup/nodes.kubernetes.prod.us-east-1.contaazul.local": *awstasks.AutoscalingGroup {"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","MinSize":2,"MaxSize":6,"Subnets":[{"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-552ec01d","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.240.0/24","Shared":false},{"Name":"us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-45cdad79","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1e","CIDR":"10.110.241.0/24","Shared":false}],"Tags":{"k8s.io/role/node":"1"},"LaunchConfiguration":{"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","UserData":{"Name":"","Resource":{}},"ImageID":"kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09","InstanceType":"t2.xlarge","SSHKey":{"Name":"kubernetes.kubernetes.prod.us-east-1.contaazul.local-11:fe:d5:e4:9f:6d:77:f0:e0:10:72:06:78:33:a1:4b","PublicKey":{"Name":"","Resource":{}},"KeyFingerprint":"05:51:03:76:b8:4b:cd:08:00:d0:61:f4:76:8a:84:9c"},"SecurityGroups":[{"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-888f45f7","Description":"Security group for nodes","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22"],"Shared":null}],"AssociatePublicIP":false,"IAMInstanceProfile":{"Name":"nodes.kubernetes.prod.us-east-1.contaazul.local","ID":"AIPAJRMALFINY2V5VPVKW"},"RootVolumeSize":100,"RootVolumeType":"gp2","SpotPrice":"","ID":"nodes.kubernetes.prod.us-east-1.contaazul.local-20170314203052"}}
I0425 21:53:16.550181   19569 natgateway.go:193] trying to match NatGateway via RouteTable %!s(*string=0xc420d0da08)
I0425 21:53:16.550973   19569 request_logger.go:45] AWS request: autoscaling/DescribeAutoScalingGroups
I0425 21:53:16.550614   19569 request_logger.go:45] AWS request: autoscaling/DescribeAutoScalingGroups
I0425 21:53:16.550652   19569 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I0425 21:53:16.550959   19569 request_logger.go:45] AWS request: route53/ListResourceRecordSets
I0425 21:53:16.551097   19569 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I0425 21:53:16.730714   19569 dnsname.go:74] Found DNS resource "NS" "kubernetes.prod.us-east-1.contaazul.local."
I0425 21:53:16.730744   19569 dnsname.go:74] Found DNS resource "SOA" "kubernetes.prod.us-east-1.contaazul.local."
I0425 21:53:16.730751   19569 dnsname.go:74] Found DNS resource "A" "api.kubernetes.prod.us-east-1.contaazul.local."
I0425 21:53:16.730766   19569 dnsname.go:108] AliasTarget for "api.kubernetes.prod.us-east-1.contaazul.local." is "api-kubernetes-prod-us-ea-pa2kem-1155968168.us-east-1.elb.amazonaws.com."
I0425 21:53:16.730963   19569 request_logger.go:45] AWS request: elasticloadbalancing/DescribeLoadBalancers
I0425 21:53:16.736680   19569 natgateway.go:208] no NatGateway found in route table rtb-a16104d8
I0425 21:53:16.736900   19569 request_logger.go:45] AWS request: ec2/DescribeTags
I0425 21:53:16.741632   19569 changes.go:173] comparing slices: 0 *awstasks.Subnet {"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-552ec01d","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.240.0/24","Shared":false} *awstasks.Subnet {"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-552ec01d","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.240.0/24","Shared":false}
I0425 21:53:16.741692   19569 changes.go:173] comparing slices: 1 *awstasks.Subnet {"Name":"us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-45cdad79","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1e","CIDR":"10.110.241.0/24","Shared":false} *awstasks.Subnet {"Name":"us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-45cdad79","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1e","CIDR":"10.110.241.0/24","Shared":false}
I0425 21:53:16.741760   19569 changes.go:148] comparing maps: KubernetesCluster kubernetes.prod.us-east-1.contaazul.local kubernetes.prod.us-east-1.contaazul.local
I0425 21:53:16.741792   19569 changes.go:148] comparing maps: Name nodes.kubernetes.prod.us-east-1.contaazul.local nodes.kubernetes.prod.us-east-1.contaazul.local
I0425 21:53:16.741810   19569 changes.go:148] comparing maps: k8s.io/role/node 1 1
I0425 21:53:16.745460   19569 changes.go:173] comparing slices: 0 *awstasks.Subnet {"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-552ec01d","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.240.0/24","Shared":false} *awstasks.Subnet {"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-552ec01d","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.240.0/24","Shared":false}
I0425 21:53:16.745517   19569 changes.go:148] comparing maps: KubernetesCluster kubernetes.prod.us-east-1.contaazul.local kubernetes.prod.us-east-1.contaazul.local
I0425 21:53:16.745532   19569 changes.go:148] comparing maps: Name master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local
I0425 21:53:16.745598   19569 changes.go:148] comparing maps: k8s.io/role/master 1 1
I0425 21:53:16.745664   19569 natgateway.go:208] no NatGateway found in route table rtb-a36104da
I0425 21:53:16.745840   19569 request_logger.go:45] AWS request: ec2/DescribeTags
I0425 21:53:16.914288   19569 natgateway.go:162] Found NatGateway via subnet tag: nat-04bf1ab2fe2fd6bf4
I0425 21:53:16.914533   19569 request_logger.go:45] AWS request: ec2/DescribeNatGateways
I0425 21:53:16.924414   19569 natgateway.go:162] Found NatGateway via subnet tag: nat-029ff47dd592ca997
I0425 21:53:16.924587   19569 request_logger.go:45] AWS request: ec2/DescribeNatGateways
I0425 21:53:16.943184   19569 load_balancer.go:258] Querying ELB tags for [api-kubernetes-prod-us-ea-pa2kem]
I0425 21:53:16.943348   19569 request_logger.go:45] AWS request: elasticloadbalancing/DescribeTags
I0425 21:53:17.127240   19569 changes.go:80] Field changed "ElasticIP" actual="{<nil> 0xc4207be688 <nil> *awstasks.Subnet null *awstasks.RouteTable null}" expected="{0xc4208b23f0 <nil> <nil> *awstasks.Subnet null *awstasks.RouteTable {\"Name\":\"private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local\",\"ID\":\"rtb-a16104d8\",\"VPC\":{\"Name\":\"kubernetes.prod.us-east-1.contaazul.local\",\"ID\":\"vpc-66673502\",\"CIDR\":\"10.110.0.0/16\",\"EnableDNSHostnames\":null,\"EnableDNSSupport\":true,\"Shared\":true}}}"
I0425 21:53:17.144860   19569 changes.go:80] Field changed "ElasticIP" actual="{<nil> 0xc4207bf4c8 <nil> *awstasks.Subnet null *awstasks.RouteTable null}" expected="{0xc4208b24b0 <nil> <nil> *awstasks.Subnet null *awstasks.RouteTable {\"Name\":\"private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local\",\"ID\":\"rtb-a36104da\",\"VPC\":{\"Name\":\"kubernetes.prod.us-east-1.contaazul.local\",\"ID\":\"vpc-66673502\",\"CIDR\":\"10.110.0.0/16\",\"EnableDNSHostnames\":null,\"EnableDNSSupport\":true,\"Shared\":true}}}"
W0425 21:53:17.144959   19569 executor.go:109] error running task "NatGateway/us-east-1a.kubernetes.prod.us-east-1.contaazul.local" (9m59s remaining to succeed): Field cannot be changed: ElasticIp
W0425 21:53:17.144989   19569 executor.go:109] error running task "NatGateway/us-east-1e.kubernetes.prod.us-east-1.contaazul.local" (9m59s remaining to succeed): Field cannot be changed: ElasticIp
I0425 21:53:17.145046   19569 executor.go:91] Tasks: 70 done / 75 total; 3 can run
I0425 21:53:17.145104   19569 executor.go:157] Executing task "NatGateway/us-east-1a.kubernetes.prod.us-east-1.contaazul.local": *awstasks.NatGateway {"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ElasticIP":{"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":null,"PublicIP":null,"TagOnSubnet":null,"AssociatedNatGatewayRouteTable":{"Name":"private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a16104d8","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}},"Subnet":{"Name":"utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-542ec01c","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.200.0/21","Shared":false},"ID":"nat-04bf1ab2fe2fd6bf4","EgressId":null,"Shared":null,"AssociatedRouteTable":{"Name":"private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a16104d8","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}}
I0425 21:53:17.145098   19569 executor.go:157] Executing task "NatGateway/us-east-1e.kubernetes.prod.us-east-1.contaazul.local": *awstasks.NatGateway {"Name":"us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ElasticIP":{"Name":"us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":null,"PublicIP":null,"TagOnSubnet":null,"AssociatedNatGatewayRouteTable":{"Name":"private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a36104da","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}},"Subnet":{"Name":"utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-44cdad78","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1e","CIDR":"10.110.208.0/21","Shared":false},"ID":"nat-029ff47dd592ca997","EgressId":null,"Shared":null,"AssociatedRouteTable":{"Name":"private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a36104da","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}}
I0425 21:53:17.145190   19569 executor.go:157] Executing task "LoadBalancerAttachment/api-master-us-east-1a": *awstasks.LoadBalancerAttachment {"Name":"api-master-us-east-1a","LoadBalancer":{"Name":"api.kubernetes.prod.us-east-1.contaazul.local","LoadBalancerName":"api-kubernetes-prod-us-ea-pa2kem","DNSName":"api-kubernetes-prod-us-ea-pa2kem-1155968168.us-east-1.elb.amazonaws.com","HostedZoneId":"Z35SXDOTRQ7X7K","Subnets":[{"Name":"utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-44cdad78","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1e","CIDR":"10.110.208.0/21","Shared":false},{"Name":"utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-542ec01c","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.200.0/21","Shared":false}],"SecurityGroups":[{"Name":"api-elb.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-878f45f8","Description":"Security group for api ELB","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=443"],"Shared":null}],"Listeners":{"443":{"InstancePort":443}},"Scheme":null,"HealthCheck":{"Target":"TCP:443","HealthyThreshold":2,"UnhealthyThreshold":2,"Interval":10,"Timeout":5},"AccessLog":null,"ConnectionDraining":null,"ConnectionSettings":{"IdleTimeout":300},"CrossZoneLoadBalancing":null},"AutoscalingGroup":{"Name":"master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local","MinSize":1,"MaxSize":1,"Subnets":[{"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-552ec01d","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.240.0/24","Shared":false}],"Tags":{"KubernetesCluster":"kubernetes.prod.us-east-1.contaazul.local","Name":"master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local","k8s.io/role/master":"1"},"LaunchConfiguration":{"Name":"master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local","UserData":{"Name":"","Resource":{}},"ImageID":"kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09","InstanceType":"m3.medium","SSHKey":{"Name":"kubernetes.kubernetes.prod.us-east-1.contaazul.local-11:fe:d5:e4:9f:6d:77:f0:e0:10:72:06:78:33:a1:4b","PublicKey":{"Name":"","Resource":{}},"KeyFingerprint":"05:51:03:76:b8:4b:cd:08:00:d0:61:f4:76:8a:84:9c"},"SecurityGroups":[{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"sg-898f45f6","Description":"Security group for masters","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null}],"AssociatePublicIP":false,"IAMInstanceProfile":{"Name":"masters.kubernetes.prod.us-east-1.contaazul.local","ID":"AIPAJHZRGBUYXRJYOV4QI"},"RootVolumeSize":20,"RootVolumeType":"gp2","SpotPrice":"","ID":"master-us-east-1a.masters.kubernetes.prod.us-east-1.contaazul.local-20170314203052"}},"Subnet":null,"Instance":null}
I0425 21:53:17.145481   19569 request_logger.go:45] AWS request: ec2/DescribeNatGateways
I0425 21:53:17.145700   19569 request_logger.go:45] AWS request: ec2/DescribeNatGateways
I0425 21:53:17.145951   19569 request_logger.go:45] AWS request: autoscaling/DescribeAutoScalingGroups
I0425 21:53:17.358818   19569 changes.go:80] Field changed "ElasticIP" actual="{<nil> 0xc420b85fa8 <nil> *awstasks.Subnet null *awstasks.RouteTable null}" expected="{0xc4208b24b0 <nil> <nil> *awstasks.Subnet null *awstasks.RouteTable {\"Name\":\"private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local\",\"ID\":\"rtb-a36104da\",\"VPC\":{\"Name\":\"kubernetes.prod.us-east-1.contaazul.local\",\"ID\":\"vpc-66673502\",\"CIDR\":\"10.110.0.0/16\",\"EnableDNSHostnames\":null,\"EnableDNSSupport\":true,\"Shared\":true}}}"
I0425 21:53:17.460799   19569 changes.go:80] Field changed "ElasticIP" actual="{<nil> 0xc4202acd48 <nil> *awstasks.Subnet null *awstasks.RouteTable null}" expected="{0xc4208b23f0 <nil> <nil> *awstasks.Subnet null *awstasks.RouteTable {\"Name\":\"private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local\",\"ID\":\"rtb-a16104d8\",\"VPC\":{\"Name\":\"kubernetes.prod.us-east-1.contaazul.local\",\"ID\":\"vpc-66673502\",\"CIDR\":\"10.110.0.0/16\",\"EnableDNSHostnames\":null,\"EnableDNSSupport\":true,\"Shared\":true}}}"
W0425 21:53:17.460914   19569 executor.go:109] error running task "NatGateway/us-east-1a.kubernetes.prod.us-east-1.contaazul.local" (9m59s remaining to succeed): Field cannot be changed: ElasticIp
W0425 21:53:17.460940   19569 executor.go:109] error running task "NatGateway/us-east-1e.kubernetes.prod.us-east-1.contaazul.local" (9m59s remaining to succeed): Field cannot be changed: ElasticIp
I0425 21:53:17.460984   19569 executor.go:91] Tasks: 71 done / 75 total; 2 can run
I0425 21:53:17.461028   19569 executor.go:157] Executing task "NatGateway/us-east-1e.kubernetes.prod.us-east-1.contaazul.local": *awstasks.NatGateway {"Name":"us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ElasticIP":{"Name":"us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":null,"PublicIP":null,"TagOnSubnet":null,"AssociatedNatGatewayRouteTable":{"Name":"private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a36104da","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}},"Subnet":{"Name":"utility-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-44cdad78","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1e","CIDR":"10.110.208.0/21","Shared":false},"ID":"nat-029ff47dd592ca997","EgressId":null,"Shared":null,"AssociatedRouteTable":{"Name":"private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a36104da","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}}
I0425 21:53:17.461040   19569 executor.go:157] Executing task "NatGateway/us-east-1a.kubernetes.prod.us-east-1.contaazul.local": *awstasks.NatGateway {"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ElasticIP":{"Name":"us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":null,"PublicIP":null,"TagOnSubnet":null,"AssociatedNatGatewayRouteTable":{"Name":"private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a16104d8","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}},"Subnet":{"Name":"utility-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"subnet-542ec01c","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true},"AvailabilityZone":"us-east-1a","CIDR":"10.110.200.0/21","Shared":false},"ID":"nat-04bf1ab2fe2fd6bf4","EgressId":null,"Shared":null,"AssociatedRouteTable":{"Name":"private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local","ID":"rtb-a16104d8","VPC":{"Name":"kubernetes.prod.us-east-1.contaazul.local","ID":"vpc-66673502","CIDR":"10.110.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true}}}
I0425 21:53:17.461401   19569 request_logger.go:45] AWS request: ec2/DescribeNatGateways
I0425 21:53:17.461454   19569 request_logger.go:45] AWS request: ec2/DescribeNatGateways
I0425 21:53:17.663976   19569 changes.go:80] Field changed "ElasticIP" actual="{<nil> 0xc420e0fac8 <nil> *awstasks.Subnet null *awstasks.RouteTable null}" expected="{0xc4208b23f0 <nil> <nil> *awstasks.Subnet null *awstasks.RouteTable {\"Name\":\"private-us-east-1a.kubernetes.prod.us-east-1.contaazul.local\",\"ID\":\"rtb-a16104d8\",\"VPC\":{\"Name\":\"kubernetes.prod.us-east-1.contaazul.local\",\"ID\":\"vpc-66673502\",\"CIDR\":\"10.110.0.0/16\",\"EnableDNSHostnames\":null,\"EnableDNSSupport\":true,\"Shared\":true}}}"
I0425 21:53:17.668491   19569 changes.go:80] Field changed "ElasticIP" actual="{<nil> 0xc420d34868 <nil> *awstasks.Subnet null *awstasks.RouteTable null}" expected="{0xc4208b24b0 <nil> <nil> *awstasks.Subnet null *awstasks.RouteTable {\"Name\":\"private-us-east-1e.kubernetes.prod.us-east-1.contaazul.local\",\"ID\":\"rtb-a36104da\",\"VPC\":{\"Name\":\"kubernetes.prod.us-east-1.contaazul.local\",\"ID\":\"vpc-66673502\",\"CIDR\":\"10.110.0.0/16\",\"EnableDNSHostnames\":null,\"EnableDNSSupport\":true,\"Shared\":true}}}"
W0425 21:53:17.668581   19569 executor.go:109] error running task "NatGateway/us-east-1a.kubernetes.prod.us-east-1.contaazul.local" (9m58s remaining to succeed): Field cannot be changed: ElasticIp
W0425 21:53:17.668607   19569 executor.go:109] error running task "NatGateway/us-east-1e.kubernetes.prod.us-east-1.contaazul.local" (9m58s remaining to succeed): Field cannot be changed: ElasticIp
I0425 21:53:17.668622   19569 executor.go:124] No progress made, sleeping before retrying 2 failed task(s)

@caarlos0
Copy link
Contributor

@caarlos0 caarlos0 commented Apr 26, 2017

The issue seems to be:

I0425 21:53:15.786888   19569 natgateway.go:208] no NatGateway found in route table rtb-a16104d8
I0425 21:53:15.786909   19569 elastic_ip.go:88] AssociatedNatGatewayRouteTable not found
I0425 21:53:15.790885   19569 routetableassociation.go:79] found matching RouteTableAssociation "rtbassoc-94f4c7ec"
I0425 21:53:15.791132   19569 route.go:98] found route matching cidr 0.0.0.0/0
I0425 21:53:15.794993   19569 natgateway.go:208] no NatGateway found in route table rtb-a36104da
I0425 21:53:15.795013   19569 elastic_ip.go:88] AssociatedNatGatewayRouteTable not found

@caarlos0
Copy link
Contributor

@caarlos0 caarlos0 commented Apr 26, 2017

rtb-a36104da e rtb-a16104d8 indeed don't have any nat gateways, should they?

@hollowimage
Copy link
Author

@hollowimage hollowimage commented Apr 27, 2017

I arrived to a similar conclusion @caarlos0 .

In our setup I take out the NGWs from the route tables as we use an in-home egress.

May i suggest that if there's no NGWs after initial setup, kops simply move on after throwing a warning?

@hollowimage
Copy link
Author

@hollowimage hollowimage commented Apr 28, 2017

So, I did an experiment and re-attached the NGWs to the private subnet routes instead of our custom egress, and that resolved the issue.

It seems if you are using a custom egress, kops does some hard level check. This isnt a huge issue, but would be nice to be able to tell kops to "ignore" networking egress changes after its been setup once

@caarlos0
Copy link
Contributor

@caarlos0 caarlos0 commented Apr 28, 2017

In my case, I found out that the nat_monitor removed the NGWs, will re-add and try to update the cluster on monday.

@fejta-bot
Copy link

@fejta-bot fejta-bot commented Dec 23, 2017

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 23, 2017
@fejta-bot
Copy link

@fejta-bot fejta-bot commented Jan 22, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 22, 2018
@erkolson
Copy link

@erkolson erkolson commented Jan 30, 2018

Seeing this now too. I would be better if kops issued a warning about the route tables rather than erroring. In my case, I am purposely routing outgoing connections through a traffic monitoring device.

@erkolson
Copy link

@erkolson erkolson commented Jan 30, 2018

/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jan 30, 2018
@erkolson
Copy link

@erkolson erkolson commented Jan 30, 2018

/remove-lifecycle stale

@enyamada
Copy link

@enyamada enyamada commented Mar 4, 2018

There are AWS availability zones where Nat gateways are simply not available (for example, sa-east-1b and sa-east-1c).

So, IMHO, kops should NOT enforce us to use nat gateways (rather than custom egress).

@fejta-bot
Copy link

@fejta-bot fejta-bot commented Jun 2, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 2, 2018
@fejta-bot
Copy link

@fejta-bot fejta-bot commented Jul 2, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 2, 2018
@fejta-bot
Copy link

@fejta-bot fejta-bot commented Aug 1, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@imsky
Copy link

@imsky imsky commented Apr 8, 2019

/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Apr 8, 2019
@k8s-ci-robot
Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Apr 8, 2019

@imsky: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@imsky
Copy link

@imsky imsky commented Apr 8, 2019

@chrislovecnm @justinsb we're seeing the same issue on our clusters provisioned with kops 1.10 after we modify the route table to replace the 0.0.0.0/0 route

@imsky
Copy link

@imsky imsky commented Aug 19, 2019

@chrislovecnm @justinsb following up

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants