Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request with "x-ambassador-test-allow: probably" is 200 OK #1144

Open
marselester opened this Issue Jan 24, 2019 · 10 comments

Comments

Projects
None yet
3 participants
@marselester
Copy link

marselester commented Jan 24, 2019

Describe the bug
I followed the rate limiting tutorial using minikube, but I don't see 429 status codes as expected:

$ curl -i -H "x-ambassador-test-allow: probably" $AMBASSADORURL/qotm/quote/1
HTTP/1.1 200 OK
content-type: application/json
content-length: 147
server: envoy
date: Thu, 24 Jan 2019 09:30:47 GMT
x-envoy-upstream-service-time: 65

{"hostname":"qotm-7c6cccd985-4s8kg","ok":true,"quote":"A late night does not make any sense.","time":"2019-01-24T09:30:47.705604","version":"1.3"}

To Reproduce
Create a new Kubernetes cluster and install Ambassador quay.io/datawire/ambassador:0.50.0-rc5.

$ minikube delete
$ minikube start
Starting local Kubernetes v1.13.2 cluster...
$ kubectl apply -f https://www.getambassador.io/yaml/ambassador/ambassador-rbac.yaml
$ kubectl apply -f - <<<'
apiVersion: v1
kind: Service
metadata:
  name: ambassador
spec:
  selector:
    service: ambassador
  type: NodePort
  ports:
    - port: 80
  # Propagate the original source IP of the client.
  externalTrafficPolicy: Local
'
$ AMBASSADORURL=$(minikube service --url ambassador)

Then deploy a qotm service:

$ kubectl apply -f - <<<'
---
apiVersion: v1
kind: Service
metadata:
  name: qotm
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v0
      kind:  Mapping
      name:  qotm_mapping
      prefix: /qotm/
      service: qotm
spec:
  selector:
    app: qotm
  ports:
  - port: 80
    name: http-qotm
    targetPort: http-api
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: qotm
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: qotm
    spec:
      containers:
      - name: qotm
        image: datawire/qotm:1.2
        ports:
        - name: http-api
          containerPort: 5000
        readinessProbe:
          httpGet:
            path: /health
            port: 5000
          initialDelaySeconds: 30
          periodSeconds: 3
        resources:
          limits:
            cpu: "0.1"
            memory: 100Mi
'

and ratelimit service:

$ kubectl apply -f - <<<'
---
apiVersion: v1
kind: Service
metadata:
  name: example-rate-limit
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v0
      kind: RateLimitService
      name: ratelimit
      service: "example-rate-limit:5000"
spec:
  type: ClusterIP
  selector:
    app: example-rate-limit
  ports:
  - port: 5000
    name: http-example-rate-limit
    targetPort: http-api
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: example-rate-limit
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: example-rate-limit
    spec:
      containers:
      - name: example-rate-limit
        image: agervais/ambassador-ratelimit-service:1.0.0
        imagePullPolicy: Always
        ports:
        - name: http-api
          containerPort: 5000
        resources:
          limits:
            cpu: "0.1"
            memory: 100Mi
'

Finally, add rate_limits to qotm service:

$ kubectl apply -f - <<<'
---
apiVersion: v1
kind: Service
metadata:
  name: qotm
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v0
      kind: Mapping
      name: qotm_mapping
      prefix: /qotm/
      service: qotm
      rate_limits:
        - descriptor: A test case
          headers:
            - "x-ambassador-test-allow"
spec:
  type: ClusterIP
  selector:
    app: qotm
  ports:
  - port: 80
    name: http-qotm
    targetPort: http-api
'

Expected behavior
I expect 429 status code as described in the tutorial.

Versions (please complete the following information):

  • Ambassador: 0.50.0-rc5
  • Kubernetes environment: Minikube v0.33.1
  • Version: v1.13.2
@marselester

This comment has been minimized.

Copy link
Author

marselester commented Jan 28, 2019

I followed the updated tutorials, still no luck with rate limiting.

Deploying Ambassador to Kubernetes

$ minikube delete
Deleting local Kubernetes cluster...
Machine deleted.

$ minikube start
Starting local Kubernetes v1.13.2 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Stopping extra container runtimes...
Starting cluster components...
Verifying kubelet health ...
Verifying apiserver health ...
Kubectl is now configured to use the cluster.
Loading cached images from config file.


Everything looks great. Please enjoy minikube!

$ kubectl cluster-info dump --namespace kube-system | grep authorization-mode
                            "--authorization-mode=Node,RBAC",
                            "--authorization-mode=Node,RBAC",

$ kubectl apply -f https://getambassador.io/yaml/ambassador/ambassador-rbac.yaml
$ kubectl scale --replicas=1 deployments ambassador
deployment.extensions/ambassador scaled
$ kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
ambassador-7ddf64cdb-zcm8t   1/1     Running   0          116s

$ kubectl apply -f - <<<'
apiVersion: v1
kind: Service
metadata:
  name: ambassador
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
   - port: 80
  selector:
    service: ambassador
'

$ kubectl apply -f - <<<'
apiVersion: v1
kind: Service
metadata:
  name: qotm
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v0
      kind:  Mapping
      name:  qotm_mapping
      prefix: /qotm/
      service: qotm
spec:
  selector:
    app: qotm
  ports:
  - port: 80
    name: http-qotm
    targetPort: http-api
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: qotm
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: qotm
    spec:
      containers:
      - name: qotm
        image: datawire/qotm:1.2
        ports:
        - name: http-api
          containerPort: 5000
        readinessProbe:
          httpGet:
            path: /health
            port: 5000
          initialDelaySeconds: 30
          periodSeconds: 3
        resources:
          limits:
            cpu: "0.1"
            memory: 100Mi
'

$ AMBASSADORURL=$(minikube service --url ambassador)
$ curl ${AMBASSADORURL}/qotm/
{"hostname":"qotm-7c6cccd985-fbq6t","ok":true,"quote":"A small mercy is nothing at all?","time":"2019-01-28T15:37:19.018596","version":"1.3"}

Rate Limiting

$ kubectl apply -f - <<<'
---
apiVersion: v1
kind: Service
metadata:
  name: example-rate-limit
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v0
      kind: RateLimitService
      name: ratelimit
      service: "example-rate-limit:5000"
spec:
  type: ClusterIP
  selector:
    app: example-rate-limit
  ports:
  - port: 5000
    name: http-example-rate-limit
    targetPort: http-api
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: example-rate-limit
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: example-rate-limit
    spec:
      containers:
      - name: example-rate-limit
        image: agervais/ambassador-ratelimit-service:1.0.0
        imagePullPolicy: Always
        ports:
        - name: http-api
          containerPort: 5000
        resources:
          limits:
            cpu: "0.1"
            memory: 100Mi
'
$ kubectl get pods
NAME                                  READY   STATUS    RESTARTS   AGE
ambassador-7ddf64cdb-zcm8t            1/1     Running   0          13m
example-rate-limit-7bbb4d6b66-q8x5m   1/1     Running   0          57s
qotm-7c6cccd985-fbq6t                 1/1     Running   0          6m54s

$ kubectl apply -f - <<<'
apiVersion: v1
kind: Service
metadata:
  name: qotm
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind:  Mapping
      name:  qotm_mapping
      prefix: /qotm/
      service: qotm
      labels:
        ambassador:
          - request_label_group:
            - x-ambassador-test-allow:
                header: "x-ambassador-test-allow"
                omit_if_not_present: true
spec:
  type: ClusterIP
  selector:
    app: qotm
  ports:
  - port: 80
    name: http-qotm
    targetPort: http-api
'
$ curl -v -H "x-ambassador-test-allow: probably" $AMBASSADORURL/qotm/quote/1
*   Trying 192.168.99.100...
* TCP_NODELAY set
* Connected to 192.168.99.100 (192.168.99.100) port 31786 (#0)
> GET /qotm/quote/1 HTTP/1.1
> Host: 192.168.99.100:31786
> User-Agent: curl/7.54.0
> Accept: */*
> x-ambassador-test-allow: probably
> 
< HTTP/1.1 200 OK
< content-type: application/json
< content-length: 147
< server: envoy
< date: Mon, 28 Jan 2019 15:42:56 GMT
< x-envoy-upstream-service-time: 9
< 
{"hostname":"qotm-7c6cccd985-fbq6t","ok":true,"quote":"A late night does not make any sense.","time":"2019-01-28T15:42:56.857318","version":"1.3"}
* Connection #0 to host 192.168.99.100 left intact

$ kubectl logs example-rate-limit-7bbb4d6b66-q8x5m

> ratelimit-service@0.0.1 start /app
> node server.js

Listening on GRPC port 5000

Ambassador Logs

Note, there is ERROR: couldn't read namespace default? (403) in the logs.

$ kubectl logs ambassador-7ddf64cdb-zcm8t
2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 INFO: kubewatch starting: mode 'sync' ambassador_config_dir '/ambassador/empty-sync-dir' envoy_config_file '/ambassador/envoy/envoy.json' debug 'True' delay '1.0' pid 'None'
2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 DEBUG: Merging config inputs from /ambassador/empty-sync-dir
2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 INFO: namespace default, watching all namespaces
2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 DEBUG: starting run
2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 DEBUG: connected to Kubernetes!
2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 DEBUG: looking up ID for namespace default
2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 ERROR: couldn't read namespace default? (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 28 Jan 2019 15:29:24 GMT', 'Content-Length': '331'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"default\" is forbidden: User \"system:serviceaccount:default:ambassador\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"name":"default","kind":"namespaces"},"code":403}


2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 DEBUG: cluster ID URL is d6e_id://00000000-0000-0000-0000-000000000000/default
2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 INFO: cluster ID is 07eb43c8-1166-5145-a060-45e4dd907e10 (from hardcoded ID)
2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 DEBUG: sync attempting to list services for all namespaces
2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 DEBUG: sync found 3 services
2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 DEBUG: update_from_svc: key ambassador-admin-default.yaml, config null
...

2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 DEBUG: delete: dropping key ambassador-admin-default.yaml
2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 DEBUG: update_from_svc: key kubernetes-default.yaml, config null
...

2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 DEBUG: delete: dropping key kubernetes-default.yaml
2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 DEBUG: update_from_svc: key kube-dns-kube-system.yaml, config null
...

2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 DEBUG: delete: dropping key kube-dns-kube-system.yaml
2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 DEBUG: Generating initial Envoy config
2019-01-28 15:29:24 kubewatch [23 TMainThread] 0.50.0 INFO: generating config with gencount 1 (0 changes)
2019-01-28 15:29:27 kubewatch [23 TMainThread] 0.50.0 DEBUG: scout result {
    "application": "ambassador",
    "cached": false,
    "latest_version": "0.40.2",
    "timestamp": 1548689367.15198
}
2019-01-28 15:29:27 kubewatch [23 TMainThread] 0.50.0 DEBUG: Rewrote bootstrap to /ambassador/bootstrap-ads.json
2019-01-28 15:29:27 kubewatch [23 TMainThread] 0.50.0 DEBUG: Wrote configuration 1 to /ambassador/envoy/envoy.json
2019-01-28 15:29:27 kubewatch [23 TMainThread] 0.50.0 DEBUG: 10-second watch loop delay
AMBASSADOR: using cluster ID 07eb43c8-1166-5145-a060-45e4dd907e10
AMBASSADOR: starting diagd
AMBASSADOR: starting ads
AMBASSADOR: starting Envoy
+ pids='30:diagd 31:ambex 32:envoy 33:kubewatch'
+ echo 'AMBASSADOR: waiting'
+ echo 'PIDS: 30:diagd 31:ambex 32:envoy 33:kubewatch'
+ wait
AMBASSADOR: waiting
PIDS: 30:diagd 31:ambex 32:envoy 33:kubewatch
time="2019-01-28T15:29:37Z" level=info msg="Ambex 0.1.1 starting..."
+ /ambassador/kubewatch --root /ambassador/ambassador-config --sync 'ambassador splitconfig --debug --k8s --bootstrap-path=/ambassador/bootstrap-ads.json --ads-path=/ambassador/envoy/envoy.json --ambex-pid=31' --warmup-delay 10s secrets services
time="2019-01-28T15:29:37Z" level=info msg=Listening port=18000
time="2019-01-28T15:29:37Z" level=info msg="Wrote PID" file=ambex.pid pid=31
time="2019-01-28T15:29:37Z" level=info msg="Loaded file /ambassador/envoy/envoy.json"
time="2019-01-28T15:29:37Z" level=info msg="Pushing snapshot v0"
[2019-01-28 15:29:37.385][000032][info][main] [source/server/server.cc:203] initializing epoch 0 (hot restart version=10.200.16384.127.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=2654312)
[2019-01-28 15:29:37.385][000032][info][main] [source/server/server.cc:205] statically linked extensions:
[2019-01-28 15:29:37.385][000032][info][main] [source/server/server.cc:207]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2019-01-28 15:29:37.386][000032][info][main] [source/server/server.cc:210]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2019-01-28 15:29:37.386][000032][info][main] [source/server/server.cc:213]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2019-01-28 15:29:37.386][000032][info][main] [source/server/server.cc:216]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.dubbo_proxy,envoy.filters.network.rbac,envoy.filters.network.sni_cluster,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2019-01-28 15:29:37.386][000032][info][main] [source/server/server.cc:218]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2019-01-28 15:29:37.386][000032][info][main] [source/server/server.cc:220]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.tracers.datadog,envoy.zipkin
[2019-01-28 15:29:37.386][000032][info][main] [source/server/server.cc:223]   transport_sockets.downstream: envoy.transport_sockets.alts,envoy.transport_sockets.capture,raw_buffer,tls
[2019-01-28 15:29:37.386][000032][info][main] [source/server/server.cc:226]   transport_sockets.upstream: envoy.transport_sockets.alts,envoy.transport_sockets.capture,raw_buffer,tls
[2019-01-28 15:29:37.394][000032][info][main] [source/server/server.cc:268] admin address: 127.0.0.1:8001
[2019-01-28 15:29:37.397][000032][info][config] [source/server/configuration_impl.cc:50] loading 0 static secret(s)
[2019-01-28 15:29:37.397][000032][info][config] [source/server/configuration_impl.cc:56] loading 1 cluster(s)
[2019-01-28 15:29:37.400][000032][info][upstream] [source/common/upstream/cluster_manager_impl.cc:132] cm init: initializing cds
[2019-01-28 15:29:37.401][000032][info][config] [source/server/configuration_impl.cc:61] loading 0 listener(s)
[2019-01-28 15:29:37.401][000032][info][config] [source/server/configuration_impl.cc:94] loading tracing configuration
[2019-01-28 15:29:37.401][000032][info][config] [source/server/configuration_impl.cc:112] loading stats sink configuration
[2019-01-28 15:29:37.401][000032][info][main] [source/server/server.cc:456] starting main dispatch loop
[2019-01-28 15:29:37.406][000032][info][upstream] [source/common/upstream/cluster_manager_impl.cc:496] add/update cluster cluster_127_0_0_1_8877 during init
[2019-01-28 15:29:37.406][000032][info][upstream] [source/common/upstream/cluster_manager_impl.cc:136] cm init: all clusters initialized
[2019-01-28 15:29:37.406][000032][info][main] [source/server/server.cc:428] all clusters initialized. initializing init manager
[2019-01-28 15:29:37.458][000032][info][upstream] [source/server/lds_api.cc:80] lds: add/update listener 'ambassador-listener-80'
[2019-01-28 15:29:37.459][000032][info][config] [source/server/listener_manager_impl.cc:908] all dependencies initialized. starting workers
2019-01-28 15:29:38 diagd 0.50.0 [P30TMainThread] INFO: thread count 5, listening on 0.0.0.0:8877
[2019-01-28 15:29:38 +0000] [30] [INFO] Starting gunicorn 19.9.0
[2019-01-28 15:29:38 +0000] [30] [INFO] Listening at: http://0.0.0.0:8877 (30)
[2019-01-28 15:29:38 +0000] [30] [INFO] Using worker: threads
[2019-01-28 15:29:38 +0000] [60] [INFO] Booting worker with pid: 60
2019-01-28 15:29:38 diagd 0.50.0 [P60TMainThread] INFO: Starting periodic updates
2019/01/28 15:29:47 writing secrets/bootstrap-token-n0jv64.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/bootstrap-token-n0jv64.yaml
2019/01/28 15:29:47 writing secrets/clusterrole-aggregation-controller-token-sczvf.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/clusterrole-aggregation-controller-token-sczvf.yaml
2019/01/28 15:29:47 writing secrets/expand-controller-token-55f87.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/expand-controller-token-55f87.yaml
2019/01/28 15:29:47 writing secrets/horizontal-pod-autoscaler-token-fpqdq.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/horizontal-pod-autoscaler-token-fpqdq.yaml
2019/01/28 15:29:47 writing secrets/namespace-controller-token-6fb2s.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/namespace-controller-token-6fb2s.yaml
2019/01/28 15:29:47 writing secrets/pvc-protection-controller-token-wzkdt.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/pvc-protection-controller-token-wzkdt.yaml
2019/01/28 15:29:47 writing secrets/storage-provisioner-token-6dtwc.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/storage-provisioner-token-6dtwc.yaml
2019/01/28 15:29:47 writing secrets/default-token-pl6rp.default to /ambassador/ambassador-config/sync-1/default/secrets/default-token-pl6rp.yaml
2019/01/28 15:29:47 writing secrets/job-controller-token-4f2lm.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/job-controller-token-4f2lm.yaml
2019/01/28 15:29:47 writing secrets/pod-garbage-collector-token-dx69b.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/pod-garbage-collector-token-dx69b.yaml
2019/01/28 15:29:47 writing secrets/disruption-controller-token-x7tt2.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/disruption-controller-token-x7tt2.yaml
2019/01/28 15:29:47 writing secrets/token-cleaner-token-jwnwf.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/token-cleaner-token-jwnwf.yaml
2019/01/28 15:29:47 writing secrets/service-account-controller-token-nv669.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/service-account-controller-token-nv669.yaml
2019/01/28 15:29:47 writing secrets/generic-garbage-collector-token-wh8nb.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/generic-garbage-collector-token-wh8nb.yaml
2019/01/28 15:29:47 writing secrets/persistent-volume-binder-token-vchn2.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/persistent-volume-binder-token-vchn2.yaml
2019/01/28 15:29:47 writing secrets/replicaset-controller-token-wkcj5.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/replicaset-controller-token-wkcj5.yaml
2019/01/28 15:29:47 writing secrets/replication-controller-token-mr6cw.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/replication-controller-token-mr6cw.yaml
2019/01/28 15:29:47 writing secrets/ambassador-token-95w6z.default to /ambassador/ambassador-config/sync-1/default/secrets/ambassador-token-95w6z.yaml
2019/01/28 15:29:47 writing secrets/attachdetach-controller-token-69nxd.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/attachdetach-controller-token-69nxd.yaml
2019/01/28 15:29:47 writing secrets/bootstrap-signer-token-z49nr.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/bootstrap-signer-token-z49nr.yaml
2019/01/28 15:29:47 writing secrets/kube-proxy-token-8tf6q.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/kube-proxy-token-8tf6q.yaml
2019/01/28 15:29:47 writing secrets/node-controller-token-gcbhb.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/node-controller-token-gcbhb.yaml
2019/01/28 15:29:47 writing secrets/pv-protection-controller-token-l4hxx.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/pv-protection-controller-token-l4hxx.yaml
2019/01/28 15:29:47 writing secrets/ttl-controller-token-k8j8x.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/ttl-controller-token-k8j8x.yaml
2019/01/28 15:29:47 writing secrets/default-token-hm9mw.kube-public to /ambassador/ambassador-config/sync-1/kube-public/secrets/default-token-hm9mw.yaml
2019/01/28 15:29:47 writing secrets/statefulset-controller-token-zpcjv.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/statefulset-controller-token-zpcjv.yaml
2019/01/28 15:29:47 writing secrets/daemon-set-controller-token-2t2km.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/daemon-set-controller-token-2t2km.yaml
2019/01/28 15:29:47 writing secrets/endpoint-controller-token-lcfpv.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/endpoint-controller-token-lcfpv.yaml
2019/01/28 15:29:47 writing secrets/resourcequota-controller-token-pxc4q.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/resourcequota-controller-token-pxc4q.yaml
2019/01/28 15:29:47 writing secrets/service-controller-token-4vksh.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/service-controller-token-4vksh.yaml
2019/01/28 15:29:47 writing secrets/coredns-token-nk2vb.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/coredns-token-nk2vb.yaml
2019/01/28 15:29:47 writing secrets/cronjob-controller-token-sg577.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/cronjob-controller-token-sg577.yaml
2019/01/28 15:29:47 writing secrets/default-token-jqcfm.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/default-token-jqcfm.yaml
2019/01/28 15:29:47 writing secrets/deployment-controller-token-q2jhz.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/deployment-controller-token-q2jhz.yaml
2019/01/28 15:29:47 writing secrets/certificate-controller-token-465t9.kube-system to /ambassador/ambassador-config/sync-1/kube-system/secrets/certificate-controller-token-465t9.yaml
2019/01/28 15:29:47 writing services/ambassador-admin.default to /ambassador/ambassador-config/sync-1/default/services/ambassador-admin.yaml
2019/01/28 15:29:47 writing services/kubernetes.default to /ambassador/ambassador-config/sync-1/default/services/kubernetes.yaml
2019/01/28 15:29:47 writing services/kube-dns.kube-system to /ambassador/ambassador-config/sync-1/kube-system/services/kube-dns.yaml
2019/01/28 15:29:47 SYNC: ambassador splitconfig --debug --k8s --bootstrap-path=/ambassador/bootstrap-ads.json --ads-path=/ambassador/envoy/envoy.json --ambex-pid=31 /ambassador/ambassador-config/sync-1
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: SCHEMA DIR    /ambassador/.cache/ambassador-0.0.0.dev0-py3.6.egg-tmp/schemas
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: ACONF RESET
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: LOAD_ALL: processed 0 resources
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: IR __init__:
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: IR: Version         0.50.0 built from 9456fbd on 0.50.0-rc5
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: IR: AMBASSADOR_ID   default
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: IR: Namespace       default
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: IR: Nodename        ambassador-default
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: IR: file checker:   isfile
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: IR: secret reader:  secret_reader
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: IRTracing: no tracing config, bailing
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: IRRateLimit: no ratelimit config, bailing
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: IRAuth: found no hosts! going inactive
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: cluster cluster_127_0_0_1_8877 service 127.0.0.1:8877
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: cluster cluster_127_0_0_1_8877 service 127.0.0.1:8877
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 DEBUG: cluster cluster_127_0_0_1_8877 service 127.0.0.1:8877
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 INFO: SAVING CONFIG
time="2019-01-28T15:29:48Z" level=info msg="Loaded file /ambassador/envoy/envoy.json"
2019/01/28 15:29:48 SYNC: 2019-01-28 15:29:48 ambassador-cli 0.50.0 INFO: RESTARTING
time="2019-01-28T15:29:48Z" level=info msg="Pushing snapshot v1"
2019/01/28 15:31:19 writing secrets/endpoint-controller-token-lcfpv.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/endpoint-controller-token-lcfpv.yaml
2019/01/28 15:31:19 writing secrets/resourcequota-controller-token-pxc4q.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/resourcequota-controller-token-pxc4q.yaml
2019/01/28 15:31:19 writing secrets/service-controller-token-4vksh.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/service-controller-token-4vksh.yaml
2019/01/28 15:31:19 writing secrets/coredns-token-nk2vb.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/coredns-token-nk2vb.yaml
2019/01/28 15:31:19 writing secrets/cronjob-controller-token-sg577.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/cronjob-controller-token-sg577.yaml
2019/01/28 15:31:19 writing secrets/default-token-jqcfm.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/default-token-jqcfm.yaml
2019/01/28 15:31:19 writing secrets/deployment-controller-token-q2jhz.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/deployment-controller-token-q2jhz.yaml
2019/01/28 15:31:19 writing secrets/certificate-controller-token-465t9.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/certificate-controller-token-465t9.yaml
2019/01/28 15:31:19 writing secrets/bootstrap-token-n0jv64.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/bootstrap-token-n0jv64.yaml
2019/01/28 15:31:19 writing secrets/clusterrole-aggregation-controller-token-sczvf.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/clusterrole-aggregation-controller-token-sczvf.yaml
2019/01/28 15:31:19 writing secrets/expand-controller-token-55f87.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/expand-controller-token-55f87.yaml
2019/01/28 15:31:19 writing secrets/horizontal-pod-autoscaler-token-fpqdq.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/horizontal-pod-autoscaler-token-fpqdq.yaml
2019/01/28 15:31:19 writing secrets/namespace-controller-token-6fb2s.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/namespace-controller-token-6fb2s.yaml
2019/01/28 15:31:19 writing secrets/pvc-protection-controller-token-wzkdt.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/pvc-protection-controller-token-wzkdt.yaml
2019/01/28 15:31:19 writing secrets/storage-provisioner-token-6dtwc.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/storage-provisioner-token-6dtwc.yaml
2019/01/28 15:31:19 writing secrets/default-token-pl6rp.default to /ambassador/ambassador-config/sync-2/default/secrets/default-token-pl6rp.yaml
2019/01/28 15:31:19 writing secrets/job-controller-token-4f2lm.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/job-controller-token-4f2lm.yaml
2019/01/28 15:31:19 writing secrets/pod-garbage-collector-token-dx69b.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/pod-garbage-collector-token-dx69b.yaml
2019/01/28 15:31:19 writing secrets/disruption-controller-token-x7tt2.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/disruption-controller-token-x7tt2.yaml
2019/01/28 15:31:19 writing secrets/token-cleaner-token-jwnwf.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/token-cleaner-token-jwnwf.yaml
2019/01/28 15:31:19 writing secrets/service-account-controller-token-nv669.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/service-account-controller-token-nv669.yaml
2019/01/28 15:31:19 writing secrets/generic-garbage-collector-token-wh8nb.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/generic-garbage-collector-token-wh8nb.yaml
2019/01/28 15:31:19 writing secrets/persistent-volume-binder-token-vchn2.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/persistent-volume-binder-token-vchn2.yaml
2019/01/28 15:31:19 writing secrets/replicaset-controller-token-wkcj5.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/replicaset-controller-token-wkcj5.yaml
2019/01/28 15:31:19 writing secrets/replication-controller-token-mr6cw.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/replication-controller-token-mr6cw.yaml
2019/01/28 15:31:19 writing secrets/ambassador-token-95w6z.default to /ambassador/ambassador-config/sync-2/default/secrets/ambassador-token-95w6z.yaml
2019/01/28 15:31:19 writing secrets/attachdetach-controller-token-69nxd.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/attachdetach-controller-token-69nxd.yaml
2019/01/28 15:31:19 writing secrets/bootstrap-signer-token-z49nr.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/bootstrap-signer-token-z49nr.yaml
2019/01/28 15:31:19 writing secrets/kube-proxy-token-8tf6q.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/kube-proxy-token-8tf6q.yaml
2019/01/28 15:31:19 writing secrets/node-controller-token-gcbhb.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/node-controller-token-gcbhb.yaml
2019/01/28 15:31:19 writing secrets/pv-protection-controller-token-l4hxx.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/pv-protection-controller-token-l4hxx.yaml
2019/01/28 15:31:19 writing secrets/ttl-controller-token-k8j8x.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/ttl-controller-token-k8j8x.yaml
2019/01/28 15:31:19 writing secrets/default-token-hm9mw.kube-public to /ambassador/ambassador-config/sync-2/kube-public/secrets/default-token-hm9mw.yaml
2019/01/28 15:31:19 writing secrets/statefulset-controller-token-zpcjv.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/statefulset-controller-token-zpcjv.yaml
2019/01/28 15:31:19 writing secrets/daemon-set-controller-token-2t2km.kube-system to /ambassador/ambassador-config/sync-2/kube-system/secrets/daemon-set-controller-token-2t2km.yaml
2019/01/28 15:31:19 writing services/ambassador-admin.default to /ambassador/ambassador-config/sync-2/default/services/ambassador-admin.yaml
2019/01/28 15:31:19 writing services/kubernetes.default to /ambassador/ambassador-config/sync-2/default/services/kubernetes.yaml
2019/01/28 15:31:19 writing services/kube-dns.kube-system to /ambassador/ambassador-config/sync-2/kube-system/services/kube-dns.yaml
2019/01/28 15:31:19 writing services/ambassador.default to /ambassador/ambassador-config/sync-2/default/services/ambassador.yaml
2019/01/28 15:31:19 SYNC: ambassador splitconfig --debug --k8s --bootstrap-path=/ambassador/bootstrap-ads.json --ads-path=/ambassador/envoy/envoy.json --ambex-pid=31 /ambassador/ambassador-config/sync-2
2019/01/28 15:31:20 SYNC: 2019-01-28 15:31:20 ambassador-cli 0.50.0 DEBUG: SCHEMA DIR    /ambassador/.cache/ambassador-0.0.0.dev0-py3.6.egg-tmp/schemas
2019/01/28 15:31:20 SYNC: 2019-01-28 15:31:20 ambassador-cli 0.50.0 DEBUG: ACONF RESET
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: LOAD_ALL: processed 0 resources
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: IR __init__:
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: IR: Version         0.50.0 built from 9456fbd on 0.50.0-rc5
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: IR: AMBASSADOR_ID   default
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: IR: Namespace       default
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: IR: Nodename        ambassador-default
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: IR: file checker:   isfile
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: IR: secret reader:  secret_reader
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: IRTracing: no tracing config, bailing
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: IRRateLimit: no ratelimit config, bailing
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: IRAuth: found no hosts! going inactive
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: cluster cluster_127_0_0_1_8877 service 127.0.0.1:8877
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: cluster cluster_127_0_0_1_8877 service 127.0.0.1:8877
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 DEBUG: cluster cluster_127_0_0_1_8877 service 127.0.0.1:8877
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 INFO: SAVING CONFIG
2019/01/28 15:31:21 SYNC: 2019-01-28 15:31:21 ambassador-cli 0.50.0 INFO: RESTARTING
time="2019-01-28T15:31:21Z" level=info msg="Loaded file /ambassador/envoy/envoy.json"
time="2019-01-28T15:31:21Z" level=info msg="Pushing snapshot v2"
2019/01/28 15:34:42 writing secrets/bootstrap-signer-token-z49nr.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/bootstrap-signer-token-z49nr.yaml
2019/01/28 15:34:42 writing secrets/kube-proxy-token-8tf6q.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/kube-proxy-token-8tf6q.yaml
2019/01/28 15:34:42 writing secrets/node-controller-token-gcbhb.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/node-controller-token-gcbhb.yaml
2019/01/28 15:34:42 writing secrets/pv-protection-controller-token-l4hxx.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/pv-protection-controller-token-l4hxx.yaml
2019/01/28 15:34:42 writing secrets/ttl-controller-token-k8j8x.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/ttl-controller-token-k8j8x.yaml
2019/01/28 15:34:42 writing secrets/default-token-hm9mw.kube-public to /ambassador/ambassador-config/sync-3/kube-public/secrets/default-token-hm9mw.yaml
2019/01/28 15:34:42 writing secrets/attachdetach-controller-token-69nxd.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/attachdetach-controller-token-69nxd.yaml
2019/01/28 15:34:42 writing secrets/daemon-set-controller-token-2t2km.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/daemon-set-controller-token-2t2km.yaml
2019/01/28 15:34:42 writing secrets/statefulset-controller-token-zpcjv.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/statefulset-controller-token-zpcjv.yaml
2019/01/28 15:34:42 writing secrets/resourcequota-controller-token-pxc4q.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/resourcequota-controller-token-pxc4q.yaml
2019/01/28 15:34:42 writing secrets/service-controller-token-4vksh.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/service-controller-token-4vksh.yaml
2019/01/28 15:34:42 writing secrets/coredns-token-nk2vb.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/coredns-token-nk2vb.yaml
2019/01/28 15:34:42 writing secrets/endpoint-controller-token-lcfpv.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/endpoint-controller-token-lcfpv.yaml
2019/01/28 15:34:42 writing secrets/default-token-jqcfm.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/default-token-jqcfm.yaml
2019/01/28 15:34:42 writing secrets/deployment-controller-token-q2jhz.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/deployment-controller-token-q2jhz.yaml
2019/01/28 15:34:42 writing secrets/certificate-controller-token-465t9.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/certificate-controller-token-465t9.yaml
2019/01/28 15:34:42 writing secrets/cronjob-controller-token-sg577.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/cronjob-controller-token-sg577.yaml
2019/01/28 15:34:42 writing secrets/clusterrole-aggregation-controller-token-sczvf.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/clusterrole-aggregation-controller-token-sczvf.yaml
2019/01/28 15:34:42 writing secrets/expand-controller-token-55f87.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/expand-controller-token-55f87.yaml
2019/01/28 15:34:42 writing secrets/horizontal-pod-autoscaler-token-fpqdq.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/horizontal-pod-autoscaler-token-fpqdq.yaml
2019/01/28 15:34:42 writing secrets/namespace-controller-token-6fb2s.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/namespace-controller-token-6fb2s.yaml
2019/01/28 15:34:42 writing secrets/pvc-protection-controller-token-wzkdt.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/pvc-protection-controller-token-wzkdt.yaml
2019/01/28 15:34:42 writing secrets/storage-provisioner-token-6dtwc.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/storage-provisioner-token-6dtwc.yaml
2019/01/28 15:34:42 writing secrets/default-token-pl6rp.default to /ambassador/ambassador-config/sync-3/default/secrets/default-token-pl6rp.yaml
2019/01/28 15:34:42 writing secrets/bootstrap-token-n0jv64.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/bootstrap-token-n0jv64.yaml
2019/01/28 15:34:42 writing secrets/pod-garbage-collector-token-dx69b.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/pod-garbage-collector-token-dx69b.yaml
2019/01/28 15:34:42 writing secrets/disruption-controller-token-x7tt2.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/disruption-controller-token-x7tt2.yaml
2019/01/28 15:34:42 writing secrets/job-controller-token-4f2lm.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/job-controller-token-4f2lm.yaml
2019/01/28 15:34:42 writing secrets/service-account-controller-token-nv669.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/service-account-controller-token-nv669.yaml
2019/01/28 15:34:42 writing secrets/token-cleaner-token-jwnwf.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/token-cleaner-token-jwnwf.yaml
2019/01/28 15:34:42 writing secrets/persistent-volume-binder-token-vchn2.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/persistent-volume-binder-token-vchn2.yaml
2019/01/28 15:34:42 writing secrets/replicaset-controller-token-wkcj5.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/replicaset-controller-token-wkcj5.yaml
2019/01/28 15:34:42 writing secrets/replication-controller-token-mr6cw.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/replication-controller-token-mr6cw.yaml
2019/01/28 15:34:42 writing secrets/ambassador-token-95w6z.default to /ambassador/ambassador-config/sync-3/default/secrets/ambassador-token-95w6z.yaml
2019/01/28 15:34:42 writing secrets/generic-garbage-collector-token-wh8nb.kube-system to /ambassador/ambassador-config/sync-3/kube-system/secrets/generic-garbage-collector-token-wh8nb.yaml
2019/01/28 15:34:42 writing services/qotm.default to /ambassador/ambassador-config/sync-3/default/services/qotm.yaml
2019/01/28 15:34:42 writing services/ambassador-admin.default to /ambassador/ambassador-config/sync-3/default/services/ambassador-admin.yaml
2019/01/28 15:34:42 writing services/kubernetes.default to /ambassador/ambassador-config/sync-3/default/services/kubernetes.yaml
2019/01/28 15:34:42 writing services/kube-dns.kube-system to /ambassador/ambassador-config/sync-3/kube-system/services/kube-dns.yaml
2019/01/28 15:34:42 writing services/ambassador.default to /ambassador/ambassador-config/sync-3/default/services/ambassador.yaml
2019/01/28 15:34:42 SYNC: ambassador splitconfig --debug --k8s --bootstrap-path=/ambassador/bootstrap-ads.json --ads-path=/ambassador/envoy/envoy.json --ambex-pid=31 /ambassador/ambassador-config/sync-3
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: SCHEMA DIR    /ambassador/.cache/ambassador-0.0.0.dev0-py3.6.egg-tmp/schemas
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: ACONF RESET
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: LOAD_ALL: <Mapping qotm.default.1> @ qotm.default.1
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 INFO: qotm.default.1: NOTICE: apiVersion ambassador/v0 is deprecated, consider upgrading
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: <Mapping qotm.default.1>: saving Mapping qotm_mapping
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: LOAD_ALL: processed 1 resource
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: IR __init__:
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: IR: Version         0.50.0 built from 9456fbd on 0.50.0-rc5
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: IR: AMBASSADOR_ID   default
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: IR: Namespace       default
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: IR: Nodename        ambassador-default
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: IR: file checker:   isfile
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: IR: secret reader:  secret_reader
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: IRTracing: no tracing config, bailing
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: IRRateLimit: no ratelimit config, bailing
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: IRAuth: found no hosts! going inactive
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: cluster cluster_qotm service qotm
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: cluster cluster_127_0_0_1_8877 service 127.0.0.1:8877
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: cluster cluster_127_0_0_1_8877 service 127.0.0.1:8877
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 DEBUG: cluster cluster_127_0_0_1_8877 service 127.0.0.1:8877
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 INFO: SAVING CONFIG
2019/01/28 15:34:43 SYNC: 2019-01-28 15:34:43 ambassador-cli 0.50.0 INFO: RESTARTING
time="2019-01-28T15:34:43Z" level=info msg="Loaded file /ambassador/envoy/envoy.json"
time="2019-01-28T15:34:43Z" level=info msg="Pushing snapshot v3"
[2019-01-28 15:34:43.818][000032][info][upstream] [source/common/upstream/cluster_manager_impl.cc:502] add/update cluster cluster_qotm starting warming
[2019-01-28 15:34:43.857][000032][warning][config] [bazel-out/k8-dbg/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:70] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener ambassador-listener-80: route: unknown weighted cluster 'cluster_qotm'
[2019-01-28 15:34:43.864][000032][warning][config] [bazel-out/k8-dbg/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:70] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener ambassador-listener-80: route: unknown weighted cluster 'cluster_qotm'
[2019-01-28 15:34:43.870][000032][warning][config] [bazel-out/k8-dbg/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:70] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener ambassador-listener-80: route: unknown weighted cluster 'cluster_qotm'
[2019-01-28 15:34:43.876][000032][warning][config] [bazel-out/k8-dbg/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:70] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener ambassador-listener-80: route: unknown weighted cluster 'cluster_qotm'
[2019-01-28 15:34:43.882][000032][warning][config] [bazel-out/k8-dbg/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:70] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener ambassador-listener-80: route: unknown weighted cluster 'cluster_qotm'
[2019-01-28 15:34:43.888][000032][warning][config] [bazel-out/k8-dbg/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:70] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener ambassador-listener-80: route: unknown weighted cluster 'cluster_qotm'
[2019-01-28 15:34:43.893][000032][warning][config] [bazel-out/k8-dbg/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:70] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener ambassador-listener-80: route: unknown weighted cluster 'cluster_qotm'
[2019-01-28 15:34:43.894][000032][info][upstream] [source/common/upstream/cluster_manager_impl.cc:514] warming cluster cluster_qotm complete
[2019-01-28 15:34:43.900][000032][info][upstream] [source/server/lds_api.cc:80] lds: add/update listener 'ambassador-listener-80'
ACCESS [2019-01-28T15:37:19.017Z] "GET /qotm/ HTTP/1.1" 200 - 0 142 40 39 "192.168.99.1" "curl/7.54.0" "030c24d9-697c-4046-a03b-6ac4a3554775" "192.168.99.100:31786" "10.104.139.255:80"
2019/01/28 15:40:39 writing secrets/service-account-controller-token-nv669.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/service-account-controller-token-nv669.yaml
2019/01/28 15:40:39 writing secrets/token-cleaner-token-jwnwf.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/token-cleaner-token-jwnwf.yaml
2019/01/28 15:40:39 writing secrets/replicaset-controller-token-wkcj5.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/replicaset-controller-token-wkcj5.yaml
2019/01/28 15:40:39 writing secrets/replication-controller-token-mr6cw.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/replication-controller-token-mr6cw.yaml
2019/01/28 15:40:39 writing secrets/ambassador-token-95w6z.default to /ambassador/ambassador-config/sync-4/default/secrets/ambassador-token-95w6z.yaml
2019/01/28 15:40:39 writing secrets/generic-garbage-collector-token-wh8nb.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/generic-garbage-collector-token-wh8nb.yaml
2019/01/28 15:40:39 writing secrets/persistent-volume-binder-token-vchn2.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/persistent-volume-binder-token-vchn2.yaml
2019/01/28 15:40:39 writing secrets/kube-proxy-token-8tf6q.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/kube-proxy-token-8tf6q.yaml
2019/01/28 15:40:39 writing secrets/node-controller-token-gcbhb.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/node-controller-token-gcbhb.yaml
2019/01/28 15:40:39 writing secrets/pv-protection-controller-token-l4hxx.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/pv-protection-controller-token-l4hxx.yaml
2019/01/28 15:40:39 writing secrets/ttl-controller-token-k8j8x.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/ttl-controller-token-k8j8x.yaml
2019/01/28 15:40:39 writing secrets/default-token-hm9mw.kube-public to /ambassador/ambassador-config/sync-4/kube-public/secrets/default-token-hm9mw.yaml
2019/01/28 15:40:39 writing secrets/attachdetach-controller-token-69nxd.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/attachdetach-controller-token-69nxd.yaml
2019/01/28 15:40:39 writing secrets/bootstrap-signer-token-z49nr.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/bootstrap-signer-token-z49nr.yaml
2019/01/28 15:40:39 writing secrets/daemon-set-controller-token-2t2km.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/daemon-set-controller-token-2t2km.yaml
2019/01/28 15:40:39 writing secrets/statefulset-controller-token-zpcjv.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/statefulset-controller-token-zpcjv.yaml
2019/01/28 15:40:39 writing secrets/service-controller-token-4vksh.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/service-controller-token-4vksh.yaml
2019/01/28 15:40:39 writing secrets/coredns-token-nk2vb.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/coredns-token-nk2vb.yaml
2019/01/28 15:40:39 writing secrets/endpoint-controller-token-lcfpv.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/endpoint-controller-token-lcfpv.yaml
2019/01/28 15:40:39 writing secrets/resourcequota-controller-token-pxc4q.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/resourcequota-controller-token-pxc4q.yaml
2019/01/28 15:40:39 writing secrets/deployment-controller-token-q2jhz.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/deployment-controller-token-q2jhz.yaml
2019/01/28 15:40:39 writing secrets/certificate-controller-token-465t9.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/certificate-controller-token-465t9.yaml
2019/01/28 15:40:39 writing secrets/cronjob-controller-token-sg577.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/cronjob-controller-token-sg577.yaml
2019/01/28 15:40:39 writing secrets/default-token-jqcfm.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/default-token-jqcfm.yaml
2019/01/28 15:40:39 writing secrets/expand-controller-token-55f87.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/expand-controller-token-55f87.yaml
2019/01/28 15:40:39 writing secrets/horizontal-pod-autoscaler-token-fpqdq.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/horizontal-pod-autoscaler-token-fpqdq.yaml
2019/01/28 15:40:39 writing secrets/namespace-controller-token-6fb2s.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/namespace-controller-token-6fb2s.yaml
2019/01/28 15:40:39 writing secrets/pvc-protection-controller-token-wzkdt.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/pvc-protection-controller-token-wzkdt.yaml
2019/01/28 15:40:39 writing secrets/storage-provisioner-token-6dtwc.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/storage-provisioner-token-6dtwc.yaml
2019/01/28 15:40:39 writing secrets/default-token-pl6rp.default to /ambassador/ambassador-config/sync-4/default/secrets/default-token-pl6rp.yaml
2019/01/28 15:40:39 writing secrets/bootstrap-token-n0jv64.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/bootstrap-token-n0jv64.yaml
2019/01/28 15:40:39 writing secrets/clusterrole-aggregation-controller-token-sczvf.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/clusterrole-aggregation-controller-token-sczvf.yaml
2019/01/28 15:40:39 writing secrets/disruption-controller-token-x7tt2.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/disruption-controller-token-x7tt2.yaml
2019/01/28 15:40:39 writing secrets/job-controller-token-4f2lm.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/job-controller-token-4f2lm.yaml
2019/01/28 15:40:39 writing secrets/pod-garbage-collector-token-dx69b.kube-system to /ambassador/ambassador-config/sync-4/kube-system/secrets/pod-garbage-collector-token-dx69b.yaml
2019/01/28 15:40:39 writing services/ambassador-admin.default to /ambassador/ambassador-config/sync-4/default/services/ambassador-admin.yaml
2019/01/28 15:40:39 writing services/kubernetes.default to /ambassador/ambassador-config/sync-4/default/services/kubernetes.yaml
2019/01/28 15:40:39 writing services/kube-dns.kube-system to /ambassador/ambassador-config/sync-4/kube-system/services/kube-dns.yaml
2019/01/28 15:40:39 writing services/ambassador.default to /ambassador/ambassador-config/sync-4/default/services/ambassador.yaml
2019/01/28 15:40:39 writing services/qotm.default to /ambassador/ambassador-config/sync-4/default/services/qotm.yaml
2019/01/28 15:40:39 writing services/example-rate-limit.default to /ambassador/ambassador-config/sync-4/default/services/example-rate-limit.yaml
2019/01/28 15:40:39 SYNC: ambassador splitconfig --debug --k8s --bootstrap-path=/ambassador/bootstrap-ads.json --ads-path=/ambassador/envoy/envoy.json --ambex-pid=31 /ambassador/ambassador-config/sync-4
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: SCHEMA DIR    /ambassador/.cache/ambassador-0.0.0.dev0-py3.6.egg-tmp/schemas
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: ACONF RESET
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: LOAD_ALL: <RateLimitService example-rate-limit.default.1> @ example-rate-limit.default.1
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 INFO: example-rate-limit.default.1: NOTICE: apiVersion ambassador/v0 is deprecated, consider upgrading
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: <RateLimitService example-rate-limit.default.1>: saving RateLimitService ratelimit
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: LOAD_ALL: <Mapping qotm.default.1> @ qotm.default.1
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 INFO: qotm.default.1: NOTICE: apiVersion ambassador/v0 is deprecated, consider upgrading
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: <Mapping qotm.default.1>: saving Mapping qotm_mapping
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: LOAD_ALL: processed 2 resources
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: IR __init__:
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: IR: Version         0.50.0 built from 9456fbd on 0.50.0-rc5
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: IR: AMBASSADOR_ID   default
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: IR: Namespace       default
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: IR: Nodename        ambassador-default
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: IR: file checker:   isfile
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: IR: secret reader:  secret_reader
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: IRTracing: no tracing config, bailing
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: IRRateLimit: ratelimit using service example-rate-limit:5000
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: IRAuth: found no hosts! going inactive
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: cluster cluster_example_rate_limit_5000 service example-rate-limit:5000
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: cluster cluster_qotm service qotm
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: cluster cluster_127_0_0_1_8877 service 127.0.0.1:8877
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: cluster cluster_127_0_0_1_8877 service 127.0.0.1:8877
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 DEBUG: cluster cluster_127_0_0_1_8877 service 127.0.0.1:8877
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 INFO: SAVING CONFIG
2019/01/28 15:40:40 SYNC: 2019-01-28 15:40:40 ambassador-cli 0.50.0 INFO: RESTARTING
time="2019-01-28T15:40:40Z" level=info msg="Loaded file /ambassador/envoy/envoy.json"
time="2019-01-28T15:40:40Z" level=info msg="Pushing snapshot v4"
[2019-01-28 15:40:40.960][000032][info][upstream] [source/common/upstream/cluster_manager_impl.cc:502] add/update cluster cluster_example_rate_limit_5000 starting warming
[2019-01-28 15:40:40.968][000032][info][upstream] [source/server/lds_api.cc:80] lds: add/update listener 'ambassador-listener-80'
[2019-01-28 15:40:40.971][000032][info][upstream] [source/common/upstream/cluster_manager_impl.cc:514] warming cluster cluster_example_rate_limit_5000 complete
2019/01/28 15:41:59 writing secrets/ambassador-token-95w6z.default to /ambassador/ambassador-config/sync-5/default/secrets/ambassador-token-95w6z.yaml
2019/01/28 15:41:59 writing secrets/generic-garbage-collector-token-wh8nb.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/generic-garbage-collector-token-wh8nb.yaml
2019/01/28 15:41:59 writing secrets/persistent-volume-binder-token-vchn2.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/persistent-volume-binder-token-vchn2.yaml
2019/01/28 15:41:59 writing secrets/replicaset-controller-token-wkcj5.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/replicaset-controller-token-wkcj5.yaml
2019/01/28 15:41:59 writing secrets/replication-controller-token-mr6cw.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/replication-controller-token-mr6cw.yaml
2019/01/28 15:41:59 writing secrets/default-token-hm9mw.kube-public to /ambassador/ambassador-config/sync-5/kube-public/secrets/default-token-hm9mw.yaml
2019/01/28 15:41:59 writing secrets/attachdetach-controller-token-69nxd.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/attachdetach-controller-token-69nxd.yaml
2019/01/28 15:41:59 writing secrets/bootstrap-signer-token-z49nr.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/bootstrap-signer-token-z49nr.yaml
2019/01/28 15:41:59 writing secrets/kube-proxy-token-8tf6q.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/kube-proxy-token-8tf6q.yaml
2019/01/28 15:41:59 writing secrets/node-controller-token-gcbhb.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/node-controller-token-gcbhb.yaml
2019/01/28 15:41:59 writing secrets/pv-protection-controller-token-l4hxx.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/pv-protection-controller-token-l4hxx.yaml
2019/01/28 15:41:59 writing secrets/ttl-controller-token-k8j8x.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/ttl-controller-token-k8j8x.yaml
2019/01/28 15:41:59 writing secrets/daemon-set-controller-token-2t2km.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/daemon-set-controller-token-2t2km.yaml
2019/01/28 15:41:59 writing secrets/statefulset-controller-token-zpcjv.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/statefulset-controller-token-zpcjv.yaml
2019/01/28 15:41:59 writing secrets/coredns-token-nk2vb.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/coredns-token-nk2vb.yaml
2019/01/28 15:41:59 writing secrets/endpoint-controller-token-lcfpv.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/endpoint-controller-token-lcfpv.yaml
2019/01/28 15:41:59 writing secrets/resourcequota-controller-token-pxc4q.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/resourcequota-controller-token-pxc4q.yaml
2019/01/28 15:41:59 writing secrets/service-controller-token-4vksh.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/service-controller-token-4vksh.yaml
2019/01/28 15:41:59 writing secrets/certificate-controller-token-465t9.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/certificate-controller-token-465t9.yaml
2019/01/28 15:41:59 writing secrets/cronjob-controller-token-sg577.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/cronjob-controller-token-sg577.yaml
2019/01/28 15:41:59 writing secrets/default-token-jqcfm.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/default-token-jqcfm.yaml
2019/01/28 15:41:59 writing secrets/deployment-controller-token-q2jhz.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/deployment-controller-token-q2jhz.yaml
2019/01/28 15:41:59 writing secrets/storage-provisioner-token-6dtwc.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/storage-provisioner-token-6dtwc.yaml
2019/01/28 15:41:59 writing secrets/default-token-pl6rp.default to /ambassador/ambassador-config/sync-5/default/secrets/default-token-pl6rp.yaml
2019/01/28 15:41:59 writing secrets/bootstrap-token-n0jv64.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/bootstrap-token-n0jv64.yaml
2019/01/28 15:41:59 writing secrets/clusterrole-aggregation-controller-token-sczvf.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/clusterrole-aggregation-controller-token-sczvf.yaml
2019/01/28 15:41:59 writing secrets/expand-controller-token-55f87.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/expand-controller-token-55f87.yaml
2019/01/28 15:41:59 writing secrets/horizontal-pod-autoscaler-token-fpqdq.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/horizontal-pod-autoscaler-token-fpqdq.yaml
2019/01/28 15:41:59 writing secrets/namespace-controller-token-6fb2s.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/namespace-controller-token-6fb2s.yaml
2019/01/28 15:41:59 writing secrets/pvc-protection-controller-token-wzkdt.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/pvc-protection-controller-token-wzkdt.yaml
2019/01/28 15:41:59 writing secrets/disruption-controller-token-x7tt2.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/disruption-controller-token-x7tt2.yaml
2019/01/28 15:41:59 writing secrets/job-controller-token-4f2lm.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/job-controller-token-4f2lm.yaml
2019/01/28 15:41:59 writing secrets/pod-garbage-collector-token-dx69b.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/pod-garbage-collector-token-dx69b.yaml
2019/01/28 15:41:59 writing secrets/service-account-controller-token-nv669.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/service-account-controller-token-nv669.yaml
2019/01/28 15:41:59 writing secrets/token-cleaner-token-jwnwf.kube-system to /ambassador/ambassador-config/sync-5/kube-system/secrets/token-cleaner-token-jwnwf.yaml
2019/01/28 15:41:59 writing services/example-rate-limit.default to /ambassador/ambassador-config/sync-5/default/services/example-rate-limit.yaml
2019/01/28 15:41:59 writing services/ambassador-admin.default to /ambassador/ambassador-config/sync-5/default/services/ambassador-admin.yaml
2019/01/28 15:41:59 writing services/kubernetes.default to /ambassador/ambassador-config/sync-5/default/services/kubernetes.yaml
2019/01/28 15:41:59 writing services/kube-dns.kube-system to /ambassador/ambassador-config/sync-5/kube-system/services/kube-dns.yaml
2019/01/28 15:41:59 writing services/ambassador.default to /ambassador/ambassador-config/sync-5/default/services/ambassador.yaml
2019/01/28 15:41:59 writing services/qotm.default to /ambassador/ambassador-config/sync-5/default/services/qotm.yaml
2019/01/28 15:41:59 SYNC: ambassador splitconfig --debug --k8s --bootstrap-path=/ambassador/bootstrap-ads.json --ads-path=/ambassador/envoy/envoy.json --ambex-pid=31 /ambassador/ambassador-config/sync-5
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: SCHEMA DIR    /ambassador/.cache/ambassador-0.0.0.dev0-py3.6.egg-tmp/schemas
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: ACONF RESET
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: LOAD_ALL: <RateLimitService example-rate-limit.default.1> @ example-rate-limit.default.1
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 INFO: example-rate-limit.default.1: NOTICE: apiVersion ambassador/v0 is deprecated, consider upgrading
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: <RateLimitService example-rate-limit.default.1>: saving RateLimitService ratelimit
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: LOAD_ALL: <Mapping qotm.default.1> @ qotm.default.1
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: <Mapping qotm.default.1>: saving Mapping qotm_mapping
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: LOAD_ALL: processed 2 resources
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: IR __init__:
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: IR: Version         0.50.0 built from 9456fbd on 0.50.0-rc5
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: IR: AMBASSADOR_ID   default
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: IR: Namespace       default
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: IR: Nodename        ambassador-default
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: IR: file checker:   isfile
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: IR: secret reader:  secret_reader
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: IRTracing: no tracing config, bailing
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: IRRateLimit: ratelimit using service example-rate-limit:5000
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: IRAuth: found no hosts! going inactive
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: cluster cluster_example_rate_limit_5000 service example-rate-limit:5000
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: ambassador: defaults None
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: cluster cluster_qotm service qotm
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: cluster cluster_127_0_0_1_8877 service 127.0.0.1:8877
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: cluster cluster_127_0_0_1_8877 service 127.0.0.1:8877
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: default_labels info for ambassador: {}
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: cluster cluster_127_0_0_1_8877 service 127.0.0.1:8877
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: V2RateLimitAction translating {'request_label_group': [{'x-ambassador-test-allow': {'header': 'x-ambassador-test-allow', 'omit_if_not_present': True}}]}
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 DEBUG: V2RateLimitAction working on '{'x-ambassador-test-allow': {'header': 'x-ambassador-test-allow', 'omit_if_not_present': True}}'
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 INFO: SAVING CONFIG
2019/01/28 15:42:00 SYNC: 2019-01-28 15:42:00 ambassador-cli 0.50.0 INFO: RESTARTING
time="2019-01-28T15:42:00Z" level=info msg="Loaded file /ambassador/envoy/envoy.json"
time="2019-01-28T15:42:00Z" level=info msg="Pushing snapshot v5"
[2019-01-28 15:42:00.523][000032][info][upstream] [source/server/lds_api.cc:80] lds: add/update listener 'ambassador-listener-80'
ACCESS [2019-01-28T15:42:19.070Z] "GET /qotm/quote/1 HTTP/1.1" 200 - 0 147 4 1 "192.168.99.1" "curl/7.54.0" "b00d3ff4-804a-4d3d-8026-e9cbaf6b83c1" "192.168.99.100:31786" "10.104.139.255:80"
ACCESS [2019-01-28T15:42:22.702Z] "GET /qotm/quote/1 HTTP/1.1" 200 - 0 147 3 2 "192.168.99.1" "curl/7.54.0" "3631f33e-2d06-480b-9b23-5afb16051993" "192.168.99.100:31786" "10.104.139.255:80"
ACCESS [2019-01-28T15:42:23.851Z] "GET /qotm/quote/1 HTTP/1.1" 200 - 0 147 6 5 "192.168.99.1" "curl/7.54.0" "806dd9af-a548-47f6-9431-6f47c3c0663d" "192.168.99.100:31786" "10.104.139.255:80"
ACCESS [2019-01-28T15:42:24.868Z] "GET /qotm/quote/1 HTTP/1.1" 200 - 0 147 5 3 "192.168.99.1" "curl/7.54.0" "5d2377b4-dd26-4972-bdf6-07909cf6e9dc" "192.168.99.100:31786" "10.104.139.255:80"
ACCESS [2019-01-28T15:42:25.803Z] "GET /qotm/quote/1 HTTP/1.1" 200 - 0 147 54 53 "192.168.99.1" "curl/7.54.0" "84a6d6df-968c-4be7-931f-cdf20884afb0" "192.168.99.100:31786" "10.104.139.255:80"
ACCESS [2019-01-28T15:42:27.264Z] "GET /qotm/quote/1 HTTP/1.1" 200 - 0 147 92 91 "192.168.99.1" "curl/7.54.0" "1cb00f2e-8192-4605-8a1a-13e3745786aa" "192.168.99.100:31786" "10.104.139.255:80"
ACCESS [2019-01-28T15:42:29.363Z] "GET /qotm/quote/1 HTTP/1.1" 200 - 0 147 93 1 "192.168.99.1" "curl/7.54.0" "07d7cc27-86df-4de6-ba11-dab1db220d76" "192.168.99.100:31786" "10.104.139.255:80"
ACCESS [2019-01-28T15:42:56.848Z] "GET /qotm/quote/1 HTTP/1.1" 200 - 0 147 9 9 "192.168.99.1" "curl/7.54.0" "681135a9-f503-4c9b-a64b-678ffa5d8248" "192.168.99.100:31786" "10.104.139.255:80"
[2019-01-28 15:44:37.407][000032][info][main] [source/server/drain_manager_impl.cc:63] shutting down parent after drain
@travisfinch

This comment has been minimized.

Copy link

travisfinch commented Feb 8, 2019

Yes, I am not able to get the rate limiter working with 0.50.0 either. Perhaps it is due to the ambassador/v1 API vs. all of the docs and examples using v0.

@travisfinch

This comment has been minimized.

Copy link

travisfinch commented Feb 12, 2019

My findings on this was that rate limiting does not work at all on 0.50.0 GA, and kinda works on 0.50.1. I say kinda because the example rate limiter works, but my grpc service does not. My grpc service never gets a rate limit check request. When I downgrade to 0.40.2 and go back to using the v0 API in my YAML files, my grpc service works.

@nbkrause

This comment has been minimized.

Copy link
Contributor

nbkrause commented Feb 26, 2019

This has been resolved in 0.50.1. Thanks for reporting it.

Closing.

@nbkrause nbkrause closed this Feb 26, 2019

@travisfinch

This comment has been minimized.

Copy link

travisfinch commented Feb 26, 2019

This should be reopened. I can reproduce with 0.50.3. The example rate limiter does not work.

@nbkrause nbkrause reopened this Feb 26, 2019

@nbkrause

This comment has been minimized.

Copy link
Contributor

nbkrause commented Feb 26, 2019

@travisfinch Could you open a separate issue for your issue with the gRPC rate limit service?

@travisfinch

This comment has been minimized.

Copy link

travisfinch commented Mar 14, 2019

Finally had a bit of time to look at this, and I have an idea of why it is failing. The example service uses the old ratelimit.proto interface.

As of Envoy 1.9, this is no longer supported. You have to use the new proto file from Lyft. As of Ambassador 0.51.2, this is the version being used:

envoy version: f60eead706a3606c545ee2cfb0d47e905fdf1813/1.9.0-dev/Modified/DEBUG

See "use_data_plane_proto" on https://www.envoyproxy.io/docs/envoy/v1.8.0/api-v2/config/ratelimit/v2/rls.proto

I am going to try it out today and see if I can get it working again.

@marselester

This comment has been minimized.

Copy link
Author

marselester commented Mar 18, 2019

I tried to implement a rate limiting service using Envoy's rls.proto from https://github.com/lyft/ratelimit/blob/master/proto/envoy/service/ratelimit/v2/rls.pb.go and managed to get 429 with this Envoy config:

static_resources:
  listeners:
    - name: http_listener
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 10000
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              config:
                stat_prefix: traveling_ingress
                use_remote_address: true
                http_filters:
                  - name: envoy.ext_authz
                    config:
                      http_service:
                        server_uri:
                          uri: "travelauth.default"
                          cluster: travelauth_service
                          timeout: 0.25s
                  - name: envoy.rate_limit
                    config:
                      domain: apis
                      stage: 0
                  - name: envoy.router
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: traveling
                      domains: ["*"]
                      routes:
                        - match: { prefix: "/v1/hotels" }
                          route: { cluster: hotel_service }
                        - match: { prefix: "/v1/cars" }
                          route: { cluster: car_service }
                      rate_limits:
                        - stage: 0
                          actions:
                            - generic_key: { descriptor_value: "default" }
  clusters:
    - name: hotel_service
      connect_timeout: 0.25s
      type: STRICT_DNS
      dns_lookup_family: V4_ONLY
      lb_policy: ROUND_ROBIN
      hosts:
        - socket_address: { address: "hotel.default", port_value: 80 }
    - name: car_service
      connect_timeout: 0.25s
      type: STRICT_DNS
      dns_lookup_family: V4_ONLY
      lb_policy: ROUND_ROBIN
      hosts:
        - socket_address: { address: "car.default", port_value: 80 }
    - name: travelauth_service
      connect_timeout: 0.25s
      type: STRICT_DNS
      dns_lookup_family: V4_ONLY
      lb_policy: ROUND_ROBIN
      hosts:
        - socket_address: { address: "travelauth.default", port_value: 80 }
    - name: travelratelimit_service
      connect_timeout: 0.25s
      type: STRICT_DNS
      dns_lookup_family: V4_ONLY
      lb_policy: ROUND_ROBIN
      http2_protocol_options: {}
      hosts:
        - socket_address: { address: "travelratelimit.default", port_value: 80 }

rate_limit_service:
  grpc_service:
    envoy_grpc:
      cluster_name: travelratelimit_service
    timeout: 0.25s

admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901

I also tested rls.proto-based rate limiter with Ambassador, but no luck so far in my minikube setup.

@travisfinch

This comment has been minimized.

Copy link

travisfinch commented Mar 18, 2019

I also tested rls.proto-based rate limiter with Ambassador, but no luck so far in my minikube setup.

Same here. Also made a custom Ambassador build with the "use_data_plane_proto" Envoy configuration removed, but no luck. I'm all out of ideas and time, so I am just going to go back to 0.40.2.

@travisfinch

This comment has been minimized.

Copy link

travisfinch commented Mar 22, 2019

Maybe another idea...

Per https://www.envoyproxy.io/docs/envoy/v1.9.0/api-v2/config/filter/http/rate_limit/v2/rate_limit.proto a "rate_limit_service" needs to be defined for the HTTP filter chain. If not specified, any calls to the rate limit service will immediately return success. On my Ambassador 0.52 from the envoy config in the container it is not there:

                                    {                                                                                                                                                                 
                                        "config": {                                                                                                                                                   
                                            "domain": "ambassador",                                                                                                                                   
                                            "request_type": "both",                                                                                                                                   
                                            "timeout": "0.020s"                                                                                                                                       
                                        },                                                                                                                                                            
                                        "name": "envoy.rate_limit"                                                                                                                                    
                                    },  

It looks like older versions of Envoy did not require this: https://www.envoyproxy.io/docs/envoy/v1.8.0/api-v2/config/filter/http/rate_limit/v2/rate_limit.proto

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.