Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] ColorGateway connection refused #142

Closed
kand617 opened this issue Apr 26, 2019 · 12 comments
Closed

[BUG] ColorGateway connection refused #142

kand617 opened this issue Apr 26, 2019 · 12 comments
Labels
bug Something isn't working

Comments

@kand617
Copy link

kand617 commented Apr 26, 2019

Describe the bug
Following https://github.com/aws/aws-app-mesh-examples/blob/master/walkthroughs/eks/base.md

I am unable to perform the curl command from curler to the gateway due to a connection refused error.

I am using us-west-2
Platform
EKS

To Reproduce
I have executed the steps on the walkthrough.

Expected behavior
I expect the curl command to comeback with OK.
The current curl command response is as follows:

curl: (7) Failed to connect to colorgateway port 9080: Connection refused

Config files, and API responses

root@curler2-58cdb99d46-j8zsl:/# ping colorgateway
PING colorgateway.appmesh-demo.svc.cluster.local (10.100.251.99) 56(84) bytes of data.
^C
--- colorgateway.appmesh-demo.svc.cluster.local ping statistics ---
34 packets transmitted, 0 received, 100% packet loss, time 33776ms


SINC02W54EDHTD6:aws-app-mesh-controller-for-k8s andalak$ kubectl get pods -n appmesh-demo
NAME                                 READY   STATUS    RESTARTS   AGE
colorgateway-69cd4fc669-cpkcg        3/3     Running   0          129m
colorteller-845959f54-gkjr6          3/3     Running   0          129m
colorteller-black-6cc98458db-mppn6   3/3     Running   0          129m
colorteller-blue-88bcffddb-r5d59     3/3     Running   0          129m
colorteller-red-6f55b447db-9mxgh     3/3     Running   0          129m
curler-5875dfcc64-49mwf              1/1     Running   1          120m
curler2-58cdb99d46-j8zsl             1/1     Running   3          66m

API Gateway Envoy Logs:

INC02W54EDHTD6:aws-app-mesh-controller-for-k8s andalak$ kubectl -n appmesh-demo logs colorgateway-69cd4fc669-cpkcg envoy
Did not find Envoy configuration file at /tmp/envoy.yaml, creating one.
added envoy.xray tracing config to /tmp/envoy_tracing_config.yaml
Appending /tmp/envoy_tracing_config.yaml to /tmp/envoy.yaml
Appending /tmp/envoy_stats_config.yaml to /tmp/envoy.yaml
Starting Envoy.
[2019-04-26 12:51:53.319][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws_Init_Cleanup Initiate AWS SDK for C++ with Version:1.6.39
[2019-04-26 12:51:53.319][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] CurlHttpClient Initializing Curl library
[2019-04-26 12:51:53.319][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSConfigFileProfileConfigLoader Initializing config loader against fileName //.aws/config and using profilePrefix = 1
[2019-04-26 12:51:53.319][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSConfigFileProfileConfigLoader Unable to open config file //.aws/config for reading.
[2019-04-26 12:51:53.319][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSProfileConfigLoader Failed to reload configuration.
[2019-04-26 12:51:53.327][000001][info][main] [source/server/server.cc:206] initializing epoch 0 (hot restart version=10.200.16384.567.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=9863272)
[2019-04-26 12:51:53.327][000001][info][main] [source/server/server.cc:208] statically linked extensions:
[2019-04-26 12:51:53.327][000001][info][main] [source/server/server.cc:210]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2019-04-26 12:51:53.327][000001][info][main] [source/server/server.cc:213]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2019-04-26 12:51:53.327][000001][info][main] [source/server/server.cc:216]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2019-04-26 12:51:53.327][000001][info][main] [source/server/server.cc:219]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.dubbo_proxy,envoy.filters.network.rbac,envoy.filters.network.sni_cluster,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2019-04-26 12:51:53.327][000001][info][main] [source/server/server.cc:221]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2019-04-26 12:51:53.327][000001][info][main] [source/server/server.cc:223]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.tracers.datadog,envoy.xray,envoy.zipkin
[2019-04-26 12:51:53.327][000001][info][main] [source/server/server.cc:226]   transport_sockets.downstream: envoy.transport_sockets.alts,envoy.transport_sockets.capture,raw_buffer,tls
[2019-04-26 12:51:53.327][000001][info][main] [source/server/server.cc:229]   transport_sockets.upstream: envoy.transport_sockets.alts,envoy.transport_sockets.capture,raw_buffer,tls
[2019-04-26 12:51:53.335][000001][info][main] [source/server/server.cc:271] admin address: 0.0.0.0:9901
[2019-04-26 12:51:53.338][000001][info][config] [source/server/configuration_impl.cc:50] loading 0 static secret(s)
[2019-04-26 12:51:53.338][000001][info][config] [source/server/configuration_impl.cc:56] loading 0 cluster(s)
[2019-04-26 12:51:53.340][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSConfigFileProfileConfigLoader Initializing config loader against fileName //.aws/config and using profilePrefix = 1
[2019-04-26 12:51:53.340][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSConfigFileProfileConfigLoader Initializing config loader against fileName //.aws/credentials and using profilePrefix = 0
[2019-04-26 12:51:53.340][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] ProfileConfigFileAWSCredentialsProvider Setting provider to read credentials from //.aws/credentials for credentials file and //.aws/config for the config file , for use with profile default
[2019-04-26 12:51:53.340][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] EC2MetadataClient Creating AWSHttpResourceClient with max connections2 and scheme http
[2019-04-26 12:51:53.340][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] CurlHandleContainer Initializing CurlHandleContainer with size 2
[2019-04-26 12:51:53.340][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] InstanceProfileCredentialsProvider Creating Instance with default EC2MetadataClient and refresh rate 300000
[2019-04-26 12:51:53.340][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] DefaultAWSCredentialsProviderChain Added EC2 metadata service credentials provider to the provider chain.
[2019-04-26 12:51:53.340][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSConfigFileProfileConfigLoader Unable to open config file //.aws/credentials for reading.
[2019-04-26 12:51:53.340][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSProfileConfigLoader Failed to reload configuration.
[2019-04-26 12:51:53.340][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSConfigFileProfileConfigLoader Unable to open config file //.aws/config for reading.
[2019-04-26 12:51:53.340][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSProfileConfigLoader Failed to reload configuration.
[2019-04-26 12:51:53.340][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] InstanceProfileCredentialsProvider Credentials have expired attempting to repull from EC2 Metadata Service.
[2019-04-26 12:51:53.340][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] CurlHandleContainer Pool grown by 2
[2019-04-26 12:51:53.340][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] CurlHandleContainer Connection has been released. Continuing.
[2019-04-26 12:51:53.341][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] CurlHandleContainer Connection has been released. Continuing.
[2019-04-26 12:51:53.343][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::EC2InstanceProfileConfigLoader Successfully pulled credentials from metadata service with access key ASIAYFZHB6A2OCGTODFH
[2019-04-26 12:51:53.343][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] CurlHandleContainer Connection has been released. Continuing.
[2019-04-26 12:51:53.345][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] EC2MetadataClient Detected current region as us-west-2
[2019-04-26 12:51:53.345][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSProfileConfigLoader Successfully reloaded configuration.
[2019-04-26 12:51:53.346][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:132] cm init: initializing cds
[2019-04-26 12:51:53.346][000001][info][config] [source/server/configuration_impl.cc:67] loading 0 listener(s)
[2019-04-26 12:51:53.346][000001][info][config] [source/server/configuration_impl.cc:92] loading tracing configuration
[2019-04-26 12:51:53.346][000001][info][config] [source/server/configuration_impl.cc:101]   loading tracing driver: envoy.xray
[2019-04-26 12:51:53.346][000001][info][tracing] [source/extensions/tracers/xray/xray_tracer_impl.cc:95] send X-Ray generated segments to daemon address on 127.0.0.1:2000
[2019-04-26 12:51:53.346][000001][info][tracing] [source/extensions/tracers/xray/sampling.cc:114] unable to parse empty json file. falling back to default rule set.
[2019-04-26 12:51:53.347][000001][info][config] [source/server/configuration_impl.cc:112] loading stats sink configuration
[2019-04-26 12:51:53.347][000001][info][main] [source/server/server.cc:463] starting main dispatch loop

SINC02W54EDHTD6:aws-app-mesh-controller-for-k8s andalak$ kubectl -n appmesh-demo logs colorgateway-69cd4fc669-cpkcg colorgateway
2019/04/26 12:51:46 starting server, listening on port 9080
2019/04/26 12:51:46 using color-teller at colorteller.appmesh-demo:9080
```

**Additional context**
Here is the full `pods describe -n appmesh-demo`
Name:               colorgateway-69cd4fc669-cpkcg
Namespace:          appmesh-demo
Priority:           0
PriorityClassName:  <none>
Node:               ip-192-168-13-9.us-west-2.compute.internal/192.168.13.9
Start Time:         Fri, 26 Apr 2019 20:51:35 +0800
Labels:             app=colorgateway
                    pod-template-hash=69cd4fc669
                    version=v1
Annotations:        <none>
Status:             Running
IP:                 192.168.7.186
Controlled By:      ReplicaSet/colorgateway-69cd4fc669
Init Containers:
  proxyinit:
    Container ID:   docker://d9522182a2d9d2402e9a2748f220da972f926d8817c6a78aa287168929dc5114
    Image:          111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-proxy-route-manager:latest
    Image ID:       docker-pullable://111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-proxy-route-manager@sha256:a055da31668a5dc6e68da49c4a8217726d8437e2a94ce6bb6a15abfdcbb1e925
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 26 Apr 2019 20:51:43 +0800
      Finished:     Fri, 26 Apr 2019 20:51:43 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      APPMESH_START_ENABLED:       1
      APPMESH_IGNORE_UID:          1337
      APPMESH_ENVOY_INGRESS_PORT:  15000
      APPMESH_ENVOY_EGRESS_PORT:   15001
      APPMESH_APP_PORTS:           9080
      APPMESH_EGRESS_IGNORED_IP:   169.254.169.254
    Mounts:                        <none>
Containers:
  colorgateway:
    Container ID:   docker://62afda07fe5974d17f184f95975af9e8926b3508e3194ca05aa047796175190e
    Image:          970805265562.dkr.ecr.us-west-2.amazonaws.com/gateway:latest
    Image ID:       docker-pullable://970805265562.dkr.ecr.us-west-2.amazonaws.com/gateway@sha256:2a81ac9f74a20a02e32b7f56848eeeb99d0c5d47c73d92e484dd7875ed057780
    Port:           9080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 26 Apr 2019 20:51:46 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      SERVER_PORT:            9080
      COLOR_TELLER_ENDPOINT:  colorteller.appmesh-demo:9080
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-22pk6 (ro)
  envoy:
    Container ID:   docker://6ad12dfd8ff63771b713355d35403219df025a0c15587b1c673d128777172394
    Image:          111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-envoy:v1.9.0.0-prod
    Image ID:       docker-pullable://111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-envoy@sha256:33e19cf3106b2ccb1ccc3f1d28b7e5b965d640f2a17a5c5564780720f63f258f
    Port:           9901/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 26 Apr 2019 20:51:53 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     10m
      memory:  32Mi
    Environment:
      APPMESH_VIRTUAL_NODE_NAME:  mesh/color-mesh/virtualNode/colorgateway-appmesh-demo
      ENVOY_LOG_LEVEL:            info
      AWS_REGION:                 us-west-2
      ENABLE_ENVOY_XRAY_TRACING:  1
      ENABLE_ENVOY_STATS_TAGS:    1
    Mounts:                       <none>
  xray-daemon:
    Container ID:   docker://ceb3d11fef37a4981752b1f0aff6dbde85da50122922d6376259b8b63b989cff
    Image:          amazon/aws-xray-daemon
    Image ID:       docker-pullable://amazon/aws-xray-daemon@sha256:0f2270a1aa8e02acd735d0ec053b4aa554dbee3fc90614617e85509f8168663e
    Port:           2000/UDP
    Host Port:      0/UDP
    State:          Running
      Started:      Fri, 26 Apr 2019 20:51:57 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        10m
      memory:     32Mi
    Environment:  <none>
    Mounts:       <none>
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-22pk6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-22pk6
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>


Name:               colorteller-845959f54-gkjr6
Namespace:          appmesh-demo
Priority:           0
PriorityClassName:  <none>
Node:               ip-192-168-78-215.us-west-2.compute.internal/192.168.78.215
Start Time:         Fri, 26 Apr 2019 20:51:37 +0800
Labels:             app=colorteller
                    pod-template-hash=845959f54
                    version=white
Annotations:        <none>
Status:             Running
IP:                 192.168.73.201
Controlled By:      ReplicaSet/colorteller-845959f54
Init Containers:
  proxyinit:
    Container ID:   docker://ed1d4850078a517ff3e47e4ada7936f0677f5e7b464c10dd031d97376d73af68
    Image:          111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-proxy-route-manager:latest
    Image ID:       docker-pullable://111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-proxy-route-manager@sha256:a055da31668a5dc6e68da49c4a8217726d8437e2a94ce6bb6a15abfdcbb1e925
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 26 Apr 2019 20:51:45 +0800
      Finished:     Fri, 26 Apr 2019 20:51:45 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      APPMESH_START_ENABLED:       1
      APPMESH_IGNORE_UID:          1337
      APPMESH_ENVOY_INGRESS_PORT:  15000
      APPMESH_ENVOY_EGRESS_PORT:   15001
      APPMESH_APP_PORTS:           9080
      APPMESH_EGRESS_IGNORED_IP:   169.254.169.254
    Mounts:                        <none>
Containers:
  colorteller:
    Container ID:   docker://1ec509fbbe9c1b4ff5916af8e4e4cecb0e8a77f8ddd12785e1aba13e2e8dba6a
    Image:          970805265562.dkr.ecr.us-west-2.amazonaws.com/colorteller:latest
    Image ID:       docker-pullable://970805265562.dkr.ecr.us-west-2.amazonaws.com/colorteller@sha256:2c292abca87af64ad5380b6e1e3f621a982e75a0c848656dd5286531994b119a
    Port:           9080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 26 Apr 2019 20:51:47 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      SERVER_PORT:  9080
      COLOR:        white
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-22pk6 (ro)
  envoy:
    Container ID:   docker://e50d408ef38bfd67cf73a7b9bdc0215c2af6452e8f547ce95be78fabfad1f482
    Image:          111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-envoy:v1.9.0.0-prod
    Image ID:       docker-pullable://111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-envoy@sha256:33e19cf3106b2ccb1ccc3f1d28b7e5b965d640f2a17a5c5564780720f63f258f
    Port:           9901/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 26 Apr 2019 20:51:55 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     10m
      memory:  32Mi
    Environment:
      APPMESH_VIRTUAL_NODE_NAME:  mesh/color-mesh/virtualNode/colorteller-appmesh-demo
      ENVOY_LOG_LEVEL:            info
      AWS_REGION:                 us-west-2
      ENABLE_ENVOY_XRAY_TRACING:  1
      ENABLE_ENVOY_STATS_TAGS:    1
    Mounts:                       <none>
  xray-daemon:
    Container ID:   docker://6c0bd345d8f75de07111f8f7599563e8d77511eae692a28186f8b9a66a5898d5
    Image:          amazon/aws-xray-daemon
    Image ID:       docker-pullable://amazon/aws-xray-daemon@sha256:0f2270a1aa8e02acd735d0ec053b4aa554dbee3fc90614617e85509f8168663e
    Port:           2000/UDP
    Host Port:      0/UDP
    State:          Running
      Started:      Fri, 26 Apr 2019 20:52:00 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        10m
      memory:     32Mi
    Environment:  <none>
    Mounts:       <none>
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-22pk6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-22pk6
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>


Name:               colorteller-black-6cc98458db-mppn6
Namespace:          appmesh-demo
Priority:           0
PriorityClassName:  <none>
Node:               ip-192-168-13-9.us-west-2.compute.internal/192.168.13.9
Start Time:         Fri, 26 Apr 2019 20:51:38 +0800
Labels:             app=colorteller
                    pod-template-hash=6cc98458db
                    version=black
Annotations:        <none>
Status:             Running
IP:                 192.168.12.200
Controlled By:      ReplicaSet/colorteller-black-6cc98458db
Init Containers:
  proxyinit:
    Container ID:   docker://d5bf1eeb97829e5118886fb5a8b7c239901cb690d7b58c83a766377f4b8507a2
    Image:          111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-proxy-route-manager:latest
    Image ID:       docker-pullable://111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-proxy-route-manager@sha256:a055da31668a5dc6e68da49c4a8217726d8437e2a94ce6bb6a15abfdcbb1e925
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 26 Apr 2019 20:51:43 +0800
      Finished:     Fri, 26 Apr 2019 20:51:43 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      APPMESH_START_ENABLED:       1
      APPMESH_IGNORE_UID:          1337
      APPMESH_ENVOY_INGRESS_PORT:  15000
      APPMESH_ENVOY_EGRESS_PORT:   15001
      APPMESH_APP_PORTS:           9080
      APPMESH_EGRESS_IGNORED_IP:   169.254.169.254
    Mounts:                        <none>
Containers:
  colorteller:
    Container ID:   docker://f47a9d55beb649fb7ff0d4753b48d8cd2efac9e838831d9033b46cf868801c4f
    Image:          970805265562.dkr.ecr.us-west-2.amazonaws.com/colorteller:latest
    Image ID:       docker-pullable://970805265562.dkr.ecr.us-west-2.amazonaws.com/colorteller@sha256:2c292abca87af64ad5380b6e1e3f621a982e75a0c848656dd5286531994b119a
    Port:           9080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 26 Apr 2019 20:51:45 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      SERVER_PORT:  9080
      COLOR:        black
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-22pk6 (ro)
  envoy:
    Container ID:   docker://bdbff9db3e81d2e8581776382b2bb7b103e27646da55bf7f383cf07a1ef899e7
    Image:          111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-envoy:v1.9.0.0-prod
    Image ID:       docker-pullable://111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-envoy@sha256:33e19cf3106b2ccb1ccc3f1d28b7e5b965d640f2a17a5c5564780720f63f258f
    Port:           9901/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 26 Apr 2019 20:51:53 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     10m
      memory:  32Mi
    Environment:
      APPMESH_VIRTUAL_NODE_NAME:  mesh/color-mesh/virtualNode/colorteller-black-appmesh-demo
      ENVOY_LOG_LEVEL:            info
      AWS_REGION:                 us-west-2
      ENABLE_ENVOY_XRAY_TRACING:  1
      ENABLE_ENVOY_STATS_TAGS:    1
    Mounts:                       <none>
  xray-daemon:
    Container ID:   docker://0cbbdb51d5ee4bbc73ed6bc612597a9bd20423c8fae6124ffa0626c8f12d633c
    Image:          amazon/aws-xray-daemon
    Image ID:       docker-pullable://amazon/aws-xray-daemon@sha256:0f2270a1aa8e02acd735d0ec053b4aa554dbee3fc90614617e85509f8168663e
    Port:           2000/UDP
    Host Port:      0/UDP
    State:          Running
      Started:      Fri, 26 Apr 2019 20:51:56 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        10m
      memory:     32Mi
    Environment:  <none>
    Mounts:       <none>
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-22pk6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-22pk6
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>


Name:               colorteller-blue-88bcffddb-r5d59
Namespace:          appmesh-demo
Priority:           0
PriorityClassName:  <none>
Node:               ip-192-168-78-215.us-west-2.compute.internal/192.168.78.215
Start Time:         Fri, 26 Apr 2019 20:51:40 +0800
Labels:             app=colorteller
                    pod-template-hash=88bcffddb
                    version=blue
Annotations:        <none>
Status:             Running
IP:                 192.168.75.251
Controlled By:      ReplicaSet/colorteller-blue-88bcffddb
Init Containers:
  proxyinit:
    Container ID:   docker://079f45d867648a99287a5aa7358fbb8a366583141e303d4538c08ea0f80d8a9f
    Image:          111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-proxy-route-manager:latest
    Image ID:       docker-pullable://111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-proxy-route-manager@sha256:a055da31668a5dc6e68da49c4a8217726d8437e2a94ce6bb6a15abfdcbb1e925
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 26 Apr 2019 20:51:45 +0800
      Finished:     Fri, 26 Apr 2019 20:51:45 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      APPMESH_START_ENABLED:       1
      APPMESH_IGNORE_UID:          1337
      APPMESH_ENVOY_INGRESS_PORT:  15000
      APPMESH_ENVOY_EGRESS_PORT:   15001
      APPMESH_APP_PORTS:           9080
      APPMESH_EGRESS_IGNORED_IP:   169.254.169.254
    Mounts:                        <none>
Containers:
  colorteller:
    Container ID:   docker://7da40e404871970650fcef74f84611e69c173828edd923c6d7a6c0f8226b2083
    Image:          970805265562.dkr.ecr.us-west-2.amazonaws.com/colorteller:latest
    Image ID:       docker-pullable://970805265562.dkr.ecr.us-west-2.amazonaws.com/colorteller@sha256:2c292abca87af64ad5380b6e1e3f621a982e75a0c848656dd5286531994b119a
    Port:           9080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 26 Apr 2019 20:51:47 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      SERVER_PORT:  9080
      COLOR:        blue
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-22pk6 (ro)
  envoy:
    Container ID:   docker://a958fd9e85555e7642429c97394a5993156f2c47caa2d7b7eb933041c4cf4c86
    Image:          111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-envoy:v1.9.0.0-prod
    Image ID:       docker-pullable://111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-envoy@sha256:33e19cf3106b2ccb1ccc3f1d28b7e5b965d640f2a17a5c5564780720f63f258f
    Port:           9901/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 26 Apr 2019 20:51:55 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     10m
      memory:  32Mi
    Environment:
      APPMESH_VIRTUAL_NODE_NAME:  mesh/color-mesh/virtualNode/colorteller-blue-appmesh-demo
      ENVOY_LOG_LEVEL:            info
      AWS_REGION:                 us-west-2
      ENABLE_ENVOY_XRAY_TRACING:  1
      ENABLE_ENVOY_STATS_TAGS:    1
    Mounts:                       <none>
  xray-daemon:
    Container ID:   docker://48527440706d8a2ff65abce59d8c67aaed3c7602ab88ce7c10587069ef8290b2
    Image:          amazon/aws-xray-daemon
    Image ID:       docker-pullable://amazon/aws-xray-daemon@sha256:0f2270a1aa8e02acd735d0ec053b4aa554dbee3fc90614617e85509f8168663e
    Port:           2000/UDP
    Host Port:      0/UDP
    State:          Running
      Started:      Fri, 26 Apr 2019 20:51:59 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        10m
      memory:     32Mi
    Environment:  <none>
    Mounts:       <none>
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-22pk6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-22pk6
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>


Name:               colorteller-red-6f55b447db-9mxgh
Namespace:          appmesh-demo
Priority:           0
PriorityClassName:  <none>
Node:               ip-192-168-78-215.us-west-2.compute.internal/192.168.78.215
Start Time:         Fri, 26 Apr 2019 20:51:42 +0800
Labels:             app=colorteller
                    pod-template-hash=6f55b447db
                    version=red
Annotations:        <none>
Status:             Running
IP:                 192.168.67.134
Controlled By:      ReplicaSet/colorteller-red-6f55b447db
Init Containers:
  proxyinit:
    Container ID:   docker://e73ab4b2de338f3a80dd896e7aeae4e06de6576281cf841c8709f1b8e1c2f925
    Image:          111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-proxy-route-manager:latest
    Image ID:       docker-pullable://111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-proxy-route-manager@sha256:a055da31668a5dc6e68da49c4a8217726d8437e2a94ce6bb6a15abfdcbb1e925
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 26 Apr 2019 20:51:45 +0800
      Finished:     Fri, 26 Apr 2019 20:51:45 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      APPMESH_START_ENABLED:       1
      APPMESH_IGNORE_UID:          1337
      APPMESH_ENVOY_INGRESS_PORT:  15000
      APPMESH_ENVOY_EGRESS_PORT:   15001
      APPMESH_APP_PORTS:           9080
      APPMESH_EGRESS_IGNORED_IP:   169.254.169.254
    Mounts:                        <none>
Containers:
  colorteller:
    Container ID:   docker://079e59d3572e54a14723ae2c464ae1633185313bd0bf0458ce2c8776926e176f
    Image:          970805265562.dkr.ecr.us-west-2.amazonaws.com/colorteller:latest
    Image ID:       docker-pullable://970805265562.dkr.ecr.us-west-2.amazonaws.com/colorteller@sha256:2c292abca87af64ad5380b6e1e3f621a982e75a0c848656dd5286531994b119a
    Port:           9080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 26 Apr 2019 20:51:47 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      SERVER_PORT:  9080
      COLOR:        red
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-22pk6 (ro)
  envoy:
    Container ID:   docker://31113fdad5a925601e9b4e6606082b4fe759aafdba94a005089a2cf275933a21
    Image:          111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-envoy:v1.9.0.0-prod
    Image ID:       docker-pullable://111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-envoy@sha256:33e19cf3106b2ccb1ccc3f1d28b7e5b965d640f2a17a5c5564780720f63f258f
    Port:           9901/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 26 Apr 2019 20:51:55 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     10m
      memory:  32Mi
    Environment:
      APPMESH_VIRTUAL_NODE_NAME:  mesh/color-mesh/virtualNode/colorteller-red-appmesh-demo
      ENVOY_LOG_LEVEL:            info
      AWS_REGION:                 us-west-2
      ENABLE_ENVOY_XRAY_TRACING:  1
      ENABLE_ENVOY_STATS_TAGS:    1
    Mounts:                       <none>
  xray-daemon:
    Container ID:   docker://4cf59385be1fc3f6996d9ab85891916a46f83ac0e81b30e5e29f3986d5e7eec3
    Image:          amazon/aws-xray-daemon
    Image ID:       docker-pullable://amazon/aws-xray-daemon@sha256:0f2270a1aa8e02acd735d0ec053b4aa554dbee3fc90614617e85509f8168663e
    Port:           2000/UDP
    Host Port:      0/UDP
    State:          Running
      Started:      Fri, 26 Apr 2019 20:52:01 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        10m
      memory:     32Mi
    Environment:  <none>
    Mounts:       <none>
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-22pk6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-22pk6
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>


Name:               curler-5875dfcc64-49mwf
Namespace:          appmesh-demo
Priority:           0
PriorityClassName:  <none>
Node:               ip-192-168-13-9.us-west-2.compute.internal/192.168.13.9
Start Time:         Fri, 26 Apr 2019 20:59:58 +0800
Labels:             pod-template-hash=5875dfcc64
                    run=curler
Annotations:        <none>
Status:             Running
IP:                 192.168.5.205
Controlled By:      ReplicaSet/curler-5875dfcc64
Containers:
  curler:
    Container ID:  docker://a0cb1d72fda692d750141de3cfd4bd65a45799e6d5e685393911d35f31b19abe
    Image:         tutum/curl
    Image ID:      docker-pullable://tutum/curl@sha256:b6f16e88387acd4e6326176b212b3dae63f5b2134e69560d0b0673cfb0fb976f
    Port:          <none>
    Host Port:     <none>
    Args:
      /bin/bash
    State:          Running
      Started:      Fri, 26 Apr 2019 21:04:44 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    130
      Started:      Fri, 26 Apr 2019 21:00:06 +0800
      Finished:     Fri, 26 Apr 2019 21:04:43 +0800
    Ready:          True
    Restart Count:  1
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-22pk6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-22pk6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-22pk6
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>


Name:               curler2-58cdb99d46-j8zsl
Namespace:          appmesh-demo
Priority:           0
PriorityClassName:  <none>
Node:               ip-192-168-13-9.us-west-2.compute.internal/192.168.13.9
Start Time:         Fri, 26 Apr 2019 21:53:54 +0800
Labels:             pod-template-hash=58cdb99d46
                    run=curler2
Annotations:        <none>
Status:             Running
IP:                 192.168.3.88
Controlled By:      ReplicaSet/curler2-58cdb99d46
Containers:
  curler2:
    Container ID:  docker://4f74aa93d3a7b68adc0425ee64ef74c73dccdab42257ae8ea27d81aac3fc8ec7
    Image:         tutum/curl
    Image ID:      docker-pullable://tutum/curl@sha256:b6f16e88387acd4e6326176b212b3dae63f5b2134e69560d0b0673cfb0fb976f
    Port:          <none>
    Host Port:     <none>
    Args:
      /bin/bash
    State:          Running
      Started:      Fri, 26 Apr 2019 23:00:35 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 26 Apr 2019 22:35:45 +0800
      Finished:     Fri, 26 Apr 2019 23:00:32 +0800
    Ready:          True
    Restart Count:  3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-22pk6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-22pk6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-22pk6
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason   Age                  From                                                 Message
  ----    ------   ----                 ----                                                 -------
  Normal  Pulling  3m52s (x4 over 70m)  kubelet, ip-192-168-13-9.us-west-2.compute.internal  pulling image "tutum/curl"
  Normal  Pulled   3m51s (x4 over 70m)  kubelet, ip-192-168-13-9.us-west-2.compute.internal  Successfully pulled image "tutum/curl"
  Normal  Created  3m50s (x4 over 70m)  kubelet, ip-192-168-13-9.us-west-2.compute.internal  Created container
  Normal  Started  3m50s (x4 over 70m)  kubelet, ip-192-168-13-9.us-west-2.compute.internal  Started container
@kand617 kand617 added the bug Something isn't working label Apr 26, 2019
@kand617
Copy link
Author

kand617 commented Apr 27, 2019

I have repeated the entire set of instructions using us-east-2 and still get the same error.
Rather annoying to be honest.

Here is the full output

SINC02W54EDHTD6:aws-app-mesh-examples andalak$ eksctl create cluster --name appmeshtest --version 1.12 --nodes-min 2 --nodes-max 3 --nodes 2 --auto-kubeconfig --full-ecr-access --appmesh-access
[ℹ]  using region us-east-2
[ℹ]  setting availability zones to [us-east-2a us-east-2c us-east-2b]
[ℹ]  subnets for us-east-2a - public:192.168.0.0/19 private:192.168.96.0/19
[ℹ]  subnets for us-east-2c - public:192.168.32.0/19 private:192.168.128.0/19
[ℹ]  subnets for us-east-2b - public:192.168.64.0/19 private:192.168.160.0/19
[ℹ]  nodegroup "ng-5531c2ff" will use "ami-04ea7cb66af82ae4a" [AmazonLinux2/1.12]
[ℹ]  creating EKS cluster "appmeshtest" in "us-east-2" region
[ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-east-2 --name=appmeshtest'
[ℹ]  2 sequential tasks: { create cluster control plane "appmeshtest", create nodegroup "ng-5531c2ff" }
[ℹ]  building cluster stack "eksctl-appmeshtest-cluster"
[ℹ]  deploying stack "eksctl-appmeshtest-cluster"
[ℹ]  buildings nodegroup stack "eksctl-appmeshtest-nodegroup-ng-5531c2ff"
[ℹ]  deploying stack "eksctl-appmeshtest-nodegroup-ng-5531c2ff"
[✔]  all EKS cluster resource for "appmeshtest" had been created
[✔]  saved kubeconfig as "/Users/andalak/.kube/eksctl/clusters/appmeshtest"
[ℹ]  adding role "arn:aws:iam::562185760820:role/eksctl-appmeshtest-nodegroup-ng-5-NodeInstanceRole-12WS57YBIBF9W" to auth ConfigMap
[ℹ]  nodegroup "ng-5531c2ff" has 0 node(s)
[ℹ]  waiting for at least 2 node(s) to become ready in "ng-5531c2ff"
[ℹ]  nodegroup "ng-5531c2ff" has 2 node(s)
[ℹ]  node "ip-192-168-46-96.us-east-2.compute.internal" is ready
[ℹ]  node "ip-192-168-64-68.us-east-2.compute.internal" is ready
[ℹ]  kubectl command should work with "/Users/andalak/.kube/eksctl/clusters/appmeshtest", try 'kubectl --kubeconfig=/Users/andalak/.kube/eksctl/clusters/appmeshtest get nodes'
[✔]  EKS cluster "appmeshtest" in "us-east-2" region is ready
SINC02W54EDHTD6:aws-app-mesh-examples andalak$ export KUBECONFIG=~/.kube/eksctl/clusters/appmeshtest
SINC02W54EDHTD6:aws-app-mesh-examples andalak$ export MESH_NAME=color-mesh
SINC02W54EDHTD6:aws-app-mesh-examples andalak$ export INJECT_XRAY_SIDECAR=true
SINC02W54EDHTD6:aws-app-mesh-examples andalak$ export ENABLE_STATS_TAGS=true
SINC02W54EDHTD6:aws-app-mesh-examples andalak$ export MESH_REGION=us-east-2
SINC02W54EDHTD6:aws-app-mesh-examples andalak$ curl https://raw.githubusercontent.com/aws/aws-app-mesh-inject/v0.1.3/hack/install.sh | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1164  100  1164    0     0   2443      0 --:--:-- --:--:-- --:--:--  2440
\nWorking directory at /var/folders/by/_mqxmh194bn7v0whkmd_zk4m0000gn/T/tmp.7d4DSVqh\n
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    68  100    68    0     0    196      0 --:--:-- --:--:-- --:--:--   196
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3489  100  3489    0     0   9815      0 --:--:-- --:--:-- --:--:--  9828
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2200  100  2200    0     0   6261      0 --:--:-- --:--:-- --:--:--  6250
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   622  100   622    0     0   1725      0 --:--:-- --:--:-- --:--:--  1727
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1076  100  1076    0     0   2988      0 --:--:-- --:--:-- --:--:--  2988
namespace/appmesh-inject created
creating certs in tmpdir /var/folders/by/_mqxmh194bn7v0whkmd_zk4m0000gn/T/tmp.2ZAfYtHu 
Generating RSA private key, 2048 bit long modulus
................................................................+++
..........................................................................+++
e is 65537 (0x10001)
certificatesigningrequest.certificates.k8s.io/aws-app-mesh-inject.appmesh-inject created
NAME                                 AGE   REQUESTOR          CONDITION
aws-app-mesh-inject.appmesh-inject   2s    kubernetes-admin   Pending
certificatesigningrequest.certificates.k8s.io/aws-app-mesh-inject.appmesh-inject approved
secret/aws-app-mesh-inject created
processing templates
Created injector manifest at:/var/folders/by/_mqxmh194bn7v0whkmd_zk4m0000gn/T/tmp.7d4DSVqh/_output/inject.yaml
serviceaccount/aws-app-mesh-inject-sa created
clusterrole.rbac.authorization.k8s.io/aws-app-mesh-inject-cr created
clusterrolebinding.rbac.authorization.k8s.io/aws-app-mesh-inject-binding created
service/aws-app-mesh-inject created
deployment.apps/aws-app-mesh-inject created
mutatingwebhookconfiguration.admissionregistration.k8s.io/aws-app-mesh-inject created
waiting for aws-app-mesh-inject to start
Waiting for deployment "aws-app-mesh-inject" rollout to finish: 0 of 1 updated replicas are available...
deployment "aws-app-mesh-inject" successfully rolled out
Mesh name has been set up
The injector is ready
SINC02W54EDHTD6:aws-app-mesh-examples andalak$ kubectl -n appmesh-inject get po
NAME                                   READY   STATUS    RESTARTS   AGE
aws-app-mesh-inject-86859c84f9-7lclq   1/1     Running   0          40s
SINC02W54EDHTD6:aws-app-mesh-examples andalak$ curl https://raw.githubusercontent.com/aws/aws-app-mesh-controller-for-k8s/v0.1.0/deploy/v0.1.0/all.yaml | kubectl apply -f -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8419  100  8419    0     0  22104      0 --:--:-- --:--:-- --:--:-- 22097
customresourcedefinition.apiextensions.k8s.io/meshes.appmesh.k8s.aws created
customresourcedefinition.apiextensions.k8s.io/virtualnodes.appmesh.k8s.aws created
customresourcedefinition.apiextensions.k8s.io/virtualservices.appmesh.k8s.aws created
namespace/appmesh-system created
deployment.apps/app-mesh-controller created
serviceaccount/app-mesh-sa created
clusterrole.rbac.authorization.k8s.io/app-mesh-controller created
clusterrolebinding.rbac.authorization.k8s.io/app-mesh-controller-binding created
SINC02W54EDHTD6:aws-app-mesh-examples andalak$ kubectl wait $(kubectl get pods -n appmesh-system -o name) \
>           --for=condition=Ready --timeout=30s -n appmesh-system
pod/app-mesh-controller-65897498cb-284qt condition met
SINC02W54EDHTD6:aws-app-mesh-examples andalak$ kubectl api-resources --api-group=appmesh.k8s.aws
NAME              SHORTNAMES   APIGROUP          NAMESPACED   KIND
meshes                         appmesh.k8s.aws   false        Mesh
virtualnodes                   appmesh.k8s.aws   true         VirtualNode
virtualservices                appmesh.k8s.aws   true         VirtualService
SINC02W54EDHTD6:aws-app-mesh-examples andalak$ ls
CODE_OF_CONDUCT.md      FAQ.md                  NOTICE                  examples
CONTRIBUTING.md         LICENSE                 README.md               walkthroughs
SINC02W54EDHTD6:aws-app-mesh-examples andalak$ cd walkthroughs/eks/
SINC02W54EDHTD6:eks andalak$ cd aws-app-mesh-controller-for-k8s/
SINC02W54EDHTD6:aws-app-mesh-controller-for-k8s andalak$ make example
./hack/example.sh
namespace/appmesh-demo created
mesh.appmesh.k8s.aws/color-mesh created
virtualnode.appmesh.k8s.aws/colorgateway created
virtualnode.appmesh.k8s.aws/colorteller created
virtualnode.appmesh.k8s.aws/colorteller-black created
virtualnode.appmesh.k8s.aws/colorteller-blue created
virtualnode.appmesh.k8s.aws/colorteller-red created
virtualservice.appmesh.k8s.aws/colorteller.appmesh-demo created
virtualservice.appmesh.k8s.aws/colorgateway.appmesh-demo created
service/colorgateway created
deployment.apps/colorgateway created
service/colorteller created
deployment.apps/colorteller created
service/colorteller-black created
deployment.apps/colorteller-black created
service/colorteller-blue created
deployment.apps/colorteller-blue created
service/colorteller-red created
deployment.apps/colorteller-red created
SINC02W54EDHTD6:aws-app-mesh-controller-for-k8s andalak$ kubectl -n appmesh-demo \
>           get deploy,po,svc,virtualnode.appmesh.k8s.aws,virtualservice.appmesh.k8s.aws
NAME                                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.extensions/colorgateway        1         1         1            1           42s
deployment.extensions/colorteller         1         1         1            1           40s
deployment.extensions/colorteller-black   1         1         1            1           38s
deployment.extensions/colorteller-blue    1         1         1            1           36s
deployment.extensions/colorteller-red     1         1         1            1           34s

NAME                                     READY   STATUS    RESTARTS   AGE
pod/colorgateway-69cd4fc669-5d5vl        3/3     Running   0          42s
pod/colorteller-845959f54-cwd67          3/3     Running   0          40s
pod/colorteller-black-6cc98458db-5jslx   3/3     Running   0          38s
pod/colorteller-blue-88bcffddb-t7h4f     3/3     Running   0          36s
pod/colorteller-red-6f55b447db-zxmx2     3/3     Running   0          34s

NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/colorgateway        ClusterIP   10.100.146.35    <none>        9080/TCP   44s
service/colorteller         ClusterIP   10.100.151.85    <none>        9080/TCP   42s
service/colorteller-black   ClusterIP   10.100.68.10     <none>        9080/TCP   40s
service/colorteller-blue    ClusterIP   10.100.227.111   <none>        9080/TCP   38s
service/colorteller-red     ClusterIP   10.100.82.10     <none>        9080/TCP   36s

NAME                                            AGE
virtualnode.appmesh.k8s.aws/colorgateway        50s
virtualnode.appmesh.k8s.aws/colorteller         49s
virtualnode.appmesh.k8s.aws/colorteller-black   48s
virtualnode.appmesh.k8s.aws/colorteller-blue    47s
virtualnode.appmesh.k8s.aws/colorteller-red     46s

NAME                                                       AGE
virtualservice.appmesh.k8s.aws/colorgateway.appmesh-demo   45s
virtualservice.appmesh.k8s.aws/colorteller.appmesh-demo    45s

SINC02W54EDHTD6:aws-app-mesh-controller-for-k8s andalak$ aws appmesh list-meshes --region us-east-2
{
    "meshes": [
        {
            "meshName": "color-mesh", 
            "arn": "arn:aws:appmesh:us-east-2:562185760820:mesh/color-mesh"
        }
    ]
}
SINC02W54EDHTD6:aws-app-mesh-controller-for-k8s andalak$ aws appmesh list-virtual-services \
>       --mesh-name color-mesh \
>       --region us-east-2
{
    "virtualServices": [
        {
            "meshName": "color-mesh", 
            "virtualServiceName": "colorgateway.appmesh-demo", 
            "arn": "arn:aws:appmesh:us-east-2:562185760820:mesh/color-mesh/virtualService/colorgateway.appmesh-demo"
        }, 
        {
            "meshName": "color-mesh", 
            "virtualServiceName": "colorteller.appmesh-demo", 
            "arn": "arn:aws:appmesh:us-east-2:562185760820:mesh/color-mesh/virtualService/colorteller.appmesh-demo"
        }
    ]
}
SINC02W54EDHTD6:aws-app-mesh-controller-for-k8s andalak$ aws appmesh list-virtual-nodes \
>       --mesh-name color-mesh \
>       --region us-east-2
{
    "virtualNodes": [
        {
            "meshName": "color-mesh", 
            "arn": "arn:aws:appmesh:us-east-2:562185760820:mesh/color-mesh/virtualNode/colorgateway-appmesh-demo", 
            "virtualNodeName": "colorgateway-appmesh-demo"
        }, 
        {
            "meshName": "color-mesh", 
            "arn": "arn:aws:appmesh:us-east-2:562185760820:mesh/color-mesh/virtualNode/colorteller-appmesh-demo", 
            "virtualNodeName": "colorteller-appmesh-demo"
        }, 
        {
            "meshName": "color-mesh", 
            "arn": "arn:aws:appmesh:us-east-2:562185760820:mesh/color-mesh/virtualNode/colorteller-black-appmesh-demo", 
            "virtualNodeName": "colorteller-black-appmesh-demo"
        }, 
        {
            "meshName": "color-mesh", 
            "arn": "arn:aws:appmesh:us-east-2:562185760820:mesh/color-mesh/virtualNode/colorteller-red-appmesh-demo", 
            "virtualNodeName": "colorteller-red-appmesh-demo"
        }, 
        {
            "meshName": "color-mesh", 
            "arn": "arn:aws:appmesh:us-east-2:562185760820:mesh/color-mesh/virtualNode/colorteller-blue-appmesh-demo", 
            "virtualNodeName": "colorteller-blue-appmesh-demo"
        }
    ]
}
SINC02W54EDHTD6:aws-app-mesh-controller-for-k8s andalak$ kubectl -n appmesh-demo \
>           run -it curler \
>           --image=tutum/curl /bin/bash
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.

root@curler-5875dfcc64-2x2hj:/# 
root@curler-5875dfcc64-2x2hj:/# curl colorgateway:9080/color
curl: (7) Failed to connect to colorgateway port 9080: Connection refused

@kiranmeduri
Copy link
Contributor

Can you confirm if the following works?

root@curler-5875dfcc64-2x2hj:/# curl http://colorgateway.appmesh-demo:9080/color

@kiranmeduri
Copy link
Contributor

You may see errors due to any of the following. But these errors should go away once the configurations is distributed.

  1. service-discovery is not ready yet. So colorgateway is not resolving to any ip.
  2. colorgateway envoy is not primed yet (i.e. Envoy config is not distributed or loaded yet).
  3. colorgateway app container is not ready yet.

I noticed a different behavior that maps to (2) and (3)

root@curler-5b467f98bb-9zq5p:/#  curl colorgateway:9080/color
500 - Unexpected Error
...
...
root@curler-5b467f98bb-9zq5p:/#  curl colorgateway:9080/color
500 - Unexpected Error
root@curler-5b467f98bb-9zq5p:/#  curl colorgateway:9080/color
{"color":"upstream connect error or disconnect/reset before headers", "stats": {"upstream connect error or disconnect/reset before headers":1}}
...
root@curler-5b467f98bb-9zq5p:/#  curl colorgateway:9080/color
{"color":"black", "stats": {"black":0.33,"upstream connect error or disconnect/reset before headers":0.67}}
 ....
root@curler-5b467f98bb-9zq5p:/#  curl colorgateway:9080/color
{"color":"blue", "stats": {"black":0.38,"blue":0.29,"upstream connect error or disconnect/reset before headers":0.01,"white":0.32}}

@rothgar
Copy link
Contributor

rothgar commented Apr 29, 2019

I appear to be having the same problem (also in us-west-2)

I can dig the endpoint but telnet also gets connection refused. I thought maybe it was a network policy but there aren't any

kubectl get netpol --all-namespaces
No resources found.

@mhausenblas
Copy link
Member

@kand617 @rothgar thanks for reporting this! I can't reproduce it but looking into it.

@nckturner
Copy link
Contributor

@rothgar @kand617 Can you:

kubectl port-forward -n appmesh-demo \
    $(kubectl get pod -n appmesh-demo |grep colorgateway \
    |awk '{print $1}') 9901

and then navigate to localhost:9901/config_dump in your browser and copy the output here?

@rothgar
Copy link
Contributor

rothgar commented May 2, 2019 •
edited

Here you go.

{
 "configs": [
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.BootstrapConfigDump",
   "bootstrap": {
    "node": {
     "id": "mesh/color-mesh/virtualNode/colorgateway-appmesh-demo",
     "cluster": "mesh/color-mesh/virtualNode/colorgateway-appmesh-demo",
     "build_version": "ae8c8aa036e58e39b3d2fba81f5bdc4683a30682/1.9.0/Clean/DEBUG/BoringSSL"
    },
    "dynamic_resources": {
     "lds_config": {
      "ads": {}
     },
     "cds_config": {
      "ads": {}
     },
     "ads_config": {
      "api_type": "GRPC",
      "grpc_services": [
       {
        "google_grpc": {
         "target_uri": "appmesh-envoy-management.us-west-2.amazonaws.com:443",
         "channel_credentials": {
          "ssl_credentials": {
           "root_certs": {
            "filename": "/etc/pki/tls/cert.pem"
           }
          }
         },
         "call_credentials": [
          {
           "from_plugin": {
            "name": "envoy.grpc_credentials.aws_iam",
            "config": {
             "region": "us-west-2",
             "service_name": "appmesh"
            }
           }
          }
         ],
         "stat_prefix": "ads",
         "credentials_factory_name": "envoy.grpc_credentials.aws_iam"
        }
       }
      ]
     }
    },
    "tracing": {
     "http": {
      "name": "envoy.xray",
      "config": {
       "daemon_endpoint": "127.0.0.1:2000"
      }
     }
    },
    "admin": {
     "access_log_path": "/tmp/envoy_admin_access.log",
     "address": {
      "socket_address": {
       "address": "0.0.0.0",
       "port_value": 9901
      }
     }
    },
    "stats_config": {
     "stats_tags": [
      {
       "tag_name": "appmesh.mesh",
       "fixed_value": "color-mesh"
      },
      {
       "tag_name": "appmesh.virtual_node",
       "fixed_value": "colorgateway-appmesh-demo"
      }
     ]
    }
   },
   "last_updated": "2019-04-29T17:46:45.101Z"
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.ClustersConfigDump",
   "version_info": "261982128",
   "dynamic_active_clusters": [
    {
     "version_info": "261982128",
     "cluster": {
      "name": "cds_egress_color-mesh_amazonaws",
      "type": "ORIGINAL_DST",
      "connect_timeout": "1s",
      "lb_policy": "ORIGINAL_DST_LB"
     },
     "last_updated": "2019-04-30T17:57:42.358Z"
    },
    {
     "version_info": "261982128",
     "cluster": {
      "name": "cds_egress_color-mesh_colorteller-appmesh-demo_http_9080",
      "type": "LOGICAL_DNS",
      "connect_timeout": "1s",
      "load_assignment": {
       "cluster_name": "cds_egress_color-mesh_colorteller-appmesh-demo_http_9080",
       "endpoints": [
        {
         "lb_endpoints": [
          {
           "endpoint": {
            "address": {
             "socket_address": {
              "address": "colorteller.appmesh-demo.svc.cluster.local",
              "port_value": 9080
             }
            }
           }
          }
         ]
        }
       ]
      }
     },
     "last_updated": "2019-04-30T17:57:42.365Z"
    },
    {
     "version_info": "261982128",
     "cluster": {
      "name": "cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080",
      "type": "LOGICAL_DNS",
      "connect_timeout": "1s",
      "load_assignment": {
       "cluster_name": "cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080",
       "endpoints": [
        {
         "lb_endpoints": [
          {
           "endpoint": {
            "address": {
             "socket_address": {
              "address": "colorteller-black.appmesh-demo.svc.cluster.local",
              "port_value": 9080
             }
            }
           }
          }
         ]
        }
       ]
      }
     },
     "last_updated": "2019-04-30T17:57:42.362Z"
    },
    {
     "version_info": "261982128",
     "cluster": {
      "name": "cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080",
      "type": "LOGICAL_DNS",
      "connect_timeout": "1s",
      "load_assignment": {
       "cluster_name": "cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080",
       "endpoints": [
        {
         "lb_endpoints": [
          {
           "endpoint": {
            "address": {
             "socket_address": {
              "address": "colorteller-blue.appmesh-demo.svc.cluster.local",
              "port_value": 9080
             }
            }
           }
          }
         ]
        }
       ]
      }
     },
     "last_updated": "2019-04-30T17:57:42.369Z"
    },
    {
     "version_info": "261982128",
     "cluster": {
      "name": "cds_ingress_color-mesh_colorgateway-appmesh-demo_http_9080",
      "connect_timeout": "0.250s",
      "load_assignment": {
       "cluster_name": "cds_ingress_color-mesh_colorgateway-appmesh-demo_http_9080",
       "endpoints": [
        {
         "lb_endpoints": [
          {
           "endpoint": {
            "address": {
             "socket_address": {
              "address": "127.0.0.1",
              "port_value": 9080
             }
            }
           }
          }
         ]
        }
       ]
      }
     },
     "last_updated": "2019-04-30T17:57:42.355Z"
    }
   ]
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.ListenersConfigDump",
   "version_info": "261982128",
   "dynamic_active_listeners": [
    {
     "version_info": "261982128",
     "listener": {
      "name": "lds_ingress_0.0.0.0_15000",
      "address": {
       "socket_address": {
        "address": "0.0.0.0",
        "port_value": 15000
       }
      },
      "filter_chains": [
       {
        "filter_chain_match": {
         "destination_port": 9080
        },
        "filters": [
         {
          "name": "envoy.http_connection_manager",
          "config": {
           "http_filters": [
            {
             "name": "envoy.router"
            }
           ],
           "rds": {
            "config_source": {
             "ads": {}
            },
            "route_config_name": "rds_ingress_http_9080"
           },
           "stat_prefix": "ingress",
           "tracing": {}
          }
         }
        ]
       }
      ],
      "listener_filters": [
       {
        "name": "envoy.listener.original_dst"
       }
      ]
     },
     "last_updated": "2019-04-30T17:57:42.385Z"
    },
    {
     "version_info": "261982128",
     "listener": {
      "name": "lds_egress_0.0.0.0_15001",
      "address": {
       "socket_address": {
        "address": "0.0.0.0",
        "port_value": 15001
       }
      },
      "filter_chains": [
       {
        "filter_chain_match": {
         "destination_port": 9080
        },
        "filters": [
         {
          "name": "envoy.http_connection_manager",
          "config": {
           "http_filters": [
            {
             "name": "envoy.router"
            }
           ],
           "rds": {
            "config_source": {
             "ads": {}
            },
            "route_config_name": "rds_egress_http_9080"
           },
           "stat_prefix": "egress",
           "tracing": {
            "operation_name": "EGRESS"
           }
          }
         }
        ]
       },
       {
        "filter_chain_match": {
         "destination_port": 443,
         "server_names": [
          "*.amazonaws.com"
         ]
        },
        "filters": [
         {
          "name": "envoy.tcp_proxy",
          "config": {
           "stat_prefix": "egress",
           "cluster": "cds_egress_color-mesh_amazonaws"
          }
         }
        ]
       }
      ],
      "listener_filters": [
       {
        "name": "envoy.listener.original_dst"
       },
       {
        "name": "envoy.listener.tls_inspector"
       }
      ]
     },
     "last_updated": "2019-04-30T17:57:42.389Z"
    }
   ]
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.RoutesConfigDump",
   "dynamic_route_configs": [
    {
     "version_info": "261982128",
     "route_config": {
      "name": "rds_egress_http_9080",
      "virtual_hosts": [
       {
        "name": "colorteller.appmesh-demo",
        "domains": [
         "colorteller.appmesh-demo",
         "colorteller.appmesh-demo:9080"
        ],
        "routes": [
         {
          "match": {
           "prefix": "/"
          },
          "route": {
           "weighted_clusters": {
            "clusters": [
             {
              "name": "cds_egress_color-mesh_colorteller-appmesh-demo_http_9080",
              "weight": 1
             },
             {
              "name": "cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080",
              "weight": 1
             },
             {
              "name": "cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080",
              "weight": 1
             }
            ],
            "total_weight": 3
           }
          }
         }
        ]
       }
      ]
     },
     "last_updated": "2019-04-30T17:57:42.393Z"
    },
    {
     "version_info": "261982128",
     "route_config": {
      "name": "rds_ingress_http_9080",
      "virtual_hosts": [
       {
        "name": "local_service",
        "domains": [
         "*"
        ],
        "routes": [
         {
          "match": {
           "prefix": "/"
          },
          "route": {
           "cluster": "cds_ingress_color-mesh_colorgateway-appmesh-demo_http_9080"
          }
         }
        ]
       }
      ]
     },
     "last_updated": "2019-04-30T17:57:42.393Z"
    }
   ]
  }
 ]
}

@rothgar
Copy link
Contributor

rothgar commented May 8, 2019

After not touching this for 6 days it appears the problem resolved itself for me. I am now able to curl the color gateway from inside the cluster.

curl http://colorgateway:9080/color
{"color":"black", "stats": {"black":1}}

Next time I see an error I'll enable debug logging on the gateway sidecar and see if I see anything in the logs.

For anyone else that also needs to do this:
To enable debug log level you need to change the appmesh inject hook deployment via

kubectl edit deploy aws-app-mesh-inject -n appmesh-inject

Change the APPMESH_LOG_LEVEL environment variable on the container spec from info to debug
Then restart any pods you want to have debug logging on the envoy sidecar.

@kiranmeduri
Copy link
Contributor

Please reopen if you see any more issues. Thanks

@jicowan
Copy link

jicowan commented Aug 27, 2019

@kiranmeduri I am also getting this error.

{
 "configs": [
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.BootstrapConfigDump",
   "bootstrap": {
    "node": {
     "id": "mesh/color-mesh/virtualNode/colorgateway-appmesh-demo",
     "cluster": "mesh/color-mesh/virtualNode/colorgateway-appmesh-demo",
     "build_version": "ae8c8aa036e58e39b3d2fba81f5bdc4683a30682/1.9.0/Clean/DEBUG/BoringSSL"
    },
    "dynamic_resources": {
     "lds_config": {
      "ads": {}
     },
     "cds_config": {
      "ads": {}
     },
     "ads_config": {
      "api_type": "GRPC",
      "grpc_services": [
       {
        "google_grpc": {
         "target_uri": "appmesh-envoy-management.us-east-1.amazonaws.com:443",
         "channel_credentials": {
          "ssl_credentials": {
           "root_certs": {
            "filename": "/etc/pki/tls/cert.pem"
           }
          }
         },
         "call_credentials": [
          {
           "from_plugin": {
            "name": "envoy.grpc_credentials.aws_iam",
            "config": {
             "service_name": "appmesh",
             "region": "us-east-1"
            }
           }
          }
         ],
         "stat_prefix": "ads",
         "credentials_factory_name": "envoy.grpc_credentials.aws_iam"
        }
       }
      ]
     }
    },
    "tracing": {
     "http": {
      "name": "envoy.xray",
      "config": {
       "daemon_endpoint": "127.0.0.1:2000"
      }
     }
    },
    "admin": {
     "access_log_path": "/tmp/envoy_admin_access.log",
     "address": {
      "socket_address": {
       "address": "0.0.0.0",
       "port_value": 9901
      }
     }
    },
    "stats_config": {
     "stats_tags": [
      {
       "tag_name": "appmesh.mesh",
       "fixed_value": "color-mesh"
      },
      {
       "tag_name": "appmesh.virtual_node",
       "fixed_value": "colorgateway-appmesh-demo"
      }
     ]
    }
   },
   "last_updated": "2019-08-27T16:35:56.090Z"
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.ClustersConfigDump",
   "version_info": "380526692",
   "dynamic_active_clusters": [
    {
     "version_info": "380509227",
     "cluster": {
      "name": "cds_egress_color-mesh_amazonaws",
      "type": "ORIGINAL_DST",
      "connect_timeout": "1s",
      "lb_policy": "ORIGINAL_DST_LB"
     },
     "last_updated": "2019-08-27T16:35:56.339Z"
    },
    {
     "version_info": "380526692",
     "cluster": {
      "name": "cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080",
      "type": "LOGICAL_DNS",
      "connect_timeout": "1s",
      "load_assignment": {
       "cluster_name": "cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080",
       "endpoints": [
        {
         "lb_endpoints": [
          {
           "endpoint": {
            "address": {
             "socket_address": {
              "address": "colorgateway.appmesh-demo.svc.cluster.local",
              "port_value": 9080
             }
            }
           }
          }
         ]
        }
       ]
      }
     },
     "last_updated": "2019-08-27T16:55:17.377Z"
    },
    {
     "version_info": "380509227",
     "cluster": {
      "name": "cds_ingress_color-mesh_colorgateway-appmesh-demo_http_9080",
      "connect_timeout": "0.250s",
      "load_assignment": {
       "cluster_name": "cds_ingress_color-mesh_colorgateway-appmesh-demo_http_9080",
       "endpoints": [
        {
         "lb_endpoints": [
          {
           "endpoint": {
            "address": {
             "socket_address": {
              "address": "127.0.0.1",
              "port_value": 9080
             }
            }
           }
          }
         ]
        }
       ]
      }
     },
     "last_updated": "2019-08-27T16:35:56.335Z"
    }
   ]
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.ListenersConfigDump",
   "version_info": "380526692",
   "dynamic_active_listeners": [
    {
     "version_info": "380509227",
     "listener": {
      "name": "lds_ingress_0.0.0.0_15000",
      "address": {
       "socket_address": {
        "address": "0.0.0.0",
        "port_value": 15000
       }
      },
      "filter_chains": [
       {
        "filter_chain_match": {
         "destination_port": 9080
        },
        "filters": [
         {
          "name": "envoy.http_connection_manager",
          "config": {
           "http_filters": [
            {
             "name": "envoy.router"
            }
           ],
           "rds": {
            "route_config_name": "rds_ingress_http_9080",
            "config_source": {
             "ads": {}
            }
           },
           "stat_prefix": "ingress",
           "tracing": {}
          }
         }
        ]
       }
      ],
      "listener_filters": [
       {
        "name": "envoy.listener.original_dst"
       }
      ]
     },
     "last_updated": "2019-08-27T16:35:56.370Z"
    },
    {
     "version_info": "380509227",
     "listener": {
      "name": "lds_egress_0.0.0.0_15001",
      "address": {
       "socket_address": {
        "address": "0.0.0.0",
        "port_value": 15001
       }
      },
      "filter_chains": [
       {
        "filter_chain_match": {
         "destination_port": 9080
        },
        "filters": [
         {
          "name": "envoy.http_connection_manager",
          "config": {
           "http_filters": [
            {
             "name": "envoy.router"
            }
           ],
           "rds": {
            "route_config_name": "rds_egress_http_9080",
            "config_source": {
             "ads": {}
            }
           },
           "stat_prefix": "egress",
           "tracing": {
            "operation_name": "EGRESS"
           }
          }
         }
        ]
       },
       {
        "filter_chain_match": {
         "destination_port": 443,
         "server_names": [
          "*.amazonaws.com"
         ]
        },
        "filters": [
         {
          "name": "envoy.tcp_proxy",
          "config": {
           "stat_prefix": "egress",
           "cluster": "cds_egress_color-mesh_amazonaws"
          }
         }
        ]
       }
      ],
      "listener_filters": [
       {
        "name": "envoy.listener.original_dst"
       },
       {
        "name": "envoy.listener.tls_inspector"
       }
      ]
     },
     "last_updated": "2019-08-27T16:35:56.377Z"
    }
   ]
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.RoutesConfigDump",
   "dynamic_route_configs": [
    {
     "version_info": "380526692",
     "route_config": {
      "name": "rds_egress_http_9080",
      "virtual_hosts": [
       {
        "name": "colorteller.appmesh-demo",
        "domains": [
         "colorteller.appmesh-demo",
         "colorteller.appmesh-demo:9080"
        ],
        "routes": [
         {
          "match": {
           "prefix": "/color"
          },
          "route": {
           "weighted_clusters": {
            "clusters": [
             {
              "name": "cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080",
              "weight": 1
             }
            ],
            "total_weight": 1
           }
          }
         }
        ]
       }
      ]
     },
     "last_updated": "2019-08-27T16:55:17.380Z"
    },
    {
     "version_info": "380509227",
     "route_config": {
      "name": "rds_ingress_http_9080",
      "virtual_hosts": [
       {
        "name": "local_service",
        "domains": [
         "*"
        ],
        "routes": [
         {
          "match": {
           "prefix": "/"
          },
          "route": {
           "cluster": "cds_ingress_color-mesh_colorgateway-appmesh-demo_http_9080"
          }
         }
        ]
       }
      ]
     },
     "last_updated": "2019-08-27T16:55:17.380Z"
    }
   ]
  }
 ]
}

@jicowan
Copy link

jicowan commented Aug 27, 2019

The logs from the color-gateway envoy container:

Did not find Envoy configuration file at /tmp/envoy.yaml, creating one.
added envoy.xray tracing config to /tmp/envoy_tracing_config.yaml
Appending /tmp/envoy_tracing_config.yaml to /tmp/envoy.yaml
Appending /tmp/envoy_stats_config.yaml to /tmp/envoy.yaml
Starting Envoy.
[2019-08-27 16:35:56.074][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws_Init_Cleanup Initiate AWS SDK for C++ with Version:1.6.39
[2019-08-27 16:35:56.075][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] CurlHttpClient Initializing Curl library
[2019-08-27 16:35:56.075][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSConfigFileProfileConfigLoader Initializing config loader against fileName //.aws/config and using profilePrefix = 1
[2019-08-27 16:35:56.075][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSConfigFileProfileConfigLoader Unable to open config file //.aws/config for reading.
[2019-08-27 16:35:56.075][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSProfileConfigLoader Failed to reload configuration.
[2019-08-27 16:35:56.084][000001][info][main] [source/server/server.cc:206] initializing epoch 0 (hot restart version=10.200.16384.567.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=9863272)
[2019-08-27 16:35:56.084][000001][info][main] [source/server/server.cc:208] statically linked extensions:
[2019-08-27 16:35:56.085][000001][info][main] [source/server/server.cc:210]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2019-08-27 16:35:56.085][000001][info][main] [source/server/server.cc:213]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2019-08-27 16:35:56.085][000001][info][main] [source/server/server.cc:216]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2019-08-27 16:35:56.085][000001][info][main] [source/server/server.cc:219]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.dubbo_proxy,envoy.filters.network.rbac,envoy.filters.network.sni_cluster,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2019-08-27 16:35:56.085][000001][info][main] [source/server/server.cc:221]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2019-08-27 16:35:56.085][000001][info][main] [source/server/server.cc:223]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.tracers.datadog,envoy.xray,envoy.zipkin
[2019-08-27 16:35:56.085][000001][info][main] [source/server/server.cc:226]   transport_sockets.downstream: envoy.transport_sockets.alts,envoy.transport_sockets.capture,raw_buffer,tls
[2019-08-27 16:35:56.085][000001][info][main] [source/server/server.cc:229]   transport_sockets.upstream: envoy.transport_sockets.alts,envoy.transport_sockets.capture,raw_buffer,tls
[2019-08-27 16:35:56.097][000001][info][main] [source/server/server.cc:271] admin address: 0.0.0.0:9901
[2019-08-27 16:35:56.100][000001][info][config] [source/server/configuration_impl.cc:50] loading 0 static secret(s)
[2019-08-27 16:35:56.100][000001][info][config] [source/server/configuration_impl.cc:56] loading 0 cluster(s)
[2019-08-27 16:35:56.101][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSConfigFileProfileConfigLoader Initializing config loader against fileName //.aws/config and using profilePrefix = 1
[2019-08-27 16:35:56.101][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSConfigFileProfileConfigLoader Initializing config loader against fileName //.aws/credentials and using profilePrefix = 0
[2019-08-27 16:35:56.101][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] ProfileConfigFileAWSCredentialsProvider Setting provider to read credentials from //.aws/credentials for credentials file and //.aws/config for the config file , for use with profile default
[2019-08-27 16:35:56.101][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] EC2MetadataClient Creating AWSHttpResourceClient with max connections2 and scheme http
[2019-08-27 16:35:56.101][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] CurlHandleContainer Initializing CurlHandleContainer with size 2
[2019-08-27 16:35:56.101][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] InstanceProfileCredentialsProvider Creating Instance with default EC2MetadataClient and refresh rate 300000
[2019-08-27 16:35:56.101][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] DefaultAWSCredentialsProviderChain Added EC2 metadata service credentials provider to the provider chain.
[2019-08-27 16:35:56.101][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSConfigFileProfileConfigLoader Unable to open config file //.aws/credentials for reading.
[2019-08-27 16:35:56.101][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSProfileConfigLoader Failed to reload configuration.
[2019-08-27 16:35:56.101][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSConfigFileProfileConfigLoader Unable to open config file //.aws/config for reading.
[2019-08-27 16:35:56.101][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSProfileConfigLoader Failed to reload configuration.
[2019-08-27 16:35:56.101][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] InstanceProfileCredentialsProvider Credentials have expired attempting to repull from EC2 Metadata Service.
[2019-08-27 16:35:56.101][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] CurlHandleContainer Pool grown by 2
[2019-08-27 16:35:56.101][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] CurlHandleContainer Connection has been released. Continuing.
[2019-08-27 16:35:56.107][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] CurlHandleContainer Connection has been released. Continuing.
[2019-08-27 16:35:56.108][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::EC2InstanceProfileConfigLoader Successfully pulled credentials from metadata service with access key ASIA36C6WWEJWDONYVPS
[2019-08-27 16:35:56.108][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] CurlHandleContainer Connection has been released. Continuing.
[2019-08-27 16:35:56.111][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] EC2MetadataClient Detected current region as us-east-1
[2019-08-27 16:35:56.111][000001][info][misc] [source/common/aws/aws_sdk_config.cc:66] Aws::Config::AWSProfileConfigLoader Successfully reloaded configuration.
[2019-08-27 16:35:56.112][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:132] cm init: initializing cds
[2019-08-27 16:35:56.113][000001][info][config] [source/server/configuration_impl.cc:67] loading 0 listener(s)
[2019-08-27 16:35:56.113][000001][info][config] [source/server/configuration_impl.cc:92] loading tracing configuration
[2019-08-27 16:35:56.113][000001][info][config] [source/server/configuration_impl.cc:101]   loading tracing driver: envoy.xray
[2019-08-27 16:35:56.113][000001][info][tracing] [source/extensions/tracers/xray/xray_tracer_impl.cc:95] send X-Ray generated segments to daemon address on 127.0.0.1:2000
[2019-08-27 16:35:56.113][000001][info][tracing] [source/extensions/tracers/xray/sampling.cc:114] unable to parse empty json file. falling back to default rule set.
[2019-08-27 16:35:56.113][000001][info][config] [source/server/configuration_impl.cc:112] loading stats sink configuration
[2019-08-27 16:35:56.114][000001][info][main] [source/server/server.cc:463] starting main dispatch loop
[2019-08-27 16:35:56.335][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:495] add/update cluster cds_ingress_color-mesh_colorgateway-appmesh-demo_http_9080 during init
[2019-08-27 16:35:56.339][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:495] add/update cluster cds_egress_color-mesh_amazonaws during init
[2019-08-27 16:35:56.343][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:495] add/update cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 during init
[2019-08-27 16:35:56.348][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:136] cm init: all clusters initialized
[2019-08-27 16:35:56.349][000001][info][main] [source/server/server.cc:435] all clusters initialized. initializing init manager
[2019-08-27 16:35:56.370][000001][info][upstream] [source/server/lds_api.cc:80] lds: add/update listener 'lds_ingress_0.0.0.0_15000'
[2019-08-27 16:35:56.377][000001][info][upstream] [source/server/lds_api.cc:80] lds: add/update listener 'lds_egress_0.0.0.0_15001'
[2019-08-27 16:35:56.380][000001][info][config] [source/server/listener_manager_impl.cc:961] all dependencies initialized. starting workers
[2019-08-27 16:35:56.381][000026][info][tracing] [source/extensions/tracers/xray/sampling.cc:114] unable to parse empty json file. falling back to default rule set.
[2019-08-27 16:35:56.384][000025][info][tracing] [source/extensions/tracers/xray/sampling.cc:114] unable to parse empty json file. falling back to default rule set.
[2019-08-27 16:36:16.692][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 starting warming
[2019-08-27 16:36:16.696][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 starting warming
[2019-08-27 16:36:16.700][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 starting warming
[2019-08-27 16:36:16.700][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080
[2019-08-27 16:36:16.705][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 complete
[2019-08-27 16:36:16.706][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 complete
[2019-08-27 16:36:16.706][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 complete
[2019-08-27 16:36:22.843][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 starting warming
[2019-08-27 16:36:22.844][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080
[2019-08-27 16:36:22.844][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080
[2019-08-27 16:36:22.844][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080
[2019-08-27 16:36:22.848][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 complete
[2019-08-27 16:36:31.221][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c0f-9690f7f98045c542f2ae421f","id":"30aab7ac92f5b2ae","start_time":1566923791.219,"end_time":1566923791.221,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"bf04a61628ef37ab","start_time":1566923791.219,"end_time":1566923791.221,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:31.230][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c0f-42162373c8c3aba2e817b927","id":"60801b68312f640d","start_time":1566923791.225,"end_time":1566923791.23,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v1","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"0b2cfbb1c71f14be","start_time":1566923791.225,"end_time":1566923791.23,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v1","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:32.246][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c10-57908fc8d5bb3c7da1884003","id":"2335d50fee5c7b09","start_time":1566923792.245,"end_time":1566923792.246,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"911e288a0cfc754d","start_time":1566923792.245,"end_time":1566923792.246,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:32.247][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c10-ce43a36f52dedf1d053de6aa","id":"a42d2e44c8970be9","start_time":1566923792.245,"end_time":1566923792.247,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"1cf46b876d50555c","start_time":1566923792.245,"end_time":1566923792.247,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:33.933][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c11-f96ffae8823598deec13f32e","id":"7b1a21207269c8da","start_time":1566923793.932,"end_time":1566923793.933,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"dc765124056c2ac0","start_time":1566923793.932,"end_time":1566923793.933,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:33.937][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c11-d0b1c281991774e4ca7160b6","id":"cd8a13d9d4294be5","start_time":1566923793.937,"end_time":1566923793.937,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v1","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"28af1df04469b170","start_time":1566923793.937,"end_time":1566923793.937,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v1","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:34.074][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c12-9b5625b8fde4f2a9113df105","id":"8ade8076afe86412","start_time":1566923794.071,"end_time":1566923794.074,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"38216aeed9a65f15","start_time":1566923794.071,"end_time":1566923794.074,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:34.087][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c12-7f394949dfb38dd74485a47d","id":"75c9e5c2a9ea96de","start_time":1566923794.086,"end_time":1566923794.086,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/v2/swagger","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"bdfa873de1dc4030","start_time":1566923794.086,"end_time":1566923794.086,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/v2/swagger","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:36.647][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c14-b393f94784998cb191072d02","id":"389c69c14e7509c9","start_time":1566923796.646,"end_time":1566923796.647,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"3e6a12ed22814fb0","start_time":1566923796.646,"end_time":1566923796.647,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:36.650][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c14-d65a4607d00d98895342b46a","id":"f004800ddf662762","start_time":1566923796.65,"end_time":1566923796.65,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v1","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"0444236bb564d431","start_time":1566923796.65,"end_time":1566923796.65,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v1","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:41.011][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c19-3f1365eafc113a9834c3003d","id":"85f24a9209ecb425","start_time":1566923801.01,"end_time":1566923801.011,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"b20b46092ab50b63","start_time":1566923801.01,"end_time":1566923801.011,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:41.016][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c19-d178e04b06cc35bdbbd93088","id":"c487e98ff604535b","start_time":1566923801.016,"end_time":1566923801.016,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v1","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"77242094b456e046","start_time":1566923801.016,"end_time":1566923801.016,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v1","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:46.921][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c1e-2f8877dd09541c5ef01232f9","id":"098bc7fe083e9855","start_time":1566923806.921,"end_time":1566923806.921,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"551a7f19db1121e9","start_time":1566923806.921,"end_time":1566923806.921,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:46.925][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c1e-181ea0f505ab291cc40620e9","id":"ba40882ce8e47935","start_time":1566923806.924,"end_time":1566923806.925,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v1","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"7bfa4c56b851ea73","start_time":1566923806.924,"end_time":1566923806.925,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v1","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:47.379][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c1f-2d93ec03ed290fef53c5679c","id":"82d443aeab25c815","start_time":1566923807.378,"end_time":1566923807.379,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"61066e7a8cff2e97","start_time":1566923807.378,"end_time":1566923807.379,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:47.384][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c1f-1ac6518eb758890c34b21358","id":"6ece48aeaed5829c","start_time":1566923807.384,"end_time":1566923807.384,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v1","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"da0daf3222b659fd","start_time":1566923807.384,"end_time":1566923807.384,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v1","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:57.115][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c29-408a902e02a3206c08aee235","id":"21a9e0161ce07082","start_time":1566923817.114,"end_time":1566923817.115,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"d16e8f575786396f","start_time":1566923817.114,"end_time":1566923817.115,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:58.175][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c2a-6a22b00c59bf010ac7e4d52d","id":"a2220a597eb6b50d","start_time":1566923818.174,"end_time":1566923818.175,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"ced55b99f7bf4105","start_time":1566923818.174,"end_time":1566923818.175,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:36:58.182][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c2a-4b389c8d89527b99aaaca981","id":"034d266b612f3cfb","start_time":1566923818.181,"end_time":1566923818.182,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v2","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"e292b3ae8635d4d9","start_time":1566923818.181,"end_time":1566923818.182,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v2","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:37:11.444][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c37-9da982261c8771fac5d1c559","id":"2d979c1084109d7b","start_time":1566923831.443,"end_time":1566923831.444,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"67a801e65d2c1860","start_time":1566923831.443,"end_time":1566923831.444,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:37:11.451][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c37-11e6da533ba1244e22c28402","id":"d55b29228545b217","start_time":1566923831.45,"end_time":1566923831.45,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v2","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"27c53c2228a171e5","start_time":1566923831.45,"end_time":1566923831.451,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger/docs/v2","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:37:15.205][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655c3b-4f5fa5a703e8ceeab1122325","id":"f3bb02660042a546","start_time":1566923835.205,"end_time":1566923835.205,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"12ac2774026b141d","start_time":1566923835.205,"end_time":1566923835.205,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/swagger.json","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:38:18.555][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 starting warming
[2019-08-27 16:38:18.560][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 starting warming
[2019-08-27 16:38:18.564][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 starting warming
[2019-08-27 16:38:18.564][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080
[2019-08-27 16:38:18.570][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 complete
[2019-08-27 16:38:18.570][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 complete
[2019-08-27 16:38:18.571][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 complete
[2019-08-27 16:38:50.963][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 starting warming
[2019-08-27 16:38:50.963][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080
[2019-08-27 16:38:50.963][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080
[2019-08-27 16:38:50.963][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080
[2019-08-27 16:38:50.967][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 complete
[2019-08-27 16:39:25.561][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655cbd-773aaf902bfc5f45c6e23dfd","id":"c7054ee4b5764622","start_time":1566923965.557,"end_time":1566923965.561,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway:9080/color","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":500,"content_length":22}},"fault":true,"subsegments":[{"id":"93d59b29fa677fa8","start_time":1566923965.557,"end_time":1566923965.561,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway:9080/color","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":500,"content_length":22}}}]}
[2019-08-27 16:39:59.847][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655cdf-457474a7a6ce1ee7d5f3fc9e","id":"0d18d773c8464405","start_time":1566923999.846,"end_time":1566923999.847,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway:9080/color","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":500,"content_length":22}},"fault":true,"subsegments":[{"id":"5c51600a7c27d6ca","start_time":1566923999.846,"end_time":1566923999.847,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway:9080/color","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":500,"content_length":22}}}]}
[2019-08-27 16:40:05.475][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655ce5-8d263ffe7d3192dbc6de3599","id":"23fb523576030e4a","start_time":1566924005.474,"end_time":1566924005.475,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway:9080/","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":404,"content_length":19}},"error":true,"subsegments":[{"id":"3ef9a9a8dd31b855","start_time":1566924005.474,"end_time":1566924005.475,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway:9080/","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":404,"content_length":19}}}]}
[2019-08-27 16:41:12.556][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655bfc-756e155007b07e0a07ce401b","id":"3b11288019966597","start_time":1566923772.557,"end_time":1566924072.556,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo","method":"POST","user_agent":"grpc-go/1.13.0"},"response":{"status":200,"content_length":0}},"subsegments":[{"id":"b66fa3df617bd770","start_time":1566923772.557,"end_time":1566924072.556,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo","method":"POST","user_agent":"grpc-go/1.13.0"},"response":{"status":200,"content_length":0}}}]}
[2019-08-27 16:42:44.265][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 starting warming
[2019-08-27 16:42:44.268][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 starting warming
[2019-08-27 16:42:44.272][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 starting warming
[2019-08-27 16:42:44.272][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080
[2019-08-27 16:42:44.277][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 complete
[2019-08-27 16:42:44.278][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 complete
[2019-08-27 16:42:44.278][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 complete
[2019-08-27 16:42:51.243][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 starting warming
[2019-08-27 16:42:51.244][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080
[2019-08-27 16:42:51.244][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080
[2019-08-27 16:42:51.244][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080
[2019-08-27 16:42:51.252][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 complete
[2019-08-27 16:45:21.407][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 starting warming
[2019-08-27 16:45:21.411][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 starting warming
[2019-08-27 16:45:21.414][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 starting warming
[2019-08-27 16:45:21.415][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080
[2019-08-27 16:45:21.420][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 complete
[2019-08-27 16:45:21.420][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 complete
[2019-08-27 16:45:21.420][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 complete
[2019-08-27 16:45:44.949][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 starting warming
[2019-08-27 16:45:44.949][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080
[2019-08-27 16:45:44.950][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080
[2019-08-27 16:45:44.950][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080
[2019-08-27 16:45:44.961][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 complete
[2019-08-27 16:46:12.561][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655d28-ef3a285d80f6070e3fef9c7b","id":"22e259786d0ce813","start_time":1566924072.556,"end_time":1566924372.561,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo","method":"POST","user_agent":"grpc-go/1.13.0"},"response":{"status":200,"content_length":0}},"subsegments":[{"id":"d7378a2306a465b3","start_time":1566924072.556,"end_time":1566924372.561,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo","method":"POST","user_agent":"grpc-go/1.13.0"},"response":{"status":200,"content_length":0}}}]}
[2019-08-27 16:46:15.217][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 starting warming
[2019-08-27 16:46:15.221][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 starting warming
[2019-08-27 16:46:15.225][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 starting warming
[2019-08-27 16:46:15.225][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080
[2019-08-27 16:46:15.229][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 complete
[2019-08-27 16:46:15.230][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 complete
[2019-08-27 16:46:15.230][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 complete
[2019-08-27 16:46:20.324][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 starting warming
[2019-08-27 16:46:20.325][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080
[2019-08-27 16:46:20.325][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080
[2019-08-27 16:46:20.325][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080
[2019-08-27 16:46:20.336][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 complete
[2019-08-27 16:47:16.965][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 starting warming
[2019-08-27 16:47:16.968][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 starting warming
[2019-08-27 16:47:16.972][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 starting warming
[2019-08-27 16:47:16.973][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080
[2019-08-27 16:47:16.977][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 complete
[2019-08-27 16:47:16.977][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 complete
[2019-08-27 16:47:16.978][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 complete
[2019-08-27 16:47:48.463][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 starting warming
[2019-08-27 16:47:48.463][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080
[2019-08-27 16:47:48.464][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080
[2019-08-27 16:47:48.464][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080
[2019-08-27 16:47:48.468][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 complete
[2019-08-27 16:48:48.893][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 starting warming
[2019-08-27 16:48:48.897][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 starting warming
[2019-08-27 16:48:48.901][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 starting warming
[2019-08-27 16:48:48.901][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080
[2019-08-27 16:48:48.912][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 complete
[2019-08-27 16:48:48.913][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 complete
[2019-08-27 16:48:48.913][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 complete
[2019-08-27 16:49:16.953][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 starting warming
[2019-08-27 16:49:16.953][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080
[2019-08-27 16:49:16.953][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080
[2019-08-27 16:49:16.953][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080
[2019-08-27 16:49:16.958][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 complete
[2019-08-27 16:49:26.701][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655f16-155f79dbc0adff965f8314e7","id":"8f48dc82fb677629","start_time":1566924566.701,"end_time":1566924566.701,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorteller.appmesh-demo:9080/","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":0}},"error":true,"subsegments":[{"id":"5b031f10ddef6e9f","start_time":1566924566.701,"end_time":1566924566.701,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorteller.appmesh-demo:9080/","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":0}}}]}
[2019-08-27 16:49:26.701][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655f16-dfa361e11e658bed8a9e3d6c","id":"caa1b6c481370778","start_time":1566924566.699,"end_time":1566924566.701,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway:9080/color","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":500,"content_length":22}},"fault":true,"subsegments":[{"id":"e66c18d57d270102","start_time":1566924566.699,"end_time":1566924566.701,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway:9080/color","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":500,"content_length":22}}}]}
[2019-08-27 16:50:56.381][000001][info][main] [source/server/drain_manager_impl.cc:63] shutting down parent after drain
[2019-08-27 16:51:15.096][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655e57-87786acbd920320f26d5c87a","id":"f461b214f287c51c","start_time":1566924375.095,"end_time":1566924675.096,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo","method":"POST","user_agent":"grpc-go/1.13.0"},"response":{"status":200,"content_length":0}},"subsegments":[{"id":"d273e95e3ff64083","start_time":1566924375.095,"end_time":1566924675.096,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo","method":"POST","user_agent":"grpc-go/1.13.0"},"response":{"status":200,"content_length":0}}}]}
[2019-08-27 16:51:51.103][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 starting warming
[2019-08-27 16:51:51.107][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 starting warming
[2019-08-27 16:51:51.111][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 starting warming
[2019-08-27 16:51:51.111][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080
[2019-08-27 16:51:51.115][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 complete
[2019-08-27 16:51:51.115][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 complete
[2019-08-27 16:51:51.116][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 complete
[2019-08-27 16:52:19.077][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 starting warming
[2019-08-27 16:52:19.078][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080
[2019-08-27 16:52:19.078][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080
[2019-08-27 16:52:19.079][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080
[2019-08-27 16:52:19.083][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 complete
[2019-08-27 16:52:32.357][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655fd0-87f42b34ce51fcb34dd6fd10","id":"8040acb300bba569","start_time":1566924752.356,"end_time":1566924752.356,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorteller.appmesh-demo:9080/","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":0}},"error":true,"subsegments":[{"id":"43bcccab587d6c42","start_time":1566924752.356,"end_time":1566924752.356,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorteller.appmesh-demo:9080/","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":0}}}]}
[2019-08-27 16:52:32.357][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655fd0-16c0ad4e24c5c6f6966a0736","id":"4eb8c192bf5cf6fc","start_time":1566924752.354,"end_time":1566924752.357,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo:9080/color","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":500,"content_length":22}},"fault":true,"subsegments":[{"id":"98a7e7dd8789bb76","start_time":1566924752.354,"end_time":1566924752.357,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo:9080/color","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":500,"content_length":22}}}]}
[2019-08-27 16:53:53.581][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d656021-e4f02793a18722535759d4a7","id":"6bc2b28fe7518b0d","start_time":1566924833.58,"end_time":1566924833.581,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo:9080/color","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":500,"content_length":22}},"fault":true,"subsegments":[{"id":"381e6f3f2c235fce","start_time":1566924833.58,"end_time":1566924833.581,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo:9080/color","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":500,"content_length":22}}}]}
[2019-08-27 16:53:55.674][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d656023-7155c0d71b9faa5f5e07cd48","id":"3976d6afae0419ae","start_time":1566924835.674,"end_time":1566924835.674,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorteller.appmesh-demo:9080/","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":0}},"error":true,"subsegments":[{"id":"9c4b68c2d75ee1f0","start_time":1566924835.674,"end_time":1566924835.674,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorteller.appmesh-demo:9080/","method":"GET","user_agent":"Go-http-client/1.1"},"response":{"status":404,"content_length":0}}}]}
[2019-08-27 16:53:55.676][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d656023-06247d57018f3ae88638801c","id":"7f707cb31ffec53b","start_time":1566924835.674,"end_time":1566924835.676,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo:9080/color","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":500,"content_length":22}},"fault":true,"subsegments":[{"id":"b718d2a93aea3843","start_time":1566924835.674,"end_time":1566924835.676,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo:9080/color","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":500,"content_length":22}}}]}
[2019-08-27 16:54:46.993][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 starting warming
[2019-08-27 16:54:46.997][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 starting warming
[2019-08-27 16:54:47.000][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 starting warming
[2019-08-27 16:54:47.001][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080
[2019-08-27 16:54:47.005][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 complete
[2019-08-27 16:54:47.005][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 complete
[2019-08-27 16:54:47.006][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 complete
[2019-08-27 16:55:17.377][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 starting warming
[2019-08-27 16:55:17.377][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080
[2019-08-27 16:55:17.377][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080
[2019-08-27 16:55:17.378][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080
[2019-08-27 16:55:17.515][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 complete
[2019-08-27 16:56:15.101][000025][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d655f83-f591df891429b9da55021758","id":"64f1b44f6c8bb2f3","start_time":1566924675.096,"end_time":1566924975.101,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo","method":"POST","user_agent":"grpc-go/1.13.0"},"response":{"status":200,"content_length":0}},"subsegments":[{"id":"fcd63c72e1565330","start_time":1566924675.096,"end_time":1566924975.101,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo","method":"POST","user_agent":"grpc-go/1.13.0"},"response":{"status":200,"content_length":0}}}]}
[2019-08-27 16:56:20.323][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 starting warming
[2019-08-27 16:56:20.327][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 starting warming
[2019-08-27 16:56:20.330][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 starting warming
[2019-08-27 16:56:20.331][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080
[2019-08-27 16:56:20.335][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 complete
[2019-08-27 16:56:20.335][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 complete
[2019-08-27 16:56:20.336][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 complete
[2019-08-27 16:56:45.453][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 starting warming
[2019-08-27 16:56:45.453][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080
[2019-08-27 16:56:45.453][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080
[2019-08-27 16:56:45.453][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080
[2019-08-27 16:56:45.457][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 complete
[2019-08-27 16:56:51.633][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 starting warming
[2019-08-27 16:56:51.637][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 starting warming
[2019-08-27 16:56:51.641][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 starting warming
[2019-08-27 16:56:51.641][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080
[2019-08-27 16:56:51.646][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 complete
[2019-08-27 16:56:51.646][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 complete
[2019-08-27 16:56:51.647][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 complete
[2019-08-27 16:57:16.283][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 starting warming
[2019-08-27 16:57:16.284][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080
[2019-08-27 16:57:16.284][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080
[2019-08-27 16:57:16.284][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080
[2019-08-27 16:57:16.289][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 complete
[2019-08-27 16:57:22.133][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 starting warming
[2019-08-27 16:57:22.137][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 starting warming
[2019-08-27 16:57:22.153][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 starting warming
[2019-08-27 16:57:22.154][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080
[2019-08-27 16:57:22.159][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080 complete
[2019-08-27 16:57:22.159][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080 complete
[2019-08-27 16:57:22.160][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080 complete
[2019-08-27 16:57:47.066][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:501] add/update cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 starting warming
[2019-08-27 16:57:47.066][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-black-appmesh-demo_http_9080
[2019-08-27 16:57:47.066][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-blue-appmesh-demo_http_9080
[2019-08-27 16:57:47.066][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster cds_egress_color-mesh_colorteller-appmesh-demo_http_9080
[2019-08-27 16:57:47.070][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:513] warming cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080 complete
[2019-08-27 16:58:07.930][000026][info][tracing] [source/extensions/tracers/xray/xray_core_types.cc:295] xray generated segments document: {"format":"json","version":1}
{"trace_id":"1-5d65611f-c568eba00bb1f2d892fe0662","id":"a2d82a7c05f56a00","start_time":1566925087.927,"end_time":1566925087.93,"name":"color-mesh/colorgateway-appmesh-demo","origin":"AWS::AppMesh::Proxy","aws":{"app_mesh":{"mesh_name":"color-mesh","virtual_node_name":"colorgateway-appmesh-demo"}},"http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/color","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":500,"content_length":22}},"fault":true,"subsegments":[{"id":"e6a675bd7b3fc26e","start_time":1566925087.927,"end_time":1566925087.93,"namespace":"remote","name":"color-mesh/colorgateway-appmesh-demo","http":{"request":{"url":"http://colorgateway.appmesh-demo.svc.cluster.local:9080/color","method":"GET","user_agent":"curl/7.35.0"},"response":{"status":500,"content_length":22}}}]}

@Y0Username
Copy link
Contributor

Y0Username commented Aug 28, 2019 •
edited

@jicowan Envoy seems to removing app clusters: removing cluster cds_egress_color-mesh_colorgateway-appmesh-demo_http_9080.

Can you paste the logs for your colorgateway and colorteller? Are they healthy and server comes up?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@kiranmeduri @mhausenblas @rothgar @nckturner @kand617 @jicowan @Y0Username