Pod becomes unreachable after istio installation.

[kbalbi]

Is this a BUG or FEATURE REQUEST?:
I do not know is this a bug or mis-configuration. I doubt mis-configuration because I did not do any customization just installed the istio with the standard yaml mentioned in the site (istio.io).

Did you review existing epics or issues to identify if this already being worked on? (please try to add the correct labels and epics):

This looks similar to the issue of 147 but with a different erro message.

Bug:
Y

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

istioctl version :  0.3.0
kubectl version : 1.8.5

Is Istio Auth enabled or not ? No
Did you install istio.yaml, istio-auth.yaml.... yep, installed only istio-all.yaml

What happened: After installation, the service become inaccessible. i was trying to troubleshoot and find the following logs on the envoy (istio-proxy).

========================

ERROR: logging before flag.Parse: I0105 21:00:57.213727 1 main.go:68] Version root@24c944bda24b-0.3.0-24ec6a3ac3a1d592d1873d2d8198278a849b8301
ERROR: logging before flag.Parse: I0105 21:00:57.216302 1 main.go:109] Proxy role: proxy.Node{Type:"sidecar", IPAddress:"10.233.70.14", ID:"my-service-64d44fdb49-j45vk.com-my-own-dev", Domain:"com-my-own-dev.svc.cluster.local"}
ERROR: logging before flag.Parse: I0105 21:00:57.217070 1 resolve.go:35] Attempting to lookup address: istio-mixer.istio-system
ERROR: logging before flag.Parse: I0105 21:00:57.230458 1 resolve.go:45] Addr resolved to: 10.233.7.3:9125
ERROR: logging before flag.Parse: I0105 21:00:57.230546 1 resolve.go:46] Finished lookup of address: istio-mixer.istio-system
ERROR: logging before flag.Parse: I0105 21:00:57.232351 1 main.go:169] Effective config: binaryPath: /usr/local/bin/envoy
configPath: /etc/istio/proxy
connectTimeout: 10.000s
discoveryAddress: istio-pilot.istio-system:15003
discoveryRefreshDelay: 1.000s
drainDuration: 45.000s
parentShutdownDuration: 60.000s
proxyAdminPort: 15000
serviceCluster: my-service
statsdUdpAddress: 10.233.7.3:9125
zipkinAddress: zipkin.istio-system:9411
ERROR: logging before flag.Parse: I0105 21:00:57.232403 1 main.go:186] Monitored certs: []envoy.CertSource{envoy.CertSource{Directory:"/etc/certs/", Files:[]string{"cert-chain.pem", "key.pem", "root-cert.pem"}}}
ERROR: logging before flag.Parse: I0105 21:00:57.234737 1 agent.go:164] Starting proxy agent
ERROR: logging before flag.Parse: I0105 21:00:57.235171 1 agent.go:186] Received new config, resetting budget
ERROR: logging before flag.Parse: I0105 21:00:57.235232 1 agent.go:261] Reconciling configuration (budget 10)
ERROR: logging before flag.Parse: I0105 21:00:57.235672 1 agent.go:281] Epoch 0 starting
ERROR: logging before flag.Parse: I0105 21:00:57.235787 1 config.go:49] writing configuration to /etc/istio/proxy/envoy-rev0.json
{
"listeners": [],
"lds": {
"cluster": "lds",
"refresh_delay_ms": 1000
},
"admin": {
"access_log_path": "/dev/stdout",
"address": "tcp://127.0.0.1:15000"
},
"cluster_manager": {
"clusters": [
{
"name": "rds",
"connect_timeout_ms": 10000,
"type": "strict_dns",
"lb_type": "round_robin",
"hosts": [
{
"url": "tcp://istio-pilot.istio-system:15003"
}
]
},
{
"name": "lds",
"connect_timeout_ms": 10000,
"type": "strict_dns",
"lb_type": "round_robin",
"hosts": [
{
"url": "tcp://istio-pilot.istio-system:15003"
}
]
},
{
"name": "zipkin",
"connect_timeout_ms": 10000,
"type": "strict_dns",
"lb_type": "round_robin",
"hosts": [
{
"url": "tcp://zipkin.istio-system:9411"
}
]
}
],
"sds": {
"cluster": {
"name": "sds",
"connect_timeout_ms": 10000,
"type": "strict_dns",
"lb_type": "round_robin",
"hosts": [
{
"url": "tcp://istio-pilot.istio-system:15003"
}
]
},
"refresh_delay_ms": 1000
},
"cds": {
"cluster": {
"name": "cds",
"connect_timeout_ms": 10000,
"type": "strict_dns",
"lb_type": "round_robin",
"hosts": [
{
"url": "tcp://istio-pilot.istio-system:15003"
}
]
},
"refresh_delay_ms": 1000
}
},
"statsd_udp_ip_address": "10.233.7.3:9125",
"tracing": {
"http": {
"driver": {
"type": "zipkin",
"config": {
"collector_cluster": "zipkin",
"collector_endpoint": "/api/v1/spans"
}
}
}
}
}ERROR: logging before flag.Parse: I0105 21:00:57.239063 1 watcher.go:270] Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster my-service --service-node sidecar10.233.70.14my-service-64d44fdb49-j45vk.com-my-own-devcom-my-own-dev.svc.cluster.local --max-obj-name-len 189]
[2018-01-05 21:00:57.366][17][warning][main] external/envoy/source/server/server.cc:157] initializing epoch 0 (hot restart version=9.200.16384.256)
[2018-01-05 21:00:57.724][17][warning][main] external/envoy/source/server/server.cc:344] starting main dispatch loop
[2018-01-05 21:01:00.070][17][warning][main] external/envoy/source/server/server.cc:328] all clusters initialized. initializing init manager
[2018-01-05 21:01:00.599][17][warning][config] external/envoy/source/server/listener_manager_impl.cc:448] all dependencies initialized. starting workers
[2018-01-05 21:02:00.590][17][warning][main] external/envoy/source/server/drain_manager_impl.cc:62] shutting down parent after drain
ERROR: logging before flag.Parse: I0105 21:07:03.280126 1 watcher.go:123] watchFileEvents: "/etc/certs//..1981_05_01_13_07_03.283210190": CREATE
ERROR: logging before flag.Parse: I0105 21:07:03.280513 1 watcher.go:123] watchFileEvents: "/etc/certs//..1981_05_01_13_07_03.283210190": MODIFY|ATTRIB
ERROR: logging before flag.Parse: I0105 21:07:03.280563 1 watcher.go:123] watchFileEvents: "/etc/certs//..data_tmp": RENAME
ERROR: logging before flag.Parse: I0105 21:07:03.280720 1 watcher.go:123] watchFileEvents: "/etc/certs//..data": CREATE
ERROR: logging before flag.Parse: I0105 21:07:03.280779 1 watcher.go:123] watchFileEvents: "/etc/certs//..1981_05_01_13_00_50.960532478": DELETE
ERROR: logging before flag.Parse: I0105 21:07:13.280731 1 watcher.go:136] watchFileEvents: notifying
ERROR: logging before flag.Parse: I0105 21:07:13.281745 1 agent.go:186] Received new config, resetting budget
ERROR: logging before flag.Parse: I0105 21:07:13.281786 1 agent.go:261] Reconciling configuration (budget 10)
ERROR: logging before flag.Parse: I0105 21:07:13.282030 1 agent.go:281] Epoch 1 starting
ERROR: logging before flag.Parse: I0105 21:07:13.282094 1 config.go:49] writing configuration to /etc/istio/proxy/envoy-rev1.json
{
"listeners": [],
"lds": {
"cluster": "lds",
"refresh_delay_ms": 1000
},
"admin": {
"access_log_path": "/dev/stdout",
"address": "tcp://127.0.0.1:15000"
},
"cluster_manager": {
"clusters": [
{
"name": "rds",
"connect_timeout_ms": 10000,
"type": "strict_dns",
"lb_type": "round_robin",
"hosts": [
{
"url": "tcp://istio-pilot.istio-system:15003"
}
]
},
{
"name": "lds",
"connect_timeout_ms": 10000,
"type": "strict_dns",
"lb_type": "round_robin",
"hosts": [
{
"url": "tcp://istio-pilot.istio-system:15003"
}
]
},
{
"name": "zipkin",
"connect_timeout_ms": 10000,
"type": "strict_dns",
"lb_type": "round_robin",
"hosts": [
{
"url": "tcp://zipkin.istio-system:9411"
}
]
}
],
"sds": {
"cluster": {
"name": "sds",
"connect_timeout_ms": 10000,
"type": "strict_dns",
"lb_type": "round_robin",
"hosts": [
{
"url": "tcp://istio-pilot.istio-system:15003"
}
]
},
"refresh_delay_ms": 1000
},
"cds": {
"cluster": {
"name": "cds",
"connect_timeout_ms": 10000,
"type": "strict_dns",
"lb_type": "round_robin",
"hosts": [
{
"url": "tcp://istio-pilot.istio-system:15003"
}
]
},
"refresh_delay_ms": 1000
}
},
"statsd_udp_ip_address": "10.233.7.3:9125",
"tracing": {
"http": {
"driver": {
"type": "zipkin",
"config": {
"collector_cluster": "zipkin",
"collector_endpoint": "/api/v1/spans"
}
}
}
}
}ERROR: logging before flag.Parse: I0105 21:07:13.282872 1 watcher.go:270] Envoy command: [-c /etc/istio/proxy/envoy-rev1.json --restart-epoch 1 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster my-service --service-node sidecar10.233.70.14my-service-64d44fdb49-j45vk.com-my-own-devcom-my-own-dev.svc.cluster.local --max-obj-name-len 189]
[2018-01-05 21:07:13.318][27][warning][main] external/envoy/source/server/server.cc:157] initializing epoch 1 (hot restart version=9.200.16384.256)
[2018-01-05 21:07:13.356][17][warning][main] external/envoy/source/server/server.cc:381] shutting down admin due to child startup
[2018-01-05 21:07:13.356][17][warning][main] external/envoy/source/server/server.cc:386] terminating parent process
[2018-01-05 21:07:13.965][27][warning][main] external/envoy/source/server/server.cc:344] starting main dispatch loop
[2018-01-05 21:07:16.163][27][warning][main] external/envoy/source/server/server.cc:328] all clusters initialized. initializing init manager
[2018-01-05 21:07:16.609][27][warning][config] external/envoy/source/server/listener_manager_impl.cc:448] all dependencies initialized. starting workers
[2018-01-05 21:07:16.610][17][warning][main] external/envoy/source/server/server.cc:91] closing and draining listeners
[2018-01-05 21:08:16.598][27][warning][main] external/envoy/source/server/drain_manager_impl.cc:62] shutting down parent after drain
[2018-01-05 21:08:16.598][17][warning][main] external/envoy/source/server/hot_restart_impl.cc:420] shutting down due to child request
[2018-01-05 21:08:16.598][17][warning][main] external/envoy/source/server/server.cc:303] caught SIGTERM
[2018-01-05 21:08:16.598][17][warning][main] external/envoy/source/server/server.cc:348] main dispatch loop exited
[2018-01-05 21:08:16.627][17][warning][main] external/envoy/source/server/server.cc:369] exiting
ERROR: logging before flag.Parse: I0105 21:08:16.642212 1 agent.go:211] Epoch 0 exited normally
[2018-01-05 21:31:16.777][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

[2018-01-05 21:31:18.673][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

[2018-01-05 21:31:20.640][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

[2018-01-05 21:31:21.944][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

[2018-01-05 21:31:23.766][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

[2018-01-05 21:31:25.382][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

[2018-01-05 21:31:27.140][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

[2018-01-05 21:31:28.590][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

===================================

What you expected to happen: Service is expected to be reachable after creating service mesh with istio.

How to reproduce it: Nothing done, it is a clean installation of istio on the existing system.

[sandylss]

I also have the same problem.Do anyone know what the problem is?

[louiscryan]

This is pretty stale. Can you retest with 0.8 ?