Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

External service not accessible shortly, most of time when a pod is just created. #14070

Closed
xiaozhongliu opened this issue May 14, 2019 · 5 comments

Comments

@xiaozhongliu
Copy link

xiaozhongliu commented May 14, 2019

Bug description
This issue starts to occur frequently right after upgrading istio from 1.0.5 to 1.1.4.
External service (database in our situation) is not accessible shortly, most of time when a pod is just created. It'll go back to normal after app container auto restarts once.

Expected behavior
External service should be accessible whenever the app container starts running.

Version (include the output of istioctl version --remote and kubectl version)

~ istioctl version --remote
client version: version.BuildInfo{Version:"1.1.4", GitRevision:"bc7465793cbff4c4189639b3f404e21c517cbdc6", User:"root", Host:"471e568b-66d7-11e9-a0d5-0a580a2c0304", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.3-20-gbc74657"}
grafana version: version.BuildInfo{Version:"", GitRevision:"", User:"", Host:"", GolangVersion:"", DockerHub:"", BuildStatus:"", GitTag:""}
citadel version: version.BuildInfo{Version:"1.1.4", GitRevision:"bc7465793cbff4c4189639b3f404e21c517cbdc6-dirty", User:"root", Host:"471e568b-66d7-11e9-a0d5-0a580a2c0304", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.3-20-gbc74657"}
galley version: version.BuildInfo{Version:"1.1.4", GitRevision:"bc7465793cbff4c4189639b3f404e21c517cbdc6-dirty", User:"root", Host:"471e568b-66d7-11e9-a0d5-0a580a2c0304", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.3-20-gbc74657"}
ingressgateway version: version.BuildInfo{Version:"1.1.4", GitRevision:"bc7465793cbff4c4189639b3f404e21c517cbdc6", User:"root", Host:"471e568b-66d7-11e9-a0d5-0a580a2c0304", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.3-20-gbc74657"}
pilot version: version.BuildInfo{Version:"1.1.4", GitRevision:"bc7465793cbff4c4189639b3f404e21c517cbdc6-dirty", User:"root", Host:"471e568b-66d7-11e9-a0d5-0a580a2c0304", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.3-20-gbc74657"}
policy version: version.BuildInfo{Version:"1.1.4", GitRevision:"bc7465793cbff4c4189639b3f404e21c517cbdc6-dirty", User:"root", Host:"471e568b-66d7-11e9-a0d5-0a580a2c0304", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.3-20-gbc74657"}
sidecar-injector version: version.BuildInfo{Version:"1.1.4", GitRevision:"bc7465793cbff4c4189639b3f404e21c517cbdc6-dirty", User:"root", Host:"471e568b-66d7-11e9-a0d5-0a580a2c0304", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.3-20-gbc74657"}
telemetry version: version.BuildInfo{Version:"1.1.4", GitRevision:"bc7465793cbff4c4189639b3f404e21c517cbdc6-dirty", User:"root", Host:"471e568b-66d7-11e9-a0d5-0a580a2c0304", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.3-20-gbc74657"}
kiali version: version.BuildInfo{Version:"", GitRevision:"", User:"", Host:"", GolangVersion:"", DockerHub:"", BuildStatus:"", GitTag:""}
prometheus version: version.BuildInfo{Version:"", GitRevision:"", User:"", Host:"", GolangVersion:"", DockerHub:"", BuildStatus:"", GitTag:""}
➜  ~ istioctl version
version.BuildInfo{Version:"1.1.4", GitRevision:"bc7465793cbff4c4189639b3f404e21c517cbdc6", User:"root", Host:"471e568b-66d7-11e9-a0d5-0a580a2c0304", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.3-20-gbc74657"}

How was Istio installed?
Helm

Environment where bug was observed (cloud vendor, OS, etc)
Alibaba ACK

Affected product area (please put an X in all that apply)
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience

@esnible
Copy link
Contributor

esnible commented May 14, 2019

Is this a duplicate of #11130 ?

@xiaozhongliu
Copy link
Author

@esnible,thanks I think this may be quite the same situation, I'll take the solution suggested.
BTW, is inbound and outbound traffic not allowed before envoy container is ready? Any official doc or article explaining this? Thanks again.

@esnible
Copy link
Contributor

esnible commented May 15, 2019

@xiaolanz An init container routes all networking to the Envoy sidecar. The sidecar is configured with just enough configuration to ask Pilot about the mesh. Until the sidecar receives configuration from Pilot it won't know if outbound traffic is allowed universally or just for a few sites.

@frankbu Is there documentation that explains this?

@xiaozhongliu
Copy link
Author

@esnible thanks this explanation makes sense to me.

@howardjohn
Copy link
Member

We are tracking this under #11130. To keep things simpler, lets track this there

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants