Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Istio Cilium 1.5.1 pilot crashes (because golang was not updated) #10981

Closed
nberlee opened this issue Apr 15, 2020 · 1 comment
Closed

Istio Cilium 1.5.1 pilot crashes (because golang was not updated) #10981

nberlee opened this issue Apr 15, 2020 · 1 comment
Assignees

Comments

@nberlee
Copy link
Contributor

nberlee commented Apr 15, 2020

Bug report

istio/istio#21672 was fixed in 1.5.1 by updating to golang 1.14.1
However, istio-cilium pilot is still compiled using golang 1.14.0:

$ docker run --rm -it cilium/istio_pilot:1.5.1 version
version.BuildInfo{Version:"51591009a6d881b5175d2d413075afb0db92374a-dirty", GitRevision:"51591009a6d881b5175d2d413075afb0db92374a-dirty", GolangVersion:"go1.14", BuildStatus:"Modified", GitTag:"1.5.1-6-g51591009a"}

This results in an mlock error

$ kubectl logs -n istio-system -n istio-system istio-pilot-7644cd958d-ngjj7  discovery
2020-04-15T07:36:13.469108Z	info	FLAG: --appNamespace=""
2020-04-15T07:36:13.469155Z	info	FLAG: --clusterRegistriesNamespace=""
2020-04-15T07:36:13.469162Z	info	FLAG: --configDir=""
2020-04-15T07:36:13.469168Z	info	FLAG: --consulserverURL=""
2020-04-15T07:36:13.469175Z	info	FLAG: --ctrlz_address="localhost"
2020-04-15T07:36:13.469183Z	info	FLAG: --ctrlz_port="9876"
2020-04-15T07:36:13.469191Z	info	FLAG: --disable-install-crds="true"
2020-04-15T07:36:13.469196Z	info	FLAG: --domain="cluster.local"
2020-04-15T07:36:13.469202Z	info	FLAG: --grpcAddr=":15010"
2020-04-15T07:36:13.469224Z	info	FLAG: --help="false"
2020-04-15T07:36:13.469230Z	info	FLAG: --httpAddr=":8080"
2020-04-15T07:36:13.469235Z	info	FLAG: --httpsAddr=":15017"
2020-04-15T07:36:13.469244Z	info	FLAG: --keepaliveInterval="30s"
2020-04-15T07:36:13.469250Z	info	FLAG: --keepaliveMaxServerConnectionAge="30m0s"
2020-04-15T07:36:13.469256Z	info	FLAG: --keepaliveTimeout="10s"
2020-04-15T07:36:13.469261Z	info	FLAG: --kubeconfig=""
2020-04-15T07:36:13.469267Z	info	FLAG: --log_as_json="false"
2020-04-15T07:36:13.469272Z	info	FLAG: --log_caller=""
2020-04-15T07:36:13.469278Z	info	FLAG: --log_output_level="default:info"
2020-04-15T07:36:13.469283Z	info	FLAG: --log_rotate=""
2020-04-15T07:36:13.469291Z	info	FLAG: --log_rotate_max_age="30"
2020-04-15T07:36:13.469297Z	info	FLAG: --log_rotate_max_backups="1000"
2020-04-15T07:36:13.469303Z	info	FLAG: --log_rotate_max_size="104857600"
2020-04-15T07:36:13.469309Z	info	FLAG: --log_stacktrace_level="default:none"
2020-04-15T07:36:13.469320Z	info	FLAG: --log_target="[stdout]"
2020-04-15T07:36:13.469331Z	info	FLAG: --mcpInitialConnWindowSize="1048576"
2020-04-15T07:36:13.469337Z	info	FLAG: --mcpInitialWindowSize="1048576"
2020-04-15T07:36:13.469343Z	info	FLAG: --mcpMaxMsgSize="4194304"
2020-04-15T07:36:13.469353Z	info	FLAG: --meshConfig="/etc/istio/config/mesh"
2020-04-15T07:36:13.469359Z	info	FLAG: --monitoringAddr=":15014"
2020-04-15T07:36:13.469364Z	info	FLAG: --namespace=""
2020-04-15T07:36:13.469370Z	info	FLAG: --networksConfig="/etc/istio/config/meshNetworks"
2020-04-15T07:36:13.469387Z	info	FLAG: --plugins="[authn,authz,health,mixer,envoyfilter,cilium]"
2020-04-15T07:36:13.469394Z	info	FLAG: --profile="true"
2020-04-15T07:36:13.469401Z	info	FLAG: --registries="[Kubernetes]"
2020-04-15T07:36:13.469411Z	info	FLAG: --resync="1m0s"
2020-04-15T07:36:13.469417Z	info	FLAG: --secureGrpcAddr=""
2020-04-15T07:36:13.469422Z	info	FLAG: --trust-domain=""
2020-04-15T07:36:13.469482Z	info	Primary Cluster name: Kubernetes
runtime: mlock of signal stack failed: 12
runtime: increase the mlock limit (ulimit -l) or
runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+
fatal error: mlock failed
uname -a
Linux REDACTEDHOSTNAME 5.4.0-0.bpo.3-amd64 #1 SMP Debian 5.4.13-1~bpo10+1 (2020-02-07) x86_64 GNU/Linux

Workaround
istio/istio#21672 (comment)

@jrajahalme
Copy link
Member

This issue has been fixed in cilium/istio_pilot:1.5.1-1, which is now automatically used when installing with cilium-istioctl (see https://docs.cilium.io/en/latest/gettingstarted/istio/)

jrajahalme added a commit that referenced this issue Apr 15, 2020
Fixes: #10981
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
jrajahalme added a commit that referenced this issue Jul 1, 2020
Fixes: #10981
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
joestringer pushed a commit that referenced this issue Jul 1, 2020
Fixes: #10981
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants