Cilium fails to launch on ARM64: Binary cilium-envoy cannot be executed

[joestringer]

A user on Slack reports:

I'm trying to run cilium on a mixed amd64 and arm64 cluster. However the arm64 server fails to run cilium. The error seems to be:
level=fatal msg="Envoy: Binary \"cilium-envoy\" cannot be executed" error="signal: aborted (core dumped)" subsys=envoy-manager

Upon further investigation by attempting to run the binary directly, they saw:

root@10:/home/cilium# cilium-envoy --version
external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:550] MmapAligned() failed (size, alignment) 1073741824 1073741824 @ 0x558548b104 0x558547dea0 0x558547e0fc 0x55854612d0 0x558438bb18 0x558545ea00 0x55843d14b0 0x55843f3840 0x558438cc48 0x5585508da8 0x7f9321d038
external/com_github_google_tcmalloc/tcmalloc/arena.cc:34] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size) 131072 48 @ 0x558548b19c 0x5585461324 0x558438bb18 0x558545ea00 0x55843d14b0 0x55843f3840 0x558438cc48 0x5585508da8 0x7f9321d038
Aborted

Upstream tcmalloc has a related issue, google/tcmalloc#33 .

The user reports that adjusting tcmalloc configuration, the problem goes away:

Indeed, this is caused by tcmalloc's mis-assumption that aarch64's virtual address is always 48 bits. According to this post, it can be 39 bits depending on the kernel configuration. Unfortunately, my kernel happens to use 39-bit virtual addresses.
Two solutions:

• Change tcmalloc's code to assume 39-bit virtual addresses.
• Recompile the kernel with CONFIG_ARM64_VA_BITS_39=y

envoyproxy/envoy#15235 (comment)

[jovalle]

@joestringer Is this something that Cilium is adjusting for? Just tried 1.10.5 on an arm64 host and got the same error.

[joestringer]

@jovalle I'm not aware of any Cilium settings for this, but I was not deeply involved in the ARM64 efforts.My interpretation of the answer provided by the user in Slack (copied above) was that they had to recompile their kernel to support these images. You might be able to find out more info by joining the Cilium slack and asking around, perhaps in the #arm64 channel.