Was prebuilt envoy with v1.17 able to run on CPU of x86_64 arch?

[xxx0624]

Title: Tried to run the pre-built envoy(v1.17) on Ubuntu(18.04 LTS Bionic Beaver) server which uses CPU of x86_64 arch but failed with errors about tcmalloc lib

Description:

Hi I tried to run the pre-built envoy on Ubuntu(18.04 LTS Bionic Beaver) server which uses CPU of x86_64 arch but failed. Specifically I use get-envoy to download and install envoy binary with v1.17 on my server and then run the command envoy --version to see if envoy works for me. However it failed and said:

root@manager1:~/.getenvoy/builds/standard/1.17.0/linux_glibc/bin# ./envoy --version
external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:550] MmapAligned() failed (size, alignment) 1073741824 1073741824 @ 0x1219551cc102 0x1219551be89a 0x1219551be253 0x1219551a2fe5 0x
1219551bac16 0x121955238286 0x121952cefdbd 0x121952cef454 0x121952cf7101 0x121955235eed
external/com_github_google_tcmalloc/tcmalloc/arena.cc:34] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size) 131072 48 @ 0x1219551cc50b 0x1219551a3076 0
x1219551bac16 0x121955238286 0x121952cefdbd 0x121952cef454 0x121952cf7101 0x121955235eed

FYI:
root@manager1: uname -a
Linux manager1 4.19.127-nn5-server #nn5 SMP PREEMPT Mon Oct 5 03:22:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@manager1: cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
And one more interesting thing is that envoy with v1.15 works fine for me (at least envoy --version works)

Based on the error msg, I'm wondering if this prebuilt envoy works for my server? If not, does it mean I should build a new one manually?

Thanks in advance!

[rojkov]

What's the output of free? It might be that v1.17 wants to allocate a bit more memory than v1.15 upon start.

[xxx0624]

Hi @rojkov here it is:

free -m
              total        used        free      shared  buff/cache   available
Mem:          24095       17283        1157           9        5655        6460

BTW is there any minimum memory requirement for v1.17 or v.1.16? And if the server doesn't have so much resource for that, is there any work around to make it working? Any related docs will be appreciated as well! Thanks.

[rojkov]

Hm.. memory situation seems to be ok.

Hi one more quick question, does it support to build an envoy without tcmalloc? (switching to use the malloc instead) Cuz seems like the tcmalloc doesn't works on my server.

The older releases were built with gperftools' allocator. You can build against it by adding --define tcmalloc=gperftools to your bazel build command.

But I wonder what your CPU model is. Google's tcmalloc uses assembly for restartable sequences.

[xxx0624]

The older releases were built with gperftools' allocator. You can build against it by adding --define tcmalloc=gperftools to your bazel build command.

Thanks will give it a try.

But I wonder what your CPU model is. Google's tcmalloc uses assembly for restartable sequences.

model name : Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz

[rojkov]

I thought you're running inside QEMU or something.

I've found Intel(R) Xeon(R) Gold 6138F CPU @ 2.00GHz in our lab. It's exactly the same model as yours plus an Omni-Path chip integrated in the same package. At least the docker release build works nicely

$ docker run --rm envoyproxy/envoy:v1.17.0  --version

envoy  version: 5c801b25cae04f06bf48248c90e87d623d7a6283/1.17.0/Clean/RELEASE/BoringSSL

Everything should work, it's not a hw compatibility issue.

[xxx0624]

OK thanks @rojkov. I've built an envoy with tcmalloc=gperfools manually and it's working.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[htuch]

@rojkov we've seen a few more recent reports here on Arm64.

[brennerm]

Seeing the same error when running on arm64 with 1.17 and 1.18 Docker images.

$ lscpu
Architecture:         aarch64
Byte Order:           Little Endian
CPU(s):               6
On-line CPU(s) list:  0-3
Off-line CPU(s) list: 4,5
Thread(s) per core:   1
Core(s) per socket:   2
Socket(s):            2
Vendor ID:            Nvidia
Model:                0
Model name:           ARMv8 Processor rev 0 (v8l)
Stepping:             0x0
CPU max MHz:          1907,2000
CPU min MHz:          115,2000
BogoMIPS:             62.50
L1d cache:            64K
L1i cache:            128K
L2 cache:             2048K
L3 cache:             4096K
Flags:                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp
$ free -m
              total        used        free      shared  buff/cache   available
Mem:           7773         552        6477          29         744        7015
Swap:          3886           0        3886
$ docker run envoyproxy/envoy:v1.18-latest
external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:550] MmapAligned() failed (size, alignment) 1073741824 1073741824 @ 0x55841c7070 0x55841b9614 0x55841b9054 0x55841a2a94 0x55841b6284
external/com_github_google_tcmalloc/tcmalloc/arena.cc:34] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size) 131072 48 @ 0x55841c7380 0x55841a2b10 0x55841b6284

1.16 images work fine.

[PKizzle]

Same issue on a RaspberryPi 4 with 8GB RAM. Starting with the 17.x.x docker images (tcmalloc enabled) they do no longer start.

Architecture:        aarch64
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
Vendor ID:           ARM
Model:               3
Model name:          Cortex-A72
Stepping:            r0p3
CPU max MHz:         1800.0000
CPU min MHz:         600.0000
BogoMIPS:            108.00
Flags:               fp asimd evtstrm crc32 cpuid

              total        used        free      shared  buff/cache   available
Mem:           7812        2466        2979          92        2366        5617
Swap:          3515          86        3429

external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:550] MmapAligned() failed (size, alignment) 1073741824 1073741824 @ 0x55867b9470 0x55867aba14 0x55867ab454 0x5586794e94 0x55867a8684
external/com_github_google_tcmalloc/tcmalloc/arena.cc:34] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size) 131072 48 @ 0x55867b9780 0x5586794f10 0x55867a8684

The issue google/tcmalloc#33 might be relevant as well. However it was closed already, so I guess that tcmalloc should already be compatible with arm64.

[rojkov]

The problem seems to stem from the fact that tcmalloc tries to allocate 1073741824 bytes (1G) of contiguous memory. It does so by getting hints from the kernel with calling mmap(NULL, kPageSize,...) in a loop where kPageSize is 256k. Apparently the kernel running on ARM fails to give a correct hint for such a big chunk of memory even if there is one.

You might want to try to build Envoy with defined TCMALLOC_SMALL_BUT_SLOW and see if it helps.

[rojkov]

You might want to try to build Envoy with defined TCMALLOC_SMALL_BUT_SLOW and see if it helps.

If somebody can give me access to an arm64 machine I'll try it myself. Or please wait until I get a working rpi4 setup.

[PKizzle]

@rojkov You could also try to cross compile it using the docker images and upload the resulting envoy executable for testing. I tried to cross compile it with my Mac but so far I have not been successful (compilation starts but fails). For the cross compilation to start I adjusted the ci/run_envoy_docker.sh script by adding --platform linux/arm64 and -e TCMALLOC_SMALL_BUT_SLOW to the docker run command as well as export TCMALLOC_SMALL_BUT_SLOW=true somewhere above that. Further I had to use the old osxfuse volume mount implementation for docker desktop and run docker run --rm --privileged multiarch/qemu-user-static --reset -p yes in advance. Maybe you want to give it a try as well.

Note: Error above: Using environment variables does NOT define TCMALLOC_SMALL_BUT_SLOW

[PKizzle]

I was able to compile it on my RaspberryPi 4, however I could not define TCMALLOC_SMALL_BUT_SLOW for tcmalloc.
How do I need to change the bazel configuration in order to make it work?

[rojkov]

I guess you can add --copt -DTCMALLOC_SMALL_BUT_SLOW to your bazel build command.

Or alternatively edit bazel/repositories.bzl to use @com_github_google_tcmalloc//tcmalloc:tcmalloc_small_but_slow instead of @com_github_google_tcmalloc//tcmalloc as tcmalloc's target.

[PKizzle]

Thank you very much. I am now rebuilding it. Sadly I wasn't able to reuse the cache since it was somehow corrupted. Will get back with the results in a few hours 😅

[PKizzle]

I still receive the same error with TCMALLOC_SMALL_BUT_SLOW defined:

$ linux/arm64/build_release/envoy --version
external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:550] MmapAligned() failed (size, alignment) 33554432 33554432 @ 0x55691da078 0x55691cc7c8 0x55691cc208 0x55691b6644 0x55691c9764
external/com_github_google_tcmalloc/tcmalloc/arena.cc:34] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size) 131072 48 @ 0x55691da388 0x55691b66c0 0x55691c9764
Aborted

$ linux/arm64/build_release_stripped/envoy --version
external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:550] MmapAligned() failed (size, alignment) 33554432 33554432 @ 0x555ec7a078 0x555ec6c7c8 0x555ec6c208 0x555ec56644 0x555ec69764
external/com_github_google_tcmalloc/tcmalloc/arena.cc:34] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size) 131072 48 @ 0x555ec7a388 0x555ec566c0 0x555ec69764
Aborted

[rojkov]

Hm.. I thought mmap'ing 32M shouldn't be a problem for RPi4 even in case of very fragmented memory. I'll try to reproduce it next week.
Meanwhile could you please post here your setup for hugepages? tcmalloc makes some assumptions about THP.

[PKizzle]

Seems like the kernel for Raspberry Pi OS is not compiled with THP support:

$ sudo modprobe configs
$ zcat /proc/config.gz | grep HUGETLBFS
CONFIG_SYS_SUPPORTS_HUGETLBFS=y
# CONFIG_HUGETLBFS is not set
$ zcat /proc/config.gz | grep TRANSPARENT
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
# CONFIG_TRANSPARENT_HUGEPAGE is not set

This is how to enable THP on a RaspberryPi 4B (recompile kernel): http://www.rkoucha.fr/raspberry_pi/rpi4b_hp_config.html

[rojkov]

Right. Finally I've got a RPi4 running an Aarch64 distro and it lacks THP. No wonder Envoy can't start there.

I'll do some experiments and update the docs.

[PKizzle]

Would it be possible to additionally upload envoy images that do not require THP support (still use gperftools' allocator)?
It is quite a big task to recompile the kernel...

[rojkov]

Turned out in case of Aarch64 the issue is not about THP (it's good to have though). The problem is that tcmalloc assumes incorrect page table levels for RPi4. Indeed https://kernel.org/doc/Documentation/arm64/memory.txt says that in Aarch64 virtual addresses for processes may be either 39-bits or 48-bits depending on the number of page table levels (3 and 4 respectively).

So, tcmalloc assumes virtual addresses are 48-bits for Aarch64 (the same size as in x86-64 unless you're on Intel Ice Lake CPUs) and calculates hints for mmap() accordingly. mmap() fails.

With this patch applied to tcmalloc

diff --git a/tcmalloc/internal/config.h b/tcmalloc/internal/config.h
index 93f979f..4f0a1c1 100644
--- a/tcmalloc/internal/config.h
+++ b/tcmalloc/internal/config.h
@@ -46,7 +46,7 @@ inline constexpr int kAddressBits =
 // According to Documentation/arm64/memory.txt of kernel 3.16,
 // AARCH64 kernel supports 48-bit virtual addresses for both user and kernel.
 inline constexpr int kAddressBits =
-    (sizeof(void*) < 8 ? (8 * sizeof(void*)) : 48);
+    (sizeof(void*) < 8 ? (8 * sizeof(void*)) : 39);
 #else
 inline constexpr int kAddressBits = 8 * sizeof(void*);
 #endif

Envoy starts to work on RPi4.

As far as I know the CPU used in RPi4 supports page tables of 4 levels (PGD->PUD->PMD->PTE), so I guess the prebuilt Envoy can work too if the kernel is compiled with CONFIG_PGTABLE_LEVELS=4, But I haven't checked that.

To check quickly the size of virtual addresses in your system you can run cat /proc/self/maps. For CONFIG_PGTABLE_LEVELS=3 on RPi4 the result is

5563950000-5563958000 r-xp 00000000 08:02 845                            /bin/cat
5563967000-5563968000 r--p 00007000 08:02 845                            /bin/cat
5563968000-5563969000 rw-p 00008000 08:02 845                            /bin/cat
5598624000-5598645000 rw-p 00000000 00:00 0                              [heap]
7fa282c000-7fa284e000 rw-p 00000000 00:00 0
7fa284e000-7fa2b33000 r--p 00000000 08:02 156607                         /usr/lib/locale/locale-archive
7fa2b33000-7fa2c8c000 r-xp 00000000 08:02 2021                           /lib/aarch64-linux-gnu/libc-2.28.so
7fa2c8c000-7fa2c9b000 ---p 00159000 08:02 2021                           /lib/aarch64-linux-gnu/libc-2.28.so
7fa2c9b000-7fa2c9f000 r--p 00158000 08:02 2021                           /lib/aarch64-linux-gnu/libc-2.28.so
7fa2c9f000-7fa2ca1000 rw-p 0015c000 08:02 2021                           /lib/aarch64-linux-gnu/libc-2.28.so
7fa2ca1000-7fa2ca5000 rw-p 00000000 00:00 0
7fa2cb9000-7fa2cd8000 r-xp 00000000 08:02 1950                           /lib/aarch64-linux-gnu/ld-2.28.so
7fa2ce3000-7fa2ce5000 rw-p 00000000 00:00 0
7fa2ce5000-7fa2ce7000 r--p 00000000 00:00 0                              [vvar]
7fa2ce7000-7fa2ce8000 r-xp 00000000 00:00 0                              [vdso]
7fa2ce8000-7fa2ce9000 r--p 0001f000 08:02 1950                           /lib/aarch64-linux-gnu/ld-2.28.so
7fa2ce9000-7fa2ceb000 rw-p 00020000 08:02 1950                           /lib/aarch64-linux-gnu/ld-2.28.so
7fc73fa000-7fc741b000 rw-p 00000000 00:00 0                              [stack]

For CONFIG_PGTABLE_LEVELS=4 on x86-64:

5636ec39c000-5636ec39e000 r--p 00000000 fd:00 1987264                    /usr/bin/cat
5636ec39e000-5636ec3a2000 r-xp 00002000 fd:00 1987264                    /usr/bin/cat
5636ec3a2000-5636ec3a4000 r--p 00006000 fd:00 1987264                    /usr/bin/cat
5636ec3a4000-5636ec3a5000 r--p 00007000 fd:00 1987264                    /usr/bin/cat
5636ec3a5000-5636ec3a6000 rw-p 00008000 fd:00 1987264                    /usr/bin/cat
5636ed4fd000-5636ed51e000 rw-p 00000000 00:00 0                          [heap]
7f2191d24000-7f2191d46000 rw-p 00000000 00:00 0
7f2191d46000-7f2191d9b000 r--p 00000000 fd:00 2235721                    /usr/lib/locale/C.utf8/LC_CTYPE
7f2191d9b000-7f2192446000 r--p 00000000 fd:00 2235720                    /usr/lib/locale/C.utf8/LC_COLLATE
7f2192446000-7f219f976000 r--p 00000000 fd:00 1977352                    /usr/lib/locale/locale-archive
7f219f976000-7f219f978000 rw-p 00000000 00:00 0
7f219f978000-7f219f99e000 r--p 00000000 fd:00 1976779                    /usr/lib64/libc-2.32.so
7f219f99e000-7f219faed000 r-xp 00026000 fd:00 1976779                    /usr/lib64/libc-2.32.so
7f219faed000-7f219fb38000 r--p 00175000 fd:00 1976779                    /usr/lib64/libc-2.32.so
7f219fb38000-7f219fb39000 ---p 001c0000 fd:00 1976779                    /usr/lib64/libc-2.32.so
7f219fb39000-7f219fb3c000 r--p 001c0000 fd:00 1976779                    /usr/lib64/libc-2.32.so
7f219fb3c000-7f219fb3f000 rw-p 001c3000 fd:00 1976779                    /usr/lib64/libc-2.32.so
7f219fb3f000-7f219fb45000 rw-p 00000000 00:00 0
7f219fb4b000-7f219fb4c000 r--p 00000000 fd:00 2235740                    /usr/lib/locale/C.utf8/LC_NUMERIC
7f219fb4c000-7f219fb4d000 r--p 00000000 fd:00 2235756                    /usr/lib/locale/C.utf8/LC_TIME
7f219fb4d000-7f219fb4e000 r--p 00000000 fd:00 2235731                    /usr/lib/locale/C.utf8/LC_MONETARY
7f219fb4e000-7f219fb4f000 r--p 00000000 fd:00 2235727                    /usr/lib/locale/C.utf8/LC_MESSAGES/SYS_LC_MESSAGES
7f219fb4f000-7f219fb50000 r--p 00000000 fd:00 2235741                    /usr/lib/locale/C.utf8/LC_PAPER
7f219fb50000-7f219fb51000 r--p 00000000 fd:00 2235738                    /usr/lib/locale/C.utf8/LC_NAME
7f219fb51000-7f219fb52000 r--p 00000000 fd:00 2228906                    /usr/lib/locale/C.utf8/LC_ADDRESS
7f219fb52000-7f219fb53000 r--p 00000000 fd:00 2235755                    /usr/lib/locale/C.utf8/LC_TELEPHONE
7f219fb53000-7f219fb54000 r--p 00000000 fd:00 2235726                    /usr/lib/locale/C.utf8/LC_MEASUREMENT
7f219fb54000-7f219fb5b000 r--s 00000000 fd:00 1976832                    /usr/lib64/gconv/gconv-modules.cache
7f219fb5b000-7f219fb5c000 r--p 00000000 fd:00 2235722                    /usr/lib/locale/C.utf8/LC_IDENTIFICATION
7f219fb5c000-7f219fb5d000 r--p 00000000 fd:00 1977681                    /usr/lib64/ld-2.32.so
7f219fb5d000-7f219fb7e000 r-xp 00001000 fd:00 1977681                    /usr/lib64/ld-2.32.so
7f219fb7e000-7f219fb87000 r--p 00022000 fd:00 1977681                    /usr/lib64/ld-2.32.so
7f219fb87000-7f219fb88000 r--p 0002a000 fd:00 1977681                    /usr/lib64/ld-2.32.so
7f219fb88000-7f219fb8a000 rw-p 0002b000 fd:00 1977681                    /usr/lib64/ld-2.32.so
7ffc78573000-7ffc78595000 rw-p 00000000 00:00 0                          [stack]
7ffc785ab000-7ffc785af000 r--p 00000000 00:00 0                          [vvar]
7ffc785af000-7ffc785b1000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]