Jemalloc on Gramine

[ila]

Hi, we finally managed to obtain good OLAP performance on SGX2 with Gramine. I am happy to announce that we get maximum 2x overhead with medium-sized, memory-intensive workloads (TPC-H): this is thanks to careful configuration of Gramine and our application, NUMA pinning and hardware-optimized instructions.

However, there is one last thing we cannot explain fully: the performance of jemalloc (https://jemalloc.net/) in Gramine is terrible (16x overhead), we had to remove it. We are working under the condition that data size < enclave size < EPC size, to minimize page swapping. So, we can (probably?) exclude spilling.

What could the cause be? As far as I can tell, jemalloc is specifically designed to avoid fragmentation (one of the enemies of SGX too). Yet, we see the opposite behaviour. Is this because Gramine does not implement its own jemalloc? Could it be that malloc is more optimized in Gramine? Or do you have any other idea? Logs aren't much useful here. Any input would be appreciated!

[mkow]

Could you try running it under a profiler and compare the traces with jemalloc and without? See https://gramine.readthedocs.io/en/latest/performance.html#profiling-with-perf.

[ila]

Yes, we are still looking into it - I noticed a 10% performance drop when using jemalloc rather than malloc. I think our application could be optimized for jemalloc with ARM processors, and running it on Intel CPUs causes some weird behaviour (however, I am not the one who developed this, so I want to be extra sure before I bring this up).

Or possibly, the slight performance difference is a tradeoff for less memory consumption in memory-intensive workloads.

However, I still cannot explain this huge difference in SGX -- 10% overhead is tolerable, but 2s vs 20s in SGX is not. When the memory is allocated (ie, mmap is called), this should be inside the enclave, right? Could this overhead be the cost of zeroing/encrypting the pages?

Currently trying to analyze perf record output, will keep you posted.

[dimakuv]

@ila Is Gramine-SGX running with EDMM enabled in your case? (sgx.edmm_enable = true)

EDMM is expensive due to how the in-Linux implementation currently works. So seeing 10x overhead is not that surprising when using EDMM.

And even without EDMM, each mmap() requires to zero out the memory region, which is also a rather slow operation.

Finally, we know of an inefficiently in memory allocations in Gramine. I submitted a patch to fix this: #1795. You can try this patch on your application, and see what difference it makes.

TLDR: In any case, memory allocations in SGX are expensive.

[ila]

No, we are not using EDMM. I understand that mmap zeroes memory, but as far as I know, this happens for security reasons -- for instance if process A uses memory, then frees it, then process B needs memory, the pages are zeroed to avoid process B accidentally reading whatever process A was doing.

Inside an enclave, allocating memory also inside an enclave, is this concern still valid? If we can trust our application and the processes accessing it, is still there a need to zero the pages?

Thanks to the pointer to this patch! I will test it now.

[dimakuv]

No, we are not using EDMM. I understand that mmap zeroes memory, but as far as I know, this happens for security reasons -- for instance if process A uses memory, then frees it, then process B needs memory, the pages are zeroed to avoid process B accidentally reading whatever process A was doing.

Not only for security reasons, but also to provide some sane default values in the freshly-allocated memory region. If the same process A first allocates [0, 4096) and writes all ones to it, then frees it, then allocates this region again, then it must have all zeros (and not the previously written all-ones).

Inside an enclave, allocating memory also inside an enclave, is this concern still valid? If we can trust our application and the processes accessing it, is still there a need to zero the pages?

No, we can't skip zeroing out. See my explanation above -- this is a requirement that applications rely on.

However, I must note that normal OS kernels like Linux and Windows play special tricks to not do literal zero-out of pages. Instead, Linux puts a special all-zeros page into the page tables upon mmap() requests. Which is a much faster operation than memset(0). I'm sure Windows has something similar.

SGX cannot do such a trick because SGX doesn't have access to page tables. So we must fall back to a more straight-forward and less efficient memset(0).

[ila]

That's very informative, thanks a lot for your explanation! Unfortunately your patch doesn't really work (using jemalloc):

gramine-sgx --version
Gramine 1.6.1post~UNRELEASED (c208c3cd58ab34d36fd7c2d9e2b73e704153d5a7)

gramine-sgx benchmark_runner benchmark/tpch/sf30_cu/q18.benchmark
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
name    run     timing
benchmark/tpch/sf30_cu/q18.benchmark    1       7.308045
benchmark/tpch/sf30_cu/q18.benchmark    2       TIMEOUT

Without jemalloc, performance seems to also slower, compared to the very first perf output I showed you:

(Edit: just tested with more benchmarks. It is indeed slower, around 50-60%)

perf stat -e branches,branch-misses,bus-cycles,cache-misses,cache-references,cycles,instructions,cpu-clock,cpu-migrations,page-faults,L1-dcache-load-misses,LLC-load-misses,LLC-store-misses,dTLB-loads,dTLB-load-misses,dTLB-stores,dTLB-store-misses gramine-sgx benchmark_runner "benchmark/tpch/sf30_cu/q18.benchmark"
-----------------------------------------------------------------------------------------------------------------------

name    run     timing
benchmark/tpch/sf30_cu/q18.benchmark    1       3.240562
benchmark/tpch/sf30_cu/q18.benchmark    2       3.295672
benchmark/tpch/sf30_cu/q18.benchmark    3       4.101344
benchmark/tpch/sf30_cu/q18.benchmark    4       3.080541
benchmark/tpch/sf30_cu/q18.benchmark    5       2.772859

 Performance counter stats for 'gramine-sgx benchmark_runner benchmark/tpch/sf30_cu/q18.benchmark':

       47699917097      branches                         #   58.602 M/sec                       (35.85%)
         135659034      branch-misses                    #    0.28% of all branches             (42.98%)
      272904831992      bus-cycles                       #  335.278 M/sec                       (50.11%)
         506351404      cache-misses                     #   71.32% of all cache refs           (57.23%)
         709986440      cache-references                 #  872.256 K/sec                       (57.21%)
      415787434382      cycles                           #    0.511 GHz                         (57.20%)
      222799482224      instructions                     #    0.54  insn per cycle              (64.35%)
         813965.65 msec cpu-clock                        #    4.772 CPUs utilized             
               800      cpu-migrations                   #    0.983 /sec                      
          16799594      page-faults                      #   20.639 K/sec                     
        2835159342      L1-dcache-load-misses                                                   (64.35%)
         152516370      LLC-load-misses                                                         (64.35%)
          60101682      LLC-store-misses                 #   73.838 K/sec                       (64.33%)
       52770855375      dTLB-loads                       #   64.832 M/sec                       (64.33%)
        2089194709      dTLB-load-misses                 #    3.96% of all dTLB cache accesses  (28.60%)
       24254876796      dTLB-stores                      #   29.798 M/sec                       (28.66%)
         296377771      dTLB-store-misses                #  364.116 K/sec                       (28.67%)

     170.566229709 seconds time elapsed

     703.499911000 seconds user
     110.423155000 seconds sys