Perf test: bind server to one NUMA node

[akuzm]

another try with disabled percpu arenas #12631 (comment)

Changelog category (leave one):

• Not for changelog (changelog entry is not required)

[akuzm]

https://clickhouse-test-reports.s3.yandex.net/15144/ab19bb25fd8c286713580649ad1c183d493ea5dc/performance_comparison/report.html#fail1

disabling percpu arenas didn't help, it still segfaults

[akuzm]

https://clickhouse-test-reports.s3.yandex.net/15144/ab19bb25fd8c286713580649ad1c183d493ea5dc/performance_comparison/report.html#fail1

disabling percpu arenas didn't help, it still segfaults

Oh wait, it did help -- only the old server segfaults, not the new one. Now to find some way to disable it in runtime...

[azat]

Suggested change

	ln -s "percpu_arena:disabled" /etc/malloc.conf
	ln -s "confirm_conf:true,percpu_arena:disabled" /etc/malloc.conf

I would also suggest adding confirm_conf:true, it will print all options with values:

$ MALLOC_CONF=confirm_conf:true,percpu_arena:disabled ./clickhouse server -V
<jemalloc>: malloc_conf #1 (string specified via --with-malloc-conf): "percpu_arena:percpu,oversize_threshold:0,muzzy_decay_ms:10000"
<jemalloc>: -- Set conf value: percpu_arena:percpu
<jemalloc>: -- Set conf value: oversize_threshold:0
<jemalloc>: -- Set conf value: muzzy_decay_ms:10000
<jemalloc>: malloc_conf #2 (string pointed to by the global variable malloc_conf): ""
<jemalloc>: malloc_conf #3 ("name" of the file referenced by the symbolic link named /etc/malloc.conf): "confirm_conf:true"
<jemalloc>: -- Set conf value: confirm_conf:true
<jemalloc>: malloc_conf #4 (value of the environment variable MALLOC_CONF): "confirm_conf:true,percpu_arena:disabled"
<jemalloc>: -- Set conf value: confirm_conf:true
<jemalloc>: -- Set conf value: percpu_arena:disabled
ClickHouse server version 20.10.1.4618.

[akuzm]

Finally it works, and we have much less unstable queries than in master (only 12 compared to ~100):
https://clickhouse-test-reports.s3.yandex.net/15144/21f9cc65c99e8210d7b14a4bef5969710451bc5e/performance_comparison/report.html#fail1

After jemalloc bug is fixed, we can enable this in master. Or maybe we can just disable percpu arenas for perf test, I'll check how it influences the performance.

[amosbird]

Maybe we can also try binding the two instances to different CPU nodes. It might have unfair CPU frequency but less cache contentions. It might also be good to leave out CPU 0 due to system interruption handling

[akuzm]

Maybe we can also try binding the two instances to different CPU nodes. It might have unfair CPU frequency but less cache contentions. It might also be good to leave out CPU 0 due to system interruption handling

I'm afraid of introducing some systemic difference by binding to different nodes, but I'll try it as well. We only have two nodes there (this is from the test log):

2020-09-24 04:03:21	 + numactl --hardware
2020-09-24 04:03:21	 available: 2 nodes (0-1)
2020-09-24 04:03:21	 node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59
2020-09-24 04:03:21	 node 0 size: 257592 MB
2020-09-24 04:03:21	 node 0 free: 126497 MB
2020-09-24 04:03:21	 node 1 cpus: 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
2020-09-24 04:03:21	 node 1 size: 258027 MB
2020-09-24 04:03:21	 node 1 free: 83471 MB
2020-09-24 04:03:21	 node distances:
2020-09-24 04:03:21	 node   0   1 
2020-09-24 04:03:21	   0:  10  21 
2020-09-24 04:03:21	   1:  21  10 
2020-09-24 04:03:21	 + lscpu
2020-09-24 04:03:21	 Architecture:        x86_64
2020-09-24 04:03:21	 CPU op-mode(s):      32-bit, 64-bit
2020-09-24 04:03:21	 Byte Order:          Little Endian
2020-09-24 04:03:21	 CPU(s):              80
2020-09-24 04:03:21	 On-line CPU(s) list: 0-79
2020-09-24 04:03:21	 Thread(s) per core:  2
2020-09-24 04:03:21	 Core(s) per socket:  20
2020-09-24 04:03:21	 Socket(s):           2
2020-09-24 04:03:21	 NUMA node(s):        2
2020-09-24 04:03:21	 Vendor ID:           GenuineIntel
2020-09-24 04:03:21	 CPU family:          6
2020-09-24 04:03:21	 Model:               85
2020-09-24 04:03:21	 Model name:          Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz
2020-09-24 04:03:21	 Stepping:            7
2020-09-24 04:03:21	 CPU MHz:             2799.957
2020-09-24 04:03:21	 CPU max MHz:         3900.0000
2020-09-24 04:03:21	 CPU min MHz:         800.0000
2020-09-24 04:03:21	 BogoMIPS:            4200.00
2020-09-24 04:03:21	 Virtualization:      VT-x
2020-09-24 04:03:21	 L1d cache:           32K
2020-09-24 04:03:21	 L1i cache:           32K
2020-09-24 04:03:21	 L2 cache:            1024K
2020-09-24 04:03:21	 L3 cache:            28160K
2020-09-24 04:03:21	 NUMA node0 CPU(s):   0-19,40-59
2020-09-24 04:03:21	 NUMA node1 CPU(s):   20-39,60-79
2020-09-24 04:03:21	 Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke flush_l1d arch_capabilities

[akuzm]

https://clickhouse-test-reports.s3.yandex.net/15144/26abe8cb30819eea1c1f3383cc99620ab2e9da9a/performance_comparison/report.html#fail1

disabling percpu arenas doesn't really influence the performance

[azat]

https://clickhouse-test-reports.s3.yandex.net/15144/26abe8cb30819eea1c1f3383cc99620ab2e9da9a/performance_comparison/report.html#fail1
disabling percpu arenas doesn't really influence the performance

In general it should not, the purpose of using percpu arena (#11084) is to avoid live leaks with big thread pool, since in this case there will be too much arenas and if some thread will not be scheduled for some time it will not free data (there is also background threads way but percpu arena was prefered #11086)

[azat]

But it should not already, since #15035 is merged (it updates jemalloc submodule to include backported version of the patch from the referenced pull request in upstream jemalloc)

[akuzm]

Great, I'll remove it.

[azat]

@akuzm just out of curiosity, what does it improves in perf tests? (taking into account that fact that you disable percpu arena, so you may have a little bit higher memory reserved, and sometimes a little bit less overhead on freeing, but this just a possibility)

[akuzm]

@akuzm just out of curiosity, what does it improves in perf tests? (taking into account that fact that you disable percpu arena, so you may have a little bit higher memory reserved, and sometimes a little bit less overhead on freeing, but this just a possibility)

It gives more stable results for test queries that reference a lot of memory. W/o NUMA binding, we had about 150 unstable queries, and now only 35 are left.