Reloading segments leads to high CPU usage

[MeihanLi]

We recently observed that reloading segments, even reloading a small set of segments one by one for a partial upsert table (~90 segment replicas and each replica is around 650MB) can cause a significant increase in CPU usage (~30-40%).

Server tenant:

We have looked into one server and even we only reloaded 11 segments during the time, the CPU usage is pretty high.
Server (# of cores):

We are using default max.parallel.refresh.threads = 1 in HelixInstanceDataManagerConfig. It means there is no parallel during segment reloading. In other words, the server load mainly comes from refreshing a single segment.

Is there a way to improve the performance? This is blocking our schema evolution for such upsert tables and any suggestions or ideas are appreciated. Thanks!

[Jackie-Jiang]

It is expected that reloading segment (especially for upsert) can take one CPU core for some time because it is CPU bound operation. When parallel refresh threads is set to 1, we allow one reload at the same time, so there is no difference reloading multiple segments or one by one, and it will use up to 1 CPU core. I don't know how could CPU usage go up 30-40%. In the second graph, less than 5% CPU usage is expected

[MeihanLi]

Thanks @Jackie-Jiang , We found also segment reloading thread can take ~1 core but it will trigger lots of G1 GC which can take ~4.5 cores. That explained why in production, we saw increased CPU usage and Young Gen Time a lot. Is there a way to improve the performance of reloading one segment? cc: @deemoliu

[Testing] Before reloading:

top - 23:29:17 up 145 days, 15:49,  0 users,  load average: 0.76, 1.01, 1.08
Threads: 372 total,   0 running, 372 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.9 us,  0.6 sy,  0.5 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 385616.8 total, 155857.7 free, 145778.1 used,  83981.0 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used. 233824.1 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   446 udocker   20   0  353.9g 178.4g  43.2g S   6.2  47.4   0:53.20 rta_comms_log_m
   480 udocker   20   0  353.9g 178.4g  43.2g S   6.2  47.4   0:52.93 rta_comms_log_m
     1 udocker   20   0  353.9g 178.4g  43.2g S   0.0  47.4   0:00.04 java
    46 udocker   20   0  353.9g 178.4g  43.2g S   0.0  47.4   0:08.96 java
    47 udocker   20   0  353.9g 178.4g  43.2g S   0.0  47.4   0:39.76 GC Thread#0
    48 udocker   20   0  353.9g 178.4g  43.2g S   0.0  47.4   0:00.00 G1 Main Marker
    49 udocker   20   0  353.9g 178.4g  43.2g S   0.0  47.4   0:00.27 G1 Conc#0
    50 udocker   20   0  353.9g 178.4g  43.2g S   0.0  47.4   1:26.17 G1 Refine#0

[Testing] During reloading:

top - 23:32:35 up 145 days, 15:53,  0 users,  load average: 3.19, 1.45, 1.21
Threads: 374 total,   3 running, 371 sleeping,   0 stopped,   0 zombie
%Cpu(s): 10.4 us,  0.8 sy,  0.5 ni, 88.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 385616.8 total, 155590.3 free, 145843.2 used,  84183.2 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used. 233759.2 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  4476 udocker   20   0  354.0g 177.7g  42.1g R  99.9  47.2   0:21.01 segment-refresh
    50 udocker   20   0  354.0g 177.7g  42.1g R  93.8  47.2   1:35.56 G1 Refine#0
   250 udocker   20   0  354.0g 177.7g  42.1g S  81.2  47.2   0:37.80 G1 Refine#1
   251 udocker   20   0  354.0g 177.7g  42.1g S  56.2  47.2   0:35.32 G1 Refine#2
   252 udocker   20   0  354.0g 177.7g  42.1g S  37.5  47.2   0:33.19 G1 Refine#3
   253 udocker   20   0  354.0g 177.7g  42.1g S  31.2  47.2   0:32.30 G1 Refine#4
   254 udocker   20   0  354.0g 177.7g  42.1g S  25.0  47.2   0:31.00 G1 Refine#5
   255 udocker   20   0  354.0g 177.7g  42.1g S  12.5  47.2   0:30.51 G1 Refine#6
   256 udocker   20   0  354.0g 177.7g  42.1g S   6.2  47.2   0:29.47 G1 Refine#7
   257 udocker   20   0  354.0g 177.7g  42.1g S   6.2  47.2   0:28.70 G1 Refine#8
   440 udocker   20   0  354.0g 177.7g  42.1g S   6.2  47.2   0:55.65 rta_comms_log_m
   459 udocker   20   0  354.0g 177.7g  42.1g S   6.2  47.2   0:54.85 rta_comms_log_m
  3584 udocker   20   0  354.0g 177.7g  42.1g S   6.2  47.2   0:46.66 rta_comms_log_m

[Testing] CPU Usage

[Testing] Young Gen Time

[Production] CPU Usage

[Production] Young Gen Time

[Jackie-Jiang]

A heap dump should be helpful to identify the garbages triggering the GC