Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CPU performance issues with Graylog 4.2.12 + Java 17 #13306

Closed
giangi opened this issue Aug 26, 2022 · 9 comments · Fixed by #14005
Closed

CPU performance issues with Graylog 4.2.12 + Java 17 #13306

giangi opened this issue Aug 26, 2022 · 9 comments · Fixed by #14005
Assignees
@mpfz0r
Labels
bug performance triaged

Comments

@giangi
Copy link

giangi commented Aug 26, 2022 •
edited

Expected Behavior

The performance of Graylog 4.2.12 running on Java 17 (suggested by installation docs) should be comparable to what observed running on Java 8.

Current Behavior

For a more extensive discussion see the community thread: High CPU usage after upgrade from Graylog 4.1 + Java 8 to Graylog 4.2 + Java 17

After upgrading an installation from Graylog 4.1.14 + Java 8 to 4.2.12 to Java 17 I observed a noticeable (~4x) increase in CPU usage. Graylog and Java were the only updated parts of the installation.

Replacing Java 17 with Java 8 brought back CPU usage to the levels observed before the upgrade.

This was previously observed in a very similar way running Graylog 4.1.11 + Java 11. The performance drop vanished when switching to Java 8. The issue was curiously absent on 4.1.1 + Java 11. See community thread: Graylog 4.1.x high CPU usage after updating for log4j.

Possible Solution

I suspect, though I lack the expertise to support my claim, that some change introduced between 4.1.1 and 4.1.9 influences performance when running on Java > 8, and that this change is still in effect in 4.2.12.

Apart from the obvious reproduction and fix, which would be ideal if this is confirmed to be a non-local issue, clarify the supported and suggested version of Java in the docs, and that it may have an effect on the final performance.

Steps to Reproduce (for bugs)

  1. Run a load test on a Graylog 4.2.12 + Java 17 installation, observing CPU usage
  2. Run the same on a Graylog 4.2.12 + Java 8

Context

Normal operation of a Graylog cluster mainly ingesting UDP and TCP RFC5424 Syslog, running pipelines to enrich messages with fields (GROK, JSON, regular expressions) and a few alerts.

Your Environment

  • Graylog Version: 4.2.12
  • Java Version: 17 and 8
  • Elasticsearch Version: OSS 7.10.2
  • MongoDB Version: 4.2
  • Operating System: Debian 10 buster
  • Browser version: Not relevant
@giangi giangi added the bug label Aug 26, 2022
@janheise
Copy link
Contributor

janheise commented Aug 26, 2022 •
edited

Hi @giangi, which hardware platform are you using? I'm guessing Intel/AMD but just want to make sure.

@giangi
Copy link
Author

giangi commented Aug 26, 2022 •
edited

Hi @giangi, which hardware platform are you using? I'm guessing Intel/AMD but just want to make sure.

Hi @janheise , the graylog-server nodes are 4 vCPU VMs running on an on-premises libvirt/KVM cluster of physical hypervisors with Intel(R) Xeon(R) Gold 5118 CPUs.

(edit: if it is relevant the qemu-kvm machine is i440FX + PIIX, while CPU is Skylake-Server)

@bernd bernd added the triaged label Aug 31, 2022
@mpfz0r
Copy link
Member

mpfz0r commented Nov 8, 2022

@giangi I tried to reproduce this with 4.3.8 using Java 8 and 11 but couldn't. (Actually I've been also benchmarking Graylog 5.0 with Java 17 but that's not related)
Is this still an issue for you? Could you try to figure out which Graylog thread is responsible for the CPU consumption?

You could be running something like top -n 1 -H -p $(cat /tmp/graylog.pid) while the spike occurs.
The output would look somewhat like this:

Threads: 224 total,  11 running, 213 sleeping,   0 stopped,   0 zombie
%Cpu(s): 80,6 us, 18,8 sy,  0,0 ni,  0,6 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
MiB Mem :  64059,5 total,  17253,8 free,  29197,4 used,  17608,3 buff/cache
MiB Swap:  67540,0 total,  64477,7 free,   3062,3 used.  32476,9 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                              
 475357 marco     20   0    9,9g   3,7g  30492 R  94,1   5,9   0:05.71 inputbufferproc                                                                                                                                                      
 475402 marco     20   0    9,9g   3,7g  30492 R  70,6   5,9   0:09.65 processbufferpr                                                                                                                                                      
 475422 marco     20   0    9,9g   3,7g  30492 R  70,6   5,9   0:12.46 LocalKafkaMessa                                                                                                                                                      
 475403 marco     20   0    9,9g   3,7g  30492 R  58,8   5,9   0:09.24 processbufferpr                                                                                                                                                      
 475400 marco     20   0    9,9g   3,7g  30492 R  47,1   5,9   0:09.31 processbufferpr                                                                                                                                                      
 475401 marco     20   0    9,9g   3,7g  30492 R  47,1   5,9   0:09.24 processbufferpr                                                                                                                                                      
 475355 marco     20   0    9,9g   3,7g  30492 S  35,3   5,9   0:03.33 inputbufferproc                                                                                                                                                      
 475356 marco     20   0    9,9g   3,7g  30492 S  35,3   5,9   0:03.16 inputbufferproc                                                                                                                                                      
 475404 marco     20   0    9,9g   3,7g  30492 R  35,3   5,9   0:09.47 processbufferpr                                                                                                                                                      
 475393 marco     20   0    9,9g   3,7g  30492 S  23,5   5,9   0:03.99 outputbufferpro                                                                                                                                                      
 475394 marco     20   0    9,9g   3,7g  30492 S  23,5   5,9   0:04.00 outputbufferpro                                                                                                                                                      
 475392 marco     20   0    9,9g   3,7g  30492 S  17,6   5,9   0:04.01 outputbufferpro                                                                                                                                                      
 476806 marco     20   0    9,9g   3,7g  30492 R  17,6   5,9   0:01.85 netty-transport                                                                                                                                                      
 476807 marco     20   0    9,9g   3,7g  30492 R  17,6   5,9   0:01.87 netty-transport                                                                                                                                                      
 476808 marco     20   0    9,9g   3,7g  30492 S  17,6   5,9   0:01.85 netty-transport                                                                                                                                                      
 475530 marco     20   0    9,9g   3,7g  30492 S  11,8   5,9   0:00.86 outputbuffer-pr                                                                                                                                                      
 475531 marco     20   0    9,9g   3,7g  30492 S  11,8   5,9   0:00.88 outputbuffer-pr                                                                                                                                                      
 476809 marco     20   0    9,9g   3,7g  30492 S  11,8   5,9   0:01.83 netty-transport

Thanks :)

@giangi
Copy link
Author

giangi commented Nov 12, 2022

Hi @mpfz0r

Is this still an issue for you? Could you try to figure out which Graylog thread is responsible for the CPU consumption?

I did not make any further change to the cluster on which I was experiencing the issue, since performance was a concern being in production. I also admit that when @janheise initially replied, I took a very naive look at some his work and other code (e.g. the POM for 4.2.12) which made me thing that since the source/target was Java 8, maybe 4.2 was really never meant to be run reliably on higher Java versions. My sentiment since then has been that if this issue helped the developers to correct some compatibility issue early on, it was fine, but otherwise using Java 8 was fine until there was at least some security support for it and there was no benefit in pushing for higher versions.

Anyway, I could be able to sneak in a switch back to Java 17 on just one of the production nodes, just to briefly see the thread performance is different between a Java 8 and Java 17 node, but cannot give any guarantee due to the production nature.

May I ask if in your opinion this could be of any real help with current development or real installations? I am asking this because due to what I already said and given the announcements about Graylog 5.0 targeting Java 17 I admit my plan was to be very conservative by doing:

  • Update to 4.3 + Java 8, not expecting significant performance changes
  • Update to 5.0 + Java 17, crossing all available appendages not to see any issue due to the more "official" support for Java 17

tackling issues only when no workaround was available.

@mpfz0r
Copy link
Member

mpfz0r commented Nov 14, 2022

Hello @giangi

since the source/target was Java 8, maybe 4.2 was really never meant to be run reliably on higher Java versions

That's the Java byte code target. Newer Java versions can run older byte code without problem.
We also documented that running with Java 11 was possible. So yeah, even 4.2 should work just fine with newer Java versions.

May I ask if in your opinion this could be of any real help with current development or real installations?

It might be. Knowing the thread that is responsible will however only give us a first clue.
Depending on how quickly you can provoke that issue, we would need more details from your setup
and probably more debugging iterations.

What we could also do, is that you remember to get us the top Graylog threads, the next time you hit this issue, and we take it from there. Which is hopefully never 🤞

@giangi
Copy link
Author

giangi commented Nov 15, 2022

Hi @mpfz0r , thank you for the explanation, and sorry for my ignorance. I have just attempted the test you suggested.

I ran top -b -H -d 30 -n 2 -p <graylog_pid> to gather 30 seconds worth of data for two nodes, behind the same load balancer, both with Graylog 4.2.12-1 (Debian package) and only different versions of Java.

Java 1.8 (Temurin build 1.8.0_345-b01) 

top - 15:43:13 up 304 days, 23:13,  1 user,  load average: 0,66, 0,71, 0,70
Threads: 192 total,   0 running, 192 sleeping,   0 stopped,   0 zombie
%Cpu(s):  9,5 us,  2,1 sy,  0,0 ni, 87,4 id,  0,9 wa,  0,0 hi,  0,2 si,  0,0 st
MiB Mem :   7976,3 total,    363,6 free,   5072,8 used,   2539,9 buff/cache
MiB Swap:   1952,0 total,   1950,2 free,      1,8 used.   2738,7 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
15337 graylog   20   0 8155268   4,4g  97456 S  11,3  56,4   5172:08 LocalKafkaMessa
15322 graylog   20   0 8155268   4,4g  97456 S   3,2  56,4   1323:44 processbufferpr
15325 graylog   20   0 8155268   4,4g  97456 S   3,2  56,4   1322:10 processbufferpr
15321 graylog   20   0 8155268   4,4g  97456 S   3,1  56,4   1323:24 processbufferpr
15323 graylog   20   0 8155268   4,4g  97456 S   3,1  56,4   1321:40 processbufferpr
15324 graylog   20   0 8155268   4,4g  97456 S   3,1  56,4   1323:03 processbufferpr
15308 graylog   20   0 8155268   4,4g  97456 S   2,8  56,4   1183:32 inputbufferproc
15307 graylog   20   0 8155268   4,4g  97456 S   1,3  56,4 510:53.48 inputbufferproc
15306 graylog   20   0 8155268   4,4g  97456 S   1,2  56,4 511:27.95 inputbufferproc
15403 graylog   20   0 8155268   4,4g  97456 S   1,2  56,4 375:40.72 netty-transport
15404 graylog   20   0 8155268   4,4g  97456 S   1,2  56,4 529:24.90 netty-transport
15318 graylog   20   0 8155268   4,4g  97456 S   0,8  56,4 323:52.31 outputbufferpro
15320 graylog   20   0 8155268   4,4g  97456 S   0,8  56,4 321:23.61 outputbufferpro
15402 graylog   20   0 8155268   4,4g  97456 S   0,8  56,4 495:37.33 netty-transport
15319 graylog   20   0 8155268   4,4g  97456 S   0,8  56,4 321:31.72 outputbufferpro
15400 graylog   20   0 8155268   4,4g  97456 S   0,6  56,4 255:58.15 netty-transport
15462 graylog   20   0 8155268   4,4g  97456 S   0,5  56,4 106:53.87 outputbuffer-pr
15466 graylog   20   0 8155268   4,4g  97456 S   0,4  56,4 106:41.97 outputbuffer-pr
15468 graylog   20   0 8155268   4,4g  97456 S   0,4  56,4 106:48.05 outputbuffer-pr
15470 graylog   20   0 8155268   4,4g  97456 S   0,4  56,4 106:55.27 outputbuffer-pr
15278 graylog   20   0 8155268   4,4g  97456 S   0,3  56,4 151:45.67 java
15279 graylog   20   0 8155268   4,4g  97456 S   0,3  56,4 151:49.16 java
15281 graylog   20   0 8155268   4,4g  97456 S   0,3  56,4 151:52.51 java
15463 graylog   20   0 8155268   4,4g  97456 S   0,3  56,4 106:13.41 outputbuffer-pr
15465 graylog   20   0 8155268   4,4g  97456 S   0,3  56,4 106:48.12 outputbuffer-pr
15280 graylog   20   0 8155268   4,4g  97456 S   0,3  56,4 151:49.70 java
15464 graylog   20   0 8155268   4,4g  97456 S   0,2  56,4 106:52.19 outputbuffer-pr
15469 graylog   20   0 8155268   4,4g  97456 S   0,2  56,4 106:40.57 outputbuffer-pr
15467 graylog   20   0 8155268   4,4g  97456 S   0,2  56,4 106:45.79 outputbuffer-pr
15283 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  44:25.04 VM Thread
15393 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  56:20.33 grizzly-nio-ker
15460 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  39:32.66 http-worker-14
15292 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  60:12.49 VM Periodic Tas
15313 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  23:59.60 I/O dispatcher
15316 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  22:38.95 I/O dispatcher
15328 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  13:58.68 scheduled-daemo
15399 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  39:29.04 http-worker-0
15437 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  39:23.84 http-worker-1
15440 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  39:18.19 http-worker-4
15441 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  39:22.68 http-worker-5
15442 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  39:21.65 http-worker-6
15447 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  39:25.36 http-worker-8
15454 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  39:26.67 http-worker-10
15459 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  39:30.64 http-worker-13
15461 graylog   20   0 8155268   4,4g  97456 S   0,1  56,4  39:19.76 http-worker-15
15302 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   3:50.88 cluster-Cluster
15309 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:58.91 scheduled-daemo
15314 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  24:24.86 I/O dispatcher
15317 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:56.07 scheduled-daemo
15359 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:34.37 scheduled-5
15367 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:56.04 scheduled-daemo
15369 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:56.52 scheduled-daemo
15370 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:56.47 scheduled-daemo
15373 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:56.87 scheduled-daemo
15374 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  14:00.38 scheduled-daemo
15382 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:55.40 scheduled-daemo
15383 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  14:01.51 scheduled-daemo
15385 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:56.03 scheduled-daemo
15389 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   3:59.27 Thread-5
15438 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  39:20.09 http-worker-2
15439 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  39:18.64 http-worker-3
15443 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  39:16.64 http-worker-7
15449 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  39:24.98 http-worker-9
15457 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  39:25.43 http-worker-11
15458 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  39:25.57 http-worker-12
15546 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.36 proxied-request
15276 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:00.00 java
15277 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:06.55 java
15282 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   9:52.21 java
15284 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:14.89 Reference Handl
15285 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.85 Finalizer
15286 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:00.00 Surrogate Locke
15287 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:00.00 Signal Dispatch
15288 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   1:59.19 C2 CompilerThre
15289 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   2:09.90 C2 CompilerThre
15290 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   1:27.48 C1 CompilerThre
15291 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:00.00 Service Thread
15300 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   3:55.98 CleanCursors-1-
15303 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   3:44.39 cluster-Cluster
15304 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   4:00.95 cluster-Cluster
15312 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   3:35.58 pool-10-thread-
15315 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   4:52.88 I/O dispatcher
15327 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:59.57 scheduled-daemo
15329 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   3:55.43 eventbus-handle
15330 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:00.00 FailureHandling
15332 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:00.05 InputSetupServi
15344 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  14:00.50 scheduled-daemo
15345 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:00.00 output-setup-se
15348 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:00.00 JobSchedulerSer
15352 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:33.83 scheduled-0
15353 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:31.54 scheduled-1
15354 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:55.58 scheduled-daemo
15356 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:37.29 scheduled-2
15357 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:41.73 scheduled-3
15358 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:41.77 scheduled-4
15360 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:57.61 scheduled-daemo
15361 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:56.66 scheduled-daemo
15362 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:55.26 scheduled-daemo
15363 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:57.43 scheduled-daemo
15364 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:56.05 scheduled-daemo
15365 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:52.22 scheduled-daemo
15366 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:57.77 scheduled-daemo
15371 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:57.30 scheduled-daemo
15375 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:57.67 scheduled-daemo
15377 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:56.40 scheduled-daemo
15378 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:53.59 scheduled-daemo
15379 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:54.54 scheduled-daemo
15380 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:53.62 scheduled-daemo
15381 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:58.97 scheduled-daemo
15384 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:56.48 scheduled-daemo
15386 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  13:56.29 scheduled-daemo
15387 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:59.85 server-timer
15390 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:40.30 scheduled-6
15391 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:39.02 scheduled-7
15392 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   5:15.86 HttpServer-0
15394 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   3:54.98 eventbus-handle
15395 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   4:08.73 netty-transport
15396 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   3:29.02 netty-transport
15401 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:01.21 netty-transport
15405 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:14.02 SessionValidati
15406 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:44.55 scheduled-8
15407 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:42.30 scheduled-9
15435 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:41.29 scheduled-10
15436 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:43.94 scheduled-11
15444 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.03 proxied-request
15445 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.16 proxied-request
15446 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.31 proxied-request
15448 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:47.94 scheduled-12
15450 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:44.53 scheduled-13
15451 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.11 proxied-request
15452 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.16 proxied-request
15453 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.01 proxied-request
15455 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:39.61 scheduled-14
15456 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:43.92 scheduled-15
15471 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:52.54 scheduled-16
15477 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:39.68 scheduled-17
15478 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:46.07 scheduled-18
15479 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:39.30 scheduled-19
15480 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.08 proxied-request
15481 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.23 proxied-request
15482 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.09 proxied-request
15483 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:37.85 scheduled-20
15484 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:38.23 scheduled-21
15485 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:36.62 scheduled-22
15486 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:37.98 scheduled-23
15487 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:07.99 proxied-request
15488 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.03 proxied-request
15489 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.03 proxied-request
15490 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:48.47 scheduled-24
15491 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:47.05 scheduled-25
15492 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:45.75 scheduled-26
15493 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:48.79 scheduled-27
15494 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:38.49 scheduled-28
15495 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   8:35.39 scheduled-29
15497 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.42 proxied-request
15498 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.28 proxied-request
15499 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.35 proxied-request
15500 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  36:21.06 prometheus-http
15501 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.33 proxied-request
15502 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.19 proxied-request
15503 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.33 proxied-request
15532 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.28 proxied-request
15533 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.65 proxied-request
15534 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.54 proxied-request
15541 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.50 proxied-request
15542 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.29 proxied-request
15543 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.17 proxied-request
15544 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.31 proxied-request
15545 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.29 proxied-request
15547 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.21 proxied-request
15548 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.11 proxied-request
15549 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.42 proxied-request
15551 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.39 proxied-request
15552 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:08.28 proxied-request
15553 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  36:16.82 prometheus-http
15554 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:42.09 metrics-meter-t
15555 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:35.88 metrics-meter-t
15556 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  36:22.17 prometheus-http
15557 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  36:34.83 prometheus-http
15590 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  10:41.87 netty-transport
15591 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4  36:29.05 prometheus-http
15625 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:21.71 netty-transport
15628 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   2:11.72 kafka-journal-s
15629 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:13.35 kafka-journal-s
15861 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:00.92 netty-transport
16495 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   1:20.54 query-engine-0
16496 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   1:20.59 query-engine-1
16707 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   1:19.69 query-engine-2
16708 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   1:20.15 query-engine-3
19104 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:00.92 netty-transport
25132 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:00.01 OkHttp Connecti
25147 graylog   20   0 8155268   4,4g  97456 S   0,0  56,4   0:00.00 Okio Watchdog

Java 17 (Temurin build 17.0.5+8) 

top - 15:43:15 up 304 days, 23:24,  1 user,  load average: 3,67, 3,20, 2,67
Threads: 192 total,   6 running, 186 sleeping,   0 stopped,   0 zombie
%Cpu(s): 57,6 us,  2,6 sy,  0,0 ni, 38,2 id,  0,4 wa,  0,0 hi,  1,1 si,  0,0 st
MiB Mem :   7976,3 total,   1600,0 free,   3618,1 used,   2758,2 buff/cache
MiB Swap:   1952,0 total,   1948,2 free,      3,8 used.   4193,2 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
26594 graylog   20   0 8263728   2,9g  55304 R  53,8  37,5  12:24.16 netty-transport
26598 graylog   20   0 8263728   2,9g  55304 R  52,4  37,5  12:08.91 netty-transport
26597 graylog   20   0 8263728   2,9g  55304 R  36,7  37,5   8:38.33 netty-transport
26599 graylog   20   0 8263728   2,9g  55304 R  33,7  37,5   7:33.32 netty-transport
26520 graylog   20   0 8263728   2,9g  55304 S  23,2  37,5   5:23.43 LocalKafkaMessa
26490 graylog   20   0 8263728   2,9g  55304 S   7,0  37,5   1:35.26 inputbufferproc
26506 graylog   20   0 8263728   2,9g  55304 S   3,8  37,5   0:51.09 processbufferpr
26504 graylog   20   0 8263728   2,9g  55304 R   3,7  37,5   0:50.97 processbufferpr
26505 graylog   20   0 8263728   2,9g  55304 S   3,6  37,5   0:51.03 processbufferpr
26507 graylog   20   0 8263728   2,9g  55304 S   3,6  37,5   0:51.20 processbufferpr
26503 graylog   20   0 8263728   2,9g  55304 S   3,5  37,5   0:51.12 processbufferpr
26488 graylog   20   0 8263728   2,9g  55304 S   1,4  37,5   0:19.03 inputbufferproc
26489 graylog   20   0 8263728   2,9g  55304 R   1,3  37,5   0:18.97 inputbufferproc
26462 graylog   20   0 8263728   2,9g  55304 S   0,9  37,5   1:08.80 C2 CompilerThre
26502 graylog   20   0 8263728   2,9g  55304 S   0,8  37,5   0:11.20 outputbufferpro
26500 graylog   20   0 8263728   2,9g  55304 S   0,8  37,5   0:11.17 outputbufferpro
26501 graylog   20   0 8263728   2,9g  55304 S   0,8  37,5   0:11.23 outputbufferpro
26637 graylog   20   0 8263728   2,9g  55304 S   0,5  37,5   0:05.92 outputbuffer-pr
26582 graylog   20   0 8263728   2,9g  55304 S   0,5  37,5   0:05.80 outputbuffer-pr
26596 graylog   20   0 8263728   2,9g  55304 S   0,5  37,5   0:06.28 outputbuffer-pr
26581 graylog   20   0 8263728   2,9g  55304 S   0,5  37,5   0:06.02 outputbuffer-pr
26772 graylog   20   0 8263728   2,9g  55304 S   0,4  37,5   0:00.56 prometheus-http
26583 graylog   20   0 8263728   2,9g  55304 S   0,4  37,5   0:06.07 outputbuffer-pr
26584 graylog   20   0 8263728   2,9g  55304 S   0,4  37,5   0:06.14 outputbuffer-pr
26585 graylog   20   0 8263728   2,9g  55304 S   0,3  37,5   0:05.87 outputbuffer-pr
26593 graylog   20   0 8263728   2,9g  55304 S   0,3  37,5   0:06.05 outputbuffer-pr
26472 graylog   20   0 8263728   2,9g  55304 S   0,3  37,5   0:04.32 GC Thread#2
26592 graylog   20   0 8263728   2,9g  55304 S   0,3  37,5   0:06.08 outputbuffer-pr
26451 graylog   20   0 8263728   2,9g  55304 S   0,3  37,5   0:04.31 GC Thread#0
26471 graylog   20   0 8263728   2,9g  55304 S   0,3  37,5   0:04.29 GC Thread#1
26473 graylog   20   0 8263728   2,9g  55304 S   0,3  37,5   0:04.30 GC Thread#3
26575 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:02.50 grizzly-nio-ker
26588 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.80 http-worker-2
26616 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:02.03 http-worker-8
26495 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.10 I/O dispatcher
26496 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.08 I/O dispatcher
26586 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.74 http-worker-0
26589 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.68 http-worker-3
26591 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.76 http-worker-5
26600 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.76 http-worker-6
26617 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.70 http-worker-9
26620 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.90 http-worker-12
26626 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.75 http-worker-15
26456 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:00.74 VM Thread
26466 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:00.80 VM Periodic Tas
26497 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.08 I/O dispatcher
26587 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.68 http-worker-1
26590 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.78 http-worker-4
26614 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.77 http-worker-7
26618 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.82 http-worker-10
26619 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.76 http-worker-11
26621 graylog   20   0 8263728   2,9g  55304 S   0,1  37,5   0:01.84 http-worker-13
26455 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.21 G1 Service
26463 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:10.49 C1 CompilerThre
26465 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.28 Notification Th
26485 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.11 cluster-Cluster
26491 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.30 scheduled-daemo
26499 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.29 scheduled-daemo
26509 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.26 scheduled-daemo
26518 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.31 scheduled-daemo
26519 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.29 scheduled-daemo
26527 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.03 scheduled-3
26532 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.27 scheduled-daemo
26538 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.27 scheduled-daemo
26539 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.28 scheduled-daemo
26543 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.28 scheduled-daemo
26550 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.30 scheduled-daemo
26552 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.30 scheduled-daemo
26555 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.28 scheduled-daemo
26556 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.28 scheduled-daemo
26563 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.29 scheduled-daemo
26568 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.26 scheduled-daemo
26576 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 eventbus-handle
26578 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.11 netty-transport
26624 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:01.70 http-worker-14
26650 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.03 scheduled-21
26683 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.08 proxied-request
26684 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26449 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.00 java
26450 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:05.85 java
26452 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.00 G1 Main Marker
26453 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.58 G1 Conc#0
26454 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.03 G1 Refine#0
26457 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.00 Reference Handl
26458 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.00 Finalizer
26459 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.00 Signal Dispatch
26460 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.04 Service Thread
26461 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.03 Monitor Deflati
26464 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:01.48 Sweeper thread
26467 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.00 Common-Cleaner
26482 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.10 CleanCursors-1-
26484 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.12 cluster-Cluster
26487 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.12 cluster-Cluster
26492 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.03 OkHttp Connecti
26493 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.02 Okio Watchdog
26494 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.11 pool-10-thread-
26498 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.06 I/O dispatcher
26510 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 eventbus-handle
26511 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.00 FailureHandling
26513 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.08 InputSetupServi
26523 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.05 scheduled-0
26525 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.28 scheduled-daemo
26526 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.03 scheduled-1
26528 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.18 scheduled-4
26533 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.28 scheduled-daemo
26535 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.00 output-setup-se
26536 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.02 scheduled-2
26537 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.04 scheduled-5
26544 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.30 scheduled-daemo
26545 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.30 scheduled-daemo
26546 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.00 JobSchedulerSer
26551 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.28 scheduled-daemo
26554 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.28 scheduled-daemo
26557 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.28 scheduled-daemo
26558 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.28 scheduled-daemo
26559 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.27 scheduled-daemo
26560 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.30 scheduled-daemo
26561 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.30 scheduled-daemo
26562 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.30 scheduled-daemo
26565 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.27 scheduled-daemo
26566 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.27 scheduled-daemo
26567 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.27 scheduled-daemo
26569 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.02 idle-timeout-ta
26571 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.14 HTTP-Dispatcher
26572 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.03 scheduled-6
26573 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.04 scheduled-7
26574 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.18 HttpServer-0
26577 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.17 netty-transport
26595 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.00 netty-transport
26601 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.00 SessionValidati
26602 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.05 scheduled-8
26603 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.06 scheduled-9
26604 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.04 scheduled-10
26605 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.02 scheduled-11
26622 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.03 scheduled-12
26623 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.05 scheduled-13
26631 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.04 scheduled-14
26632 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.02 scheduled-15
26641 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.03 scheduled-16
26642 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.03 scheduled-17
26643 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.04 scheduled-18
26644 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.04 scheduled-19
26649 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.02 scheduled-20
26651 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.03 scheduled-22
26652 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.02 scheduled-23
26665 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.04 scheduled-24
26666 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.02 scheduled-25
26668 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.03 scheduled-26
26669 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.03 scheduled-27
26670 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.02 scheduled-28
26671 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.03 scheduled-29
26672 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.09 proxied-request
26673 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.09 proxied-request
26674 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26675 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26676 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.08 proxied-request
26677 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.08 proxied-request
26678 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26679 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26680 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.05 proxied-request
26681 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.61 prometheus-http
26682 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.08 proxied-request
26685 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.08 proxied-request
26686 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.06 proxied-request
26687 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26688 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.06 proxied-request
26689 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26690 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26695 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26696 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26697 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26700 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26701 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.08 proxied-request
26702 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26704 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26705 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.08 proxied-request
26706 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.06 proxied-request
26709 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26710 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26711 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26713 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26714 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.07 proxied-request
26719 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.53 prometheus-http
26721 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.02 metrics-meter-t
26722 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.01 metrics-meter-t
26723 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.00 G1 Refine#1
26724 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.00 G1 Refine#2
26731 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.49 prometheus-http
26776 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.08 kafka-journal-s
26779 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.42 prometheus-http
26781 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.00 kafka-journal-s
26784 graylog   20   0 8263728   2,9g  55304 S   0,0  37,5   0:00.34 query-engine-0

Unfortunately I did not find any way to make the COMMAND longer. I tried setting COLUMNS like suggested in the manpage, but It does not seem to affect output. I hope it gives you enough for the clue you are looking for, let me know if I can do something else for it or I am missing something.

@mpfz0r
Copy link
Member

mpfz0r commented Nov 16, 2022

Hi @giangi thanks for taking the time. Much appreciated.
netty-transport narrows this down to certain Inputs. The processing (pipelines, extractors, etc.) are not concerned.

Can you tell us a bit more about your setup? Which inputs are you running and how are they configured? Which one is receiving the most traffic? Is the traffic equally distributed between the two nodes?

@giangi
Copy link
Author

giangi commented Nov 16, 2022 •
edited

Can you tell us a bit more about your setup? Which inputs are you running and how are they configured? Which one is receiving the most traffic? Is the traffic equally distributed between the two nodes?

The cluster is running global (all nodes) UDP and TCP syslog inputs.

Syslog UDP

allow_override_date: true
bind_address: 0.0.0.0
expand_structured_data: true
force_rdns: false
number_worker_threads: 4
override_source: <empty>
port: 1514
recv_buffer_size: 4194304
store_full_message: false

Syslog TCP

allow_override_date: true
bind_address: 0.0.0.0
expand_structured_data: true
force_rdns: false
max_message_size: 2097152
number_worker_threads: 4
override_source: <empty>
port: 1514
recv_buffer_size: 4194304
store_full_message: true
tcp_keepalive: false
tls_cert_file: <empty>
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: false
tls_key_file: <empty>
tls_key_password: ********
use_null_delimiter: false

Regarding traffic:

  • It is overwhelmingly dominated by UDP (~99%)
  • UDP balancing is message-per-message round-robin via IPVS
  • TCP balancing is connection-oriented round-robin via HAProxy (TCP tunnel)
  • UDP seems to achieve near-perfect balancing
  • TCP can be unbalanced due to being used by only a few clients, but it never seemed concerning due to the low traffic

Each node during the test was processing ~700 messages/s. Here is a graph showing input rates and CPU usage for the whole test for the three nodes.

Screenshot 2022-11-16 at 16-37-43 Graylog - Grafana

Let me know if there is some other interesting detail you would like to know!

@mpfz0r mpfz0r self-assigned this Nov 18, 2022
@mpfz0r
Copy link
Member

mpfz0r commented Nov 18, 2022

Hi @giangi
Thanks for the debugging infos. It actually helped me to find the issue. Working on a fix now 😅

mpfz0r added a commit that referenced this issue Nov 18, 2022
@mpfz0r
Only allocate the maximum needed size for one UDP packet
b88a759 
Don't allocate the size of the entire UDP socket buffer for each packet!

This has been wrong for a long time, but only came up as
a regression when using Java > 9.

On newer Java versions Netty does not use direct buffers via "Unsafe" to allocate memory.
It falls back to ByteBuffer.allocateDirect which has inferior performance.

This behavior could be restored by adding these JVM options:
```
-Dio.netty.tryReflectionSetAccessible=true
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
```
But since fixing the receive buffer size to 64k gives us the same
performance boost, we can avoid using these hacks, and not having
to rely on Unsafe is also a good idea.

Refs
 netty/netty#7650

 https://github.com/netty/netty/blame/e8df52e442629214e0355528c00e873e213f0139/common/src/main/java/io/netty/util/internal/PlatformDependent0.java#L979-L980

Fixes #13306
@mpfz0r mpfz0r mentioned this issue Nov 18, 2022
Merged
bernd pushed a commit that referenced this issue Nov 21, 2022
@mpfz0r
Only allocate the maximum needed size for one UDP packet (#14005)
e73ecfb 
Don't allocate the size of the entire UDP socket buffer for each packet!

This has been wrong for a long time, but only came up as
a regression when using Java > 9.

On newer Java versions Netty does not use direct buffers via "Unsafe" to allocate memory.
It falls back to ByteBuffer.allocateDirect which has inferior performance.

This behavior could be restored by adding these JVM options:
```
-Dio.netty.tryReflectionSetAccessible=true
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
```
But since fixing the receive buffer size to 64k gives us the same
performance boost, we can avoid using these hacks, and not having
to rely on Unsafe is also a good idea.

Refs
 netty/netty#7650

 https://github.com/netty/netty/blame/e8df52e442629214e0355528c00e873e213f0139/common/src/main/java/io/netty/util/internal/PlatformDependent0.java#L979-L980

Fixes #13306
bernd pushed a commit that referenced this issue Nov 21, 2022
@mpfz0r @bernd
Only allocate the maximum needed size for one UDP packet (#14005)
6d248b2 
Don't allocate the size of the entire UDP socket buffer for each packet!

This has been wrong for a long time, but only came up as
a regression when using Java > 9.

On newer Java versions Netty does not use direct buffers via "Unsafe" to allocate memory.
It falls back to ByteBuffer.allocateDirect which has inferior performance.

This behavior could be restored by adding these JVM options:
```
-Dio.netty.tryReflectionSetAccessible=true
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
```
But since fixing the receive buffer size to 64k gives us the same
performance boost, we can avoid using these hacks, and not having
to rely on Unsafe is also a good idea.

Refs
 netty/netty#7650

 https://github.com/netty/netty/blame/e8df52e442629214e0355528c00e873e213f0139/common/src/main/java/io/netty/util/internal/PlatformDependent0.java#L979-L980

Fixes #13306

(cherry picked from commit e73ecfb)
@bernd bernd mentioned this issue Nov 21, 2022
Merged
mpfz0r added a commit that referenced this issue Nov 21, 2022
@mpfz0r
Only allocate the maximum needed size for one UDP packet (#14005)
8ed5cdf 
Don't allocate the size of the entire UDP socket buffer for each packet!

This has been wrong for a long time, but only came up as
a regression when using Java > 9.

On newer Java versions Netty does not use direct buffers via "Unsafe" to allocate memory.
It falls back to ByteBuffer.allocateDirect which has inferior performance.

This behavior could be restored by adding these JVM options:
```
-Dio.netty.tryReflectionSetAccessible=true
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
```
But since fixing the receive buffer size to 64k gives us the same
performance boost, we can avoid using these hacks, and not having
to rely on Unsafe is also a good idea.

Refs
 netty/netty#7650

 https://github.com/netty/netty/blame/e8df52e442629214e0355528c00e873e213f0139/common/src/main/java/io/netty/util/internal/PlatformDependent0.java#L979-L980

Fixes #13306

(cherry picked from commit e73ecfb)
@mpfz0r mpfz0r mentioned this issue Nov 21, 2022
Merged
mpfz0r added a commit that referenced this issue Nov 21, 2022
@bernd @mpfz0r
Only allocate the maximum needed size for one UDP packet (#14005) (#1…
b91b2d5 
…4016)

Don't allocate the size of the entire UDP socket buffer for each packet!

This has been wrong for a long time, but only came up as
a regression when using Java > 9.

On newer Java versions Netty does not use direct buffers via "Unsafe" to allocate memory.
It falls back to ByteBuffer.allocateDirect which has inferior performance.

This behavior could be restored by adding these JVM options:
```
-Dio.netty.tryReflectionSetAccessible=true
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
```
But since fixing the receive buffer size to 64k gives us the same
performance boost, we can avoid using these hacks, and not having
to rely on Unsafe is also a good idea.

Refs
 netty/netty#7650

 https://github.com/netty/netty/blame/e8df52e442629214e0355528c00e873e213f0139/common/src/main/java/io/netty/util/internal/PlatformDependent0.java#L979-L980

Fixes #13306

(cherry picked from commit e73ecfb)

Co-authored-by: Marco Pfatschbacher <marco@graylog.com>
bernd pushed a commit that referenced this issue Nov 21, 2022
@mpfz0r
Only allocate the maximum needed size for one UDP packet (#14005) (#1…
31a803d 
…4019)

Don't allocate the size of the entire UDP socket buffer for each packet!

This has been wrong for a long time, but only came up as
a regression when using Java > 9.

On newer Java versions Netty does not use direct buffers via "Unsafe" to allocate memory.
It falls back to ByteBuffer.allocateDirect which has inferior performance.

This behavior could be restored by adding these JVM options:
```
-Dio.netty.tryReflectionSetAccessible=true
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
```
But since fixing the receive buffer size to 64k gives us the same
performance boost, we can avoid using these hacks, and not having
to rely on Unsafe is also a good idea.

Refs
 netty/netty#7650

 https://github.com/netty/netty/blame/e8df52e442629214e0355528c00e873e213f0139/common/src/main/java/io/netty/util/internal/PlatformDependent0.java#L979-L980

Fixes #13306

(cherry picked from commit e73ecfb)
bernd pushed a commit that referenced this issue Dec 9, 2022
@mpfz0r @bernd
Only allocate the maximum needed size for one UDP packet (#14005)
d3ddd05 
Don't allocate the size of the entire UDP socket buffer for each packet!

This has been wrong for a long time, but only came up as
a regression when using Java > 9.

On newer Java versions Netty does not use direct buffers via "Unsafe" to allocate memory.
It falls back to ByteBuffer.allocateDirect which has inferior performance.

This behavior could be restored by adding these JVM options:
```
-Dio.netty.tryReflectionSetAccessible=true
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
```
But since fixing the receive buffer size to 64k gives us the same
performance boost, we can avoid using these hacks, and not having
to rely on Unsafe is also a good idea.

Refs
 netty/netty#7650

 https://github.com/netty/netty/blame/e8df52e442629214e0355528c00e873e213f0139/common/src/main/java/io/netty/util/internal/PlatformDependent0.java#L979-L980

Fixes #13306
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mpfz0r mpfz0r

Labels
bug performance triaged
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Only allocate the maximum needed size for one UDP packet Graylog2/graylog2-server
5 participants
@bernd @boosty @janheise @giangi @mpfz0r