[Bug] skywalking 9.1.0 startup failed. #9349
Description
Search before asking
- I had searched in the issues and found no similar issues.
Apache SkyWalking Component
OAP server (apache/skywalking)
What happened
hi,some errors logs when i startup skywalking 9.1.0.
What you expected to happen
There is insufficient memory for the Java Runtime Environment to continue.
Cannot create worker GC thread. Out of system resources.
Possible reasons:
The system is out of physical RAM or swap space
The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize=
JVM is running with Unscaled Compressed Oops mode in which the Java heap is
placed in the first 4GB address space. The Java Heap base address is the
maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
to set the Java Heap base and to place the Java Heap above 4GB virtual address.
This output file may be truncated or incomplete.
Out of Memory Error (workerManager.hpp:70), pid=1, tid=1
JRE version: (11.0.15+10) (build )
Java VM: OpenJDK 64-Bit Server VM (11.0.15+10, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
Core dump will be written. Default location: /skywalking/core.1
--------------- S U M M A R Y ------------
Command Line: -Xms2G org.apache.skywalking.oap.server.starter.OAPServerStartUp
Host: Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz, 4 cores, 15G, Ubuntu 22.04 LTS
Time: Mon Jul 18 13:59:22 2022 CST elapsed time: 0.004166 seconds (0d 0h 0m 0s)
--------------- T H R E A D ---------------
Current thread (0x0000558d4e315000): JavaThread "Unknown thread" [_thread_in_vm, id=1, stack(0x00007ffc44e7f000,0x00007ffc4567c000)]
Stack: [0x00007ffc44e7f000,0x00007ffc4567c000], sp=0x00007ffc45676410, free space=8157k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xf424ca] VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x1ca
V [libjvm.so+0xf4338b] VMError::report_and_die(Thread*, char const*, int, unsigned long, VMErrorType, char const*, __va_list_tag*)+0x2b
V [libjvm.so+0x6cf005] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0xd5
V [libjvm.so+0xf6f58b] AbstractWorkGang::add_workers(unsigned int, bool)+0x14b
V [libjvm.so+0x7aa8c5] G1CollectedHeap::G1CollectedHeap(G1CollectorPolicy*)+0x5b5
V [libjvm.so+0x79c3b9] G1Arguments::create_heap()+0x69
V [libjvm.so+0xefdc11] Universe::initialize_heap()+0x31
V [libjvm.so+0xefdec6] universe_init()+0x46
V [libjvm.so+0x8a4207] init_globals()+0x57
V [libjvm.so+0xed5537] Threads::create_vm(JavaVMInitArgs*, bool*)+0x357
V [libjvm.so+0x960262] JNI_CreateJavaVM+0x52
C [libjli.so+0x49af] JavaMain+0x8f
C [libjli.so+0x9d94] CallJavaMainInNewThread+0x74
"hs_err_pid1.log" 362L, 22259C
number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize=
JVM is running with Unscaled Compressed Oops mode in which the Java heap is
placed in the first 4GB address space. The Java Heap base address is the
maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
to set the Java Heap base and to place the Java Heap above 4GB virtual address.
This output file may be truncated or incomplete.
Out of Memory Error (workerManager.hpp:70), pid=1, tid=1
JRE version: (11.0.15+10) (build )
Java VM: OpenJDK 64-Bit Server VM (11.0.15+10, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
Core dump will be written. Default location: /skywalking/core.1
--------------- S U M M A R Y ------------
Command Line: -Xms2G org.apache.skywalking.oap.server.starter.OAPServerStartUp
Host: Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz, 4 cores, 15G, Ubuntu 22.04 LTS
Time: Mon Jul 18 13:59:22 2022 CST elapsed time: 0.004166 seconds (0d 0h 0m 0s)
--------------- T H R E A D ---------------
Current thread (0x0000558d4e315000): JavaThread "Unknown thread" [_thread_in_vm, id=1, stack(0x00007ffc44e7f000,0x00007ffc4567c000)]
Stack: [0x00007ffc44e7f000,0x00007ffc4567c000], sp=0x00007ffc45676410, free space=8157k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xf424ca] VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x1ca
V [libjvm.so+0xf4338b] VMError::report_and_die(Thread*, char const*, int, unsigned long, VMErrorType, char const*, __va_list_tag*)+0x2b
V [libjvm.so+0x6cf005] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0xd5
V [libjvm.so+0xf6f58b] AbstractWorkGang::add_workers(unsigned int, bool)+0x14b
V [libjvm.so+0x7aa8c5] G1CollectedHeap::G1CollectedHeap(G1CollectorPolicy*)+0x5b5
V [libjvm.so+0x79c3b9] G1Arguments::create_heap()+0x69
V [libjvm.so+0xefdc11] Universe::initialize_heap()+0x31
V [libjvm.so+0xefdec6] universe_init()+0x46
V [libjvm.so+0x8a4207] init_globals()+0x57
V [libjvm.so+0xed5537] Threads::create_vm(JavaVMInitArgs*, bool*)+0x357
V [libjvm.so+0x960262] JNI_CreateJavaVM+0x52
C [libjli.so+0x49af] JavaMain+0x8f
C [libjli.so+0x9d94] CallJavaMainInNewThread+0x74
C [libjli.so+0x637d] ContinueInNewThread+0x4d
C [libjli.so+0x7a00] JLI_Launch+0x1470
C [java+0xa73]
--------------- P R O C E S S ---------------
Threads class SMR info:
_java_thread_list=0x0000558d4e2d8770, length=0, elements={
Java Threads: ( => current thread )
Other Threads:
=>0x0000558d4e315000 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=1, stack(0x00007ffc44e7f000,0x00007ffc4567c000)]
Threads with active compile tasks:
VM state:not at safepoint (not fully initialized)
VM Mutex/Monitor currently owned by a thread: None
[error occurred during error reporting (printing compressed oops mode), id 0xb, SIGSEGV (0xb) at pc=0x00007f10f9afc543]
GC Heap History (0 events):
No events
Deoptimization events (0 events):
No events
Classes redefined (0 events):
No events
Internal exceptions (0 events):
No events
Events (2 events):
Event: 0.002 Protecting memory [0x00007ffc44e7f000,0x00007ffc44e83000] with protection modes 0
Event: 0.002 Loaded shared library /opt/java/openjdk/lib/libzip.so
Dynamic libraries:
558d4d2a6000-558d4d2a7000 r-xp 00000000 fd:11 30937083 /opt/java/openjdk/bin/java
558d4d4a7000-558d4d4a8000 r--p 00001000 fd:11 30937083 /opt/java/openjdk/bin/java
558d4d4a8000-558d4d4a9000 rw-p 00002000 fd:11 30937083 /opt/java/openjdk/bin/java
558d4e2d2000-558d4e355000 rw-p 00000000 00:00 0 [heap]
7f10e2d20000-7f10e30dd000 rw-p 00000000 00:00 0
7f10e30dd000-7f10e334d000 rwxp 00000000 00:00 0
7f10e334d000-7f10e366d000 ---p 00000000 00:00 0
7f10e366d000-7f10e38dd000 rwxp 00000000 00:00 0
7f10e38dd000-7f10eaba5000 ---p 00000000 00:00 0
7f10eaba5000-7f10eae15000 rwxp 00000000 00:00 0
7f10eae15000-7f10f20dd000 ---p 00000000 00:00 0
7f10f20dd000-7f10f7c00000 r--s 00000000 fd:11 30937404 /opt/java/openjdk/lib/modules
7f10f7c00000-7f10f7c07000 r-xp 00000000 fd:11 30937403 /opt/java/openjdk/lib/libzip.so
7f10f7c07000-7f10f7e07000 ---p 00007000 fd:11 30937403 /opt/java/openjdk/lib/libzip.so
7f10f7e07000-7f10f7e08000 r--p 00007000 fd:11 30937403 /opt/java/openjdk/lib/libzip.so
7f10f7e08000-7f10f7e09000 rw-p 00008000 fd:11 30937403 /opt/java/openjdk/lib/libzip.so
How to reproduce
I use docker-compose to start up skywalking.
version: '3'
services:
skywalking-oap:
image: apache/skywalking-oap-server:9.1.0
container_name: skywalking-oap
restart: always
ports:
- 11800:11800
- 12800:12800
environment:
SW_STORAGE: elasticsearch
SW_STORAGE_ES_CLUSTER_NODES: xx.xx.xx.xx:9200
SW_HEALTH_CHECKER: default
SW_TELEMETRY: prometheus
TZ: Asia/Shanghai
JAVA_OPTS: "-Xms4096m -Xmx4096m"
healthcheck:
test: ["CMD", "./bin/swctl", "ch"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
skywalking-ui:
image: apache/skywalking-ui:9.1.0
container_name: skywalking-ui
depends_on:
- skywalking-oap
links:
- skywalking-oap
restart: always
ports:
- 3080:8080
environment:
SW_OAP_ADDRESS: xx.xx.xx.xx:12800
Anything else
And i think have enough memory in centos.
total used free shared buff/cache available
Mem: 15615 3356 8493 0 3766 11929
Swap: 0 0 0
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct