OAP goes down...

[sooryanadar]

Hii,

The OAP server and entire Skywalking goes down frequently.

Skywalking Version:- 9.7
Java version:- openjdk version "11.0.22" 2024-01-16 LTS
Please find the attached logs below.
skywalking-oap-server.zip

Regards,
Surya

[CodePrometheus]

java.nio.channels.UnresolvedAddressException
KubernetesClientException: Operation: [get]  for kind: [Pod]  with name: [coredns-787d4945fb-5ltgf]  in namespace: [kube-system]  failed

Have you made sure the environment is ok?

[sooryanadar]

It's going down frequently and unable to figure it out why it's happening.

[wu-sheng]

If you are running with default configurations, it keeps data in H2 memory mode.

Pick a storage option to keep things stable.

[wu-sheng]

If you run with DB, make sure they are stable, and your OAP should have enough memory and CPU match your payload.

[sooryanadar]

Now shifted the DB to elastic, still facing the same issue.

2024-02-27 16:35:42,998 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 91 [pool-7-thread-1] INFO [] - Beginning to remove expired metrics from the storage.
2024-02-27 16:40:42,998 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 91 [pool-7-thread-1] INFO [] - Beginning to remove expired metrics from the storage.
2024-02-27 16:44:54,479 - com.linecorp.armeria.server.logging.LoggingService - 95 [armeria-eventloop-epoll-5-1] WARN [] - [sreqId=e7a58a93, chanId=57456519, raddr=127.0.0.1:59484, laddr=127.0.0.1:12800][h2c://apache-skywaking-srv/internal/l7check#HEAD] Request: {startTime=2024-02-27T11:14:21.606Z(1709032461606315), length=0B, duration=32847ms(32847260022ns), scheme=none+h2c, name=HEAD, headers=[EOS, :method=HEAD, :authority=localhost:12800, :scheme=http, :path=/internal/l7check, if-none-match="healthy", prefer=wait=60, user-agent=armeria/1.25.0]}
2024-02-27 16:44:54,479 - com.linecorp.armeria.server.logging.LoggingService - 156 [armeria-eventloop-epoll-5-1] WARN [] - [sreqId=e7a58a93, chanId=57456519, raddr=127.0.0.1:59484, laddr=127.0.0.1:12800][h2c://apache-skywaking-srv/internal/l7check#HEAD] Response: {startTime=2024-02-27T11:14:21.606Z(1709032461606891), length=0B, duration=32847ms(32847173006ns), totalDuration=32847ms(32847750230ns), cause=com.linecorp.armeria.common.stream.ClosedStreamException: received a RST_STREAM frame: CANCEL, headers=[:status=0]}
com.linecorp.armeria.common.stream.ClosedStreamException: received a RST_STREAM frame: CANCEL
at com.linecorp.armeria.server.Http2RequestDecoder.onRstStreamRead(Http2RequestDecoder.java:396) [armeria-1.25.0.jar:?]
at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onRstStreamRead(Http2FrameListenerDecorator.java:59) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.http2.Http2MaxRstFrameListener.onRstStreamRead(Http2MaxRstFrameListener.java:56) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onRstStreamRead(Http2FrameListenerDecorator.java:59) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onRstStreamRead(DefaultHttp2ConnectionDecoder.java:442) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onRstStreamRead(Http2InboundFrameLogger.java:80) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readRstStreamFrame(DefaultHttp2FrameReader.java:509) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:259) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:63) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:63) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) [netty-codec-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152) [netty-handler-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) [netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509) [netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) [netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
2024-02-27 16:45:42,998 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 91 [pool-7-thread-1] INFO [] - Beginning to remove expired metrics from the storage.
2024-02-27 16:50:42,997 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 91 [pool-7-thread-1] INFO [] - Beginning to remove expired metrics from the storage.
2024-02-27 16:55:42,998 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 91 [pool-7-thread-1] INFO [] - Beginning to remove expired metrics from the storage.
2024-02-27 16:59:44,017 - org.apache.skywalking.oap.server.receiver.log.provider.handler.grpc.LogReportServiceGrpcHandler - 101 [grpc-default-executor-13] ERROR [] - CANCELLED: client cancelled
io.grpc.StatusRuntimeException: CANCELLED: client cancelled
at io.grpc.Status.asRuntimeException(Status.java:530) ~[grpc-api-1.53.0.jar:1.53.0]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:291) [grpc-stub-1.53.0.jar:1.53.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:378) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:365) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:923) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) [grpc-core-1.53.0.jar:1.53.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
2024-02-27 17:00:42,999 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 91 [pool-7-thread-1] INFO [] - Beginning to remove expired metrics from the storage.
2024-02-27 17:05:42,998 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 91 [pool-7-thread-1] INFO [] - Beginning to remove expired metrics from the storage.
2024-02-27 17:10:42,998 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 91 [pool-7-thread-1] INFO [] - Beginning to remove expired metrics from the storage.
2024-02-27 17:15:42,998 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 91 [pool-7-thread-1] INFO [] - Beginning to remove expired metrics from the storage.
2024-02-27 17:20:42,998 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 91 [pool-7-thread-1] INFO [] - Beginning to remove expired metrics from the storage.
2024-02-27 17:25:42,998 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 91 [pool-7-thread-1] INFO [] - Beginning to remove expired metrics from the storage.
2024-02-27 17:30:42,998 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 91 [pool-7-thread-1] INFO [] - Beginning to remove expired metrics from the storage.
2024-02-27 17:35:42,998 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 91 [pool-7-thread-1] INFO [] - Beginning to remove expired metrics from the storage.
2024-02-27 17:40:42,998 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 91 [pool-7-thread-1] INFO [] - Beginning to remove expired metrics from the storage.

[wu-sheng]

This means, remote elastic closed the connection. I don't know why. It is most likely your local env issues.