[Bug] [Skywalking Version 9.1.0] Failed to create / update index template sw_log

[BertOnline]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

OAP server (apache/skywalking)

What happened

An error happened during the OAP server starts;
Error logs:

2022-06-14 11:16:20,540 - com.linecorp.armeria.internal.common.JavaVersionSpecific - 45 [main] INFO  [] - Using the APIs optimized for: Java 9+
2022-06-14 11:16:20,987 - org.apache.skywalking.library.elasticsearch.ElasticSearch - 150 [armeria-common-worker-epoll-2-1] INFO  [] - ElasticSearch version is: ElasticSearch 6.8
2022-06-14 11:16:21,133 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller - 55 [main] INFO  [] - table: log does not exist
2022-06-14 11:16:21,478 - org.apache.skywalking.library.elasticsearch.client.TemplateClient - 139 [armeria-eventloop-epoll-5-2] ERROR [] - Failed to create / update index template sw_log
java.util.concurrent.CompletionException: java.lang.RuntimeException: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"analyzer [oap_analyzer] not found for field [endpoint_name_match]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [type]: analyzer [oap_analyzer] not found for field [endpoint_name_match]","caused_by":{"type":"mapper_parsing_exception","reason":"analyzer [oap_analyzer] not found for field [endpoint_name_match]"}},"status":400}
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:645) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
        at com.linecorp.armeria.common.util.UnmodifiableFuture.doComplete(UnmodifiableFuture.java:124) ~[armeria-1.14.1.jar:?]
        at com.linecorp.armeria.common.util.UnmodifiableFuture.lambda$wrap$0(UnmodifiableFuture.java:100) ~[armeria-1.14.1.jar:?]
        at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
        at com.linecorp.armeria.common.stream.DeferredStreamMessage.lambda$delegate$0(DeferredStreamMessage.java:131) ~[armeria-1.14.1.jar:?]
        at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
        at com.linecorp.armeria.common.stream.StreamMessageCollector.onComplete(StreamMessageCollector.java:67) ~[armeria-1.14.1.jar:?]
        at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:257) ~[armeria-1.14.1.jar:?]
        at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent0(DefaultStreamMessage.java:291) ~[armeria-1.14.1.jar:?]
        at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent(DefaultStreamMessage.java:283) ~[armeria-1.14.1.jar:?]
        at com.linecorp.armeria.common.stream.DefaultStreamMessage.handleCloseEvent(DefaultStreamMessage.java:420) ~[armeria-1.14.1.jar:?]
        at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:363) ~[armeria-1.14.1.jar:?]
        at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber(DefaultStreamMessage.java:319) ~[armeria-1.14.1.jar:?]
        at com.linecorp.armeria.common.stream.DefaultStreamMessage.addObjectOrEvent(DefaultStreamMessage.java:305) ~[armeria-1.14.1.jar:?]
        at com.linecorp.armeria.common.stream.DefaultStreamMessage.close(DefaultStreamMessage.java:427) ~[armeria-1.14.1.jar:?]
        at com.linecorp.armeria.client.HttpResponseDecoder$HttpResponseWrapper.closeAction(HttpResponseDecoder.java:337) ~[armeria-1.14.1.jar:?]
        at com.linecorp.armeria.client.HttpResponseDecoder$HttpResponseWrapper.cancelTimeoutOrLog(HttpResponseDecoder.java:370) [armeria-1.14.1.jar:?]
        at com.linecorp.armeria.client.HttpResponseDecoder$HttpResponseWrapper.close(HttpResponseDecoder.java:320) [armeria-1.14.1.jar:?]
        at com.linecorp.armeria.client.HttpResponseDecoder$HttpResponseWrapper.close(HttpResponseDecoder.java:309) [armeria-1.14.1.jar:?]
        at com.linecorp.armeria.client.Http1ResponseDecoder.channelRead(Http1ResponseDecoder.java:245) [armeria-1.14.1.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152) [netty-handler-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) [netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487) [netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385) [netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) [netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.77.Final.jar:4.1.77.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.RuntimeException: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"analyzer [oap_analyzer] not found for field [endpoint_name_match]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [type]: analyzer [oap_analyzer] not found for field [endpoint_name_match]","caused_by":{"type":"mapper_parsing_exception","reason":"analyzer [oap_analyzer] not found for field [endpoint_name_match]"}},"status":400}
        at org.apache.skywalking.library.elasticsearch.client.TemplateClient.lambda$null$9(TemplateClient.java:135) ~[library-elasticsearch-client-9.1.0.jar:9.1.0]
        at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642) ~[?:?]
        ... 56 more

What you expected to happen

Service starts normally.

How to reproduce

Skywalking Version 9.1.0
Elasticsearch Version 6.8.6

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

I can see demo env and showcase boot successfully. @kezhenxu94 Do you have any clue?

[kezhenxu94]

@BertOnline did you upgrade from an old version or start in a fresh new environment?

[BertOnline]

@kezhenxu94 I upgraded from 8.9.1 to 9.1.0.

[wu-sheng]

This may be related to your old indices and templates? Could you try on a clean env?

[BertOnline]

@wu-sheng In addition, I downloaded tar from the official website, decompressed the tar, modified the elasticsearch configuration, and ran the startup script to upgrade.

[wu-sheng]

Startup script never covers the upgrade. SkyWalking community don't provide such script.

[lesliey]

Anyone has solution? I have the same trouble.
I had clean es indexes as follfows, but it does not work.

DELETE /sw_*
DELETE /.apm-agent-configuration
DELETE /error-2023.04.23
DELETE /error-2023.04.24
DELETE /.apm-custom-link
DELETE /sw_ui_template
DELETE /.async-search