Rebuilt ElasticSearch client on top of their REST API

[kezhenxu94]

• If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes Implement SkyWalking's native ElasticSearch protocol client for ElasticSearch 6/7/8 and OpenSearch 1.x #7471 , closes [Bug] ElasticSearch storage secretsManagementFile hot reloading doesn't work as expected #7655.
• Update the CHANGES log.

[wu-sheng]

LGTM. Wait for others' review.

[JaredTan95]

I tested locally, all looks great to me except EmptyEndpointGroupException as follows when es is shutdown and connected failures:

2021-09-06 11:14:10,020 org.apache.skywalking.library.elasticsearch.ElasticSearch 149 [armeria-eventloop-nio-4-2] DEBUG [] - [creqId=3fe1891d, preqId=8179d41f][http://UNKNOWN/#GET] Request: {startTime=2021-09-06T03:14:10.010Z(1630898050010987), length=0B, duration=4538µs(4538520ns), cause=com.linecorp.armeria.client.UnprocessedRequestException: com.linecorp.armeria.client.endpoint.EmptyEndpointGroupException, scheme=none+http, name=GET, headers=[:method=GET, :path=/, :scheme=http, :authority=UNKNOWN]}
2021-09-06 11:14:10,021 org.apache.skywalking.library.elasticsearch.ElasticSearch 186 [armeria-eventloop-nio-4-2] WARN  [] - [creqId=3fe1891d, preqId=8179d41f][http://UNKNOWN/#GET] Response: {startTime=2021-09-06T03:14:10.020Z(1630898050020718), length=0B, duration=0ns, totalDuration=9749µs(9749393ns), cause=com.linecorp.armeria.client.UnprocessedRequestException: com.linecorp.armeria.client.endpoint.EmptyEndpointGroupException, headers=[:status=0]}
com.linecorp.armeria.client.UnprocessedRequestException: com.linecorp.armeria.client.endpoint.EmptyEndpointGroupException

I think EmptyEndpointGroupException is not enough clear to point out the cause of the error?

[kezhenxu94]

I tested locally, all looks great to me except EmptyEndpointGroupException as follows when es is shutdown and connected failures:

2021-09-06 11:14:10,020 org.apache.skywalking.library.elasticsearch.ElasticSearch 149 [armeria-eventloop-nio-4-2] DEBUG [] - [creqId=3fe1891d, preqId=8179d41f][http://UNKNOWN/#GET] Request: {startTime=2021-09-06T03:14:10.010Z(1630898050010987), length=0B, duration=4538µs(4538520ns), cause=com.linecorp.armeria.client.UnprocessedRequestException: com.linecorp.armeria.client.endpoint.EmptyEndpointGroupException, scheme=none+http, name=GET, headers=[:method=GET, :path=/, :scheme=http, :authority=UNKNOWN]}
2021-09-06 11:14:10,021 org.apache.skywalking.library.elasticsearch.ElasticSearch 186 [armeria-eventloop-nio-4-2] WARN  [] - [creqId=3fe1891d, preqId=8179d41f][http://UNKNOWN/#GET] Response: {startTime=2021-09-06T03:14:10.020Z(1630898050020718), length=0B, duration=0ns, totalDuration=9749µs(9749393ns), cause=com.linecorp.armeria.client.UnprocessedRequestException: com.linecorp.armeria.client.endpoint.EmptyEndpointGroupException, headers=[:status=0]}
com.linecorp.armeria.client.UnprocessedRequestException: com.linecorp.armeria.client.endpoint.EmptyEndpointGroupException

I think EmptyEndpointGroupException is not enough clear to point out the cause of the error?

There is warning / error message before what you posted, also, it has health check registered .

2021-09-06 12:52:21,329 org.apache.skywalking.oap.server.telemetry.api.HealthCheckMetrics 50 [armeria-common-worker-nio-2-1] WARN  [] - Health check fails. reason: No healthy endpoint
2021-09-06 12:52:21,330 org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient 146 [armeria-common-worker-nio-2-1] ERROR [] - Failed to recreate ElasticSearch client based on new config

[JaredTan95]

LGTM

[kezhenxu94]

Thank you @wu-sheng @JaredTan95 for reviewing and checking, I'm going to merge this

[wu-sheng]

@wankai123 is confirming too.

[hanahmily]

docker/oap/Dockerfile.oap will be fixed in follow-up PRs?

[hanahmily]

docker/oap/Dockerfile.oap will be fixed in follow-up PRs?

ignore this

[hanahmily]

Some nits:

• remove https://github.com/apache/skywalking/blob/master/docker/.env#L1
• update https://github.com/apache/skywalking/blob/master/docker/.env#L2 to ES7

[hanahmily]

LGTM

some nits regarding docker/.env

[wankai123]

LGTM