[Feature] Support ElasticSearch 7 as backend storage

[kezhenxu94]

Add storage-elasticsearch7-plugin module

Motivation:

Support ES7 as storage

Modification:

Add storage-elasticsearch7-plugin, make the compilation passed, add maven profiles to distinguish different version of ES

Result:

ES7 storage plugin can co-exist with ES6 storage plugin

Most of the codes come from the existed ES6 plugin, and the credits should belong to @peng-yongsheng

[wu-sheng]

Before all codes related discussion, a question, as this profile based solution, how should developers run the local test for both ES6 and ES7? I mean in the IDE.

[kezhenxu94]

Before all codes related discussion, a question, as this profile based solution, how should developers run the local test for both ES6 and ES7? I mean in the IDE.

Simply switch the profile in maven tab

and I'm not sure if it's possible to achieve the same goal without using profiles, we have to build two distribution package, and have two dependency hierarchies

[wu-sheng]

and I'm not sure if it's possible to achieve the same goal without using profiles, we have to build two distribution package, and have two dependency hierarchies

That is why I proposed two starter module. After you have that, you don't need profile anymore.
From my understanding, our project repo has been too complex already, if need profile to package different dist, it is very hard for a new developer.

[wu-sheng]

Another thing, if you have the exclude and dependency right in new starter and storage, you could use most of existing es6 storage codes, rather than copy them again. It increases the payload to maintain codes. You should just need to implement the ES7 DAO, if ES API changed.

[kezhenxu94]

@wu-sheng Review comments are addressed, please recheck when you have got some time

[wu-sheng]

Two comments. The most concerns are duplicated config files. Suggest to optimize.

[wu-sheng]

@kezhenxu94 You need to sync and resolve the new YAML file. Facing conflicts.

[dmsolr]

Hi @kezhenxu94
I found thread_pool.index.queue_size is deprecated in es7.4.2. Do we need to update the FAQ of the document?

[kezhenxu94]

Hi @kezhenxu94
I found thread_pool.index.queue_size is deprecated in es7.4.2. Do we need to update the FAQ of the document?

The FAQ is for users who are facing the exceptions, if the thread_pool.index.queue_size is not set by default in ES7, which I thing is the case, the users won't reach the FAQ

[wu-sheng]

What we recommend for es7? Is write queue size still there? Add these two in document directly?

[wu-sheng]

@dmsolr I think we also need to update the document, another PR for that?

[dmsolr]

@dmsolr I think we also need to update the document, another PR for that?

Yes, I will do that.

[wu-sheng]

LGTM. @dmsolr @arugal Please recheck. And as @arugal told me, he is very busy these days, we could merge this after @dmsolr finished the local test, and he will test later.

[wu-sheng]

/run agent-plugin-test-4

[wu-sheng]

Merging.