ES plugins: Specifying types in bulk requests is deprecated.

[abitmore]

User Story

Found in elasticsearch_deprecation.log:

[types removal] Specifying types in bulk requests is deprecated.

More info: https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html

#1997 was an attempt to fix this issue.

Impacts
Describe which portion(s) of BitShares Core may be impacted by your request. Please tick at least one box.

• API (the application programming interface)
• Build (the build process or something prior to compiled code)
• CLI (the command line wallet)
• Deployment (the deployment process after building such as Docker, Travis, etc.)
• DEX (the Decentralized EXchange, market engine, etc.)
• P2P (the peer-to-peer network for transaction/block propagation)
• Performance (system or user efficiency, etc.)
• Protocol (the blockchain logic, consensus, validation, etc.)
• Security (the security of system or user data, etc.)
• UX (the User Experience)
• Other (please add below)

CORE TEAM TASK LIST

• Evaluate / Prioritize Feature Request
• Refine User Stories / Requirements
• Define Test Cases
• Design / Develop Solution
• Perform QA/Testing
• Update Documentation

[abitmore]

#1997 was an attempt to fix this issue.

[abitmore]

This is more complicated than I thought.

The changes are:

• In ES 6.x the type field was needed.
• In ES 7.x (current version) the type field is deprecated, but it is still accepted, with the warning.
• In ES 8.x we will be no longer able to specify a type.

So sooner or later we need to stop specifying types.

Although it is easy to fix the plugin in bitshares-core, the real trouble lies in service maintenance and client application compatibility.

Currently in ES, we use the path pattern bitshares-*/data/2.9.x, where data is the type. If we stop specifying the type, the path pattern will become bitshares-*/_doc/2.9.x. Of course, the existing data will not change by itself.

• ES service maintainers will need to manually delete all existing indexes from their ES databases and regenerate all data with new code, or use a new prefix other than the default one (bitshares-), or install a new ES database. This may take days or weeks to complete. If the old data is not deleted, it uses around 500G of disk space.
  By the way, we will need more disk space to store the new data due to Save operation_result and proposed_ops as objects in ElasticSearch plugin #2380.
• Client libraries and applications will need to query data through the new path pattern and, if applicable, a new prefix.

A side benefit of not specifying a type is that it allows us to customize data mapping, which is required to store arrays using the nested data type. Arrays are stored using the object data type now, which has led to some inconveniences. More info: https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html

@sschiessl-bcp @ety001 what do you think? If it's OK, I can make this change in the next feature release (6.1.0).

[sschiessl-bcp]

ElasticSearch has a license change after 7 (https://techgenix.com/elasticsearchs-licensing-change/), offspring of that is OpenSearch by Amazon:

• https://github.com/opensearch-project/OpenSearch

Are they following the change of type?

[ety001]

I prefer to regenerate all data with new path pattern. Keeping old data is not a good way for developing.

[abitmore]

@sschiessl-bcp according to their blog about license change (https://www.elastic.co/blog/licensing-change) I think we can still use it as is, because we are using the software by our own, but not selling it as a service.

[abitmore]

@lebinbit I guess your applications would be impacted by the change. Please take a look.

[abitmore]

Fixed by #1997.