Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

[PIO-209] Upgrade Elasticsearch to 6.8 for pre-built binary distribution #516

Merged
merged 14 commits into from Nov 5, 2019

Conversation

shimamoto
Copy link
Member

@shimamoto shimamoto commented Jun 13, 2019

Most users use pre-built binary distribution and would like to run on new versions as much as possible.

before:
Spark 2.1.3
Elasticsearch 5.6.9
Hbase 1.2.6

after:
Spark 2.4.3 ---> separate a PR
Elasticsearch 6.8.1
Hbase 1.4.10 ---> separate a PR

The dependencies are complex, so I will try to handle tasks separately.

This PR upgrade Elasticsearch to 6.8 and deprecate Elasticsearch 5 because of EOL (https://www.elastic.co/support/eol).

@shimamoto shimamoto changed the title [PIO-209] Upgrade Spark to 2.4 for pre-built binary distribution [PIO-209] Upgrade Elasticsearch to 6.8 for pre-built binary distribution Jun 27, 2019
@shimamoto
Copy link
Member Author

Currently, in the develop branch (Elasticsearch 6.4.2), there are no HEAD request warnings.

[INFO] basic_app_usecases 2019-06-04 02:28:26,772: Adding a new application
[WARN] [RestClient] request [PUT http://elasticsearch:9200/pio_meta_apps] returned 1 warnings: [299 Elasticsearch-6.4.2-04711c2 "the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template" "Tue, 04 Jun 2019 02:28:29 GMT"]
[WARN] [RestClient] request [PUT http://elasticsearch:9200/pio_event_1] returned 1 warnings: [299 Elasticsearch-6.4.2-04711c2 "the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template" "Tue, 04 Jun 2019 02:28:30 GMT"]
[INFO] [App$] Initialized Event Store for this app ID: 1.
[INFO] [Pio$] Created a new app:
[INFO] [Pio$]       Name: MyRecommender
[INFO] [Pio$]         ID: 1
[INFO] [Pio$] Access Key: xPsroclOrmAViB0OA7Q8EsYX_uRNOPvPLLGMEL6coSmN8hFA-F4ouGwETCxQvF1p

When upgrading Elasticsearch to 6.8.1, I have run into the following warnings:

pio@b507ffc58515:/PredictionIO-0.15.0-SNAPSHOT/MyRecommendation$ $PIO_HOME/bin/pio app new MyApp1
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_sequences] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_sequences/_mapping/sequences] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_apps] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_apps/_mapping/apps] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_event_1] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [PUT http://localhost:9200/pio_event_1] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template"]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_event_1/_mapping/1] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [PUT http://localhost:9200/pio_event_1/_mapping/1] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] Specifying types in put mapping requests is deprecated. To be compatible with 7.0, the mapping definition should not be nested under the type name, and the parameter include_type_name must be provided and set to false."]
[INFO] [App$] Initialized Event Store for this app ID: 1.
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_accesskeys] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_accesskeys/_mapping/accesskeys] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[INFO] [Pio$] Created a new app:
[INFO] [Pio$]       Name: MyApp1
[INFO] [Pio$]         ID: 1
[INFO] [Pio$] Access Key: pYqtHCiLzYwTyiodCAFC_cekeAfy-k8l0Fta4yWWvC42-dEB9piso5rPnzXCsP90

pio@b507ffc58515:/PredictionIO-0.15.0-SNAPSHOT/MyRecommendation$ $PIO_HOME/bin/pio app list  
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_sequences] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_sequences/_mapping/sequences] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_apps] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_apps/_mapping/apps] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_accesskeys] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_accesskeys/_mapping/accesskeys] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[INFO] [Pio$]                 Name |   ID |                                                       Access Key | Allowed Event(s)
[INFO] [Pio$]               MyApp1 |    1 | pYqtHCiLzYwTyiodCAFC_cekeAfy-k8l0Fta4yWWvC42-dEB9piso5rPnzXCsP90 | (all)
[INFO] [Pio$] Finished listing 1 app(s).
pio@b507ffc58515:/PredictionIO-0.15.0-SNAPSHOT/MyRecommendation$ 

That's why tests fail. When executing pio commands, the HEAD request warnings will always be visible to users.

@marevol @dszeto How should we deal with that?

Copy link
Member

@Wei-1 Wei-1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Elasticsearch 6.x

In 6.8, Not setting include_type_name will result in a deprecation warning.

Elasticsearch 7.x

The include_type_name parameter in the index creation, index template, and mapping APIs will default to false. Setting the parameter at all will result in a deprecation warning.

Elasticsearch 8.x

The include_type_name parameter is removed.


Given the context on the Elastic Search Website
I think it makes sense that we bypass this warning.


I believe we should modify the show function in utils
Since using the number of lines to determine whether it is an id, name, or description might be too risky, as we might face similar issues in the future.

@shimamoto
Copy link
Member Author

@Wei-1 Thanks! This is good info.

I would like to hear other committer's judgement on this, and work out details.

@shimamoto
Copy link
Member Author

The Elasticsearch team has been extremely aggressive with breaking changes continuously.

@shimamoto
Copy link
Member Author

We have to remove 5.6 support before supporting 7.0 because method performRequest(String method, String endpoint) is removed as of ES 7.0.

@shimamoto
Copy link
Member Author

@dszeto @marevol @takezoe please review and let me know for any suggestions. Thanks

Unless I hear from you to the contrary, I will merge at the end of the month.

Copy link
Member

@takezoe takezoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks very good to me. Great work!

@morrismatts
Copy link

Does this PR need anything more? Or is it ready to merge?

Thanks! 👏👏👏

@EmergentOrder EmergentOrder self-requested a review November 5, 2019 15:19
Copy link
Member

@EmergentOrder EmergentOrder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@EmergentOrder EmergentOrder merged commit 9ed7d84 into apache:develop Nov 5, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
5 participants