DATAES-285 - Support for Elasticsearch 5.0

[withccm]

Hello. I open a new PR by reformatting. (Previous PR)
There are related issue that support for Elasticsearch 5.x.

No suggest support.

No suggest facet.
Facets have been replaced by aggregations in Elasticsearch 1.0, which
are a superset of facets.

SoredFields is not working in Elasticsearch 5.x,
I guess it's an Elasticsearch bug.

The string field type will continue to work during the 5.x series.

• Filed.String -> Filed.Text, Filed.Keyword

Painless, Groovy Plugins not added.

I will add the missing specification.

• You have read the Spring Data contribution guidelines.
• There is a ticket in the bug tracker for the project in our JIRA.
• You use the code formatters provided here and have them applied to your changes. Don’t submit any formatting related changes.
• You submit test cases (unit or integration tests) that back your changes.
• You added yourself as author in the headers of the classes you touched. Amend the date range in the Apache license header if needed. For new types, add the license header (copy from another file and set the current year only).

[withccm]

I've looked into creating a snapshots-repository ...
@otrosien
How can I build off a snapshot? I want you to guide me.

[otrosien]

That comment was into the direction of the project admins (@mp911de @olivergierke @akonczak). To my knowledge all what's needed is a copy of your branch in the main repository and name it "issue/DATAES-282". With this convention the CI is instructed to publish an artifact off that branch.

[withccm]

I want to copy my branch to the main repository. But I can't push the branch named "issue/DATES-285" of the main repository. Or, if "issue/DATES-285" branch is created, I want to pull request to the branch. What should I do?

[awanczowski]

Hello All,

We are looking to use Spring Data Elasticsearch with our project. However, we would like to use ES 5.1.x. Is there an ETA for support of Elasticsearch 5?

Thanks
Drew

[odrotbohm]

We're pretty late in the Ingalls release train (GA scheduled for this week), which makes it very unlikely this is going to make it in. We usually ship these kinds of changes at least in an RC. Also, this is a community project and not driven by the Spring Data team, so we're relying very much on the assessment of the actual maintainers.

Beyond that, the PR looks like it'll need quite some work. A lot of unrelated changes, changes that leak into the APIs, tests that have changed. All that needs to be taken care of, isn't done in a day and needs at least proper explanation in the commit message.

We might be able to consider this for Kay (the 2.0 release) but then again that totally depends on how much time the actual maintainers can invest.

[awanczowski]

@olivergierke thanks for the information around the release cycle. I will keep an eye on things for updates.

[smartlv]

Hello All,

We are looking to use Spring Data Elasticsearch with our project. However, we would like to use ES 5.1.x.
help！help！ es server current is 5.1.2

Thanks

[z0mb1ek]

@olivergierke why is it not driven by the Spring Data team?

[odrotbohm]

Because we have limited resources and are kept busy with all the other stuff we work on.

[z0mb1ek]

@olivergierke is ES not so interesting like others? strange choice

[schleifer-john]

@olivergierke who are the "actual" maintainers of this project? Excluding README changes there hasn't been an approved Pull Request since August 2016 (5+ months ago). There is even a Pull Request for a simple README URL fix (#168) that seems like a no-brainer to merge. The lack of response and activity from the "actual" maintainers is very frustrating and discourages people from making contributions to this project.

[mp911de]

@schleifer-john the maintainers are @mohsinh and @akonczak.

[otrosien]

Indeed, current project status is sad. I really hope pivotal would put some resources on this project. To @mohsinh and @akonczak: could you add some comments on your view of the situation?

[mohsinh]

Hello, starting with project status, we are not at that sad level. every major release of elasticsearch causes major api changes in library and 5.x is mojor to all of the releases so far.

Given that we will look into upgrading elasticsearch to latest version and as @olivergierke suggested it will be released with Kay if we will be able to merge changes before RC1 which is in mid March.

This pull request still require major week or two of a work, its not straight forward merge. We are independent resource(s) willing to contribute on this project wherever we can, anyone who is willing to do the same is more than welcome to contribute.

We will keep posting update from our side about upgrade on the same thread.

[mp911de]

@mohsinh We discussed with @olivergierke and @christophstrobl the dependency to the Transport API which causes some work on each upgrade. We were also wondering whether it could make sense switching to the HTTP API which seems more stable throughout the versions.

[garpinc]

Also have you given any thought to search-guard (https://github.com/floragunncom/search-guard ) support from spring-data?

[smartlv]

I need 5.0 es,,what can I do?

[miklosbarabas]

How one can join to help you guys progress on this?

[akonczak]

Hi @olivergierke, @christophstrobl, @mp911de,
Each solution has a pros and cons.
Transport client - is the easiest/fastest option but we need update it with each release of ES.
Rest Client - we need to build entire model and this will take a time.

For now we are working on the easiest option to modify module to use Transport client but our longer term goal is to migrate to Rest Client.

Artur

[judeys]

@mp911de agree

[dennisfoconnor]

@akonczak I noticed that you've been doing work on a 5.0.x-prep branch. Is that work meant to compliment this PR or replace it?

[frekele]

I would like to use Data Spring with Elasticsearch 5.2.2, but it looks like it will take a while, this version 5.x.
An alternative that i see, is use hibernate-search 5.7 that already has support for elasticsearch 2.x;
And hibernate-search 5.8 is what's coming out of the oven with support for Elasticsearch 5.x or higher.

See this post:
http://in.relation.to/2017/02/22/hibernate-search-5-7-0-Final/
Jira roadmap.
https://hibernate.atlassian.net/projects/HSEARCH/versions/26101/tab/release-report-done

But I would still like to use spring-data 😢

[mohsinh]

Hello, as many of you have already noticed that we are working in the branch for upgrade to 5.x.

We now have compiled and all test working version of library in the branch. only exception or blocker we have is NodeClient(in-memory client) is not support anymore, so we are currently running all the integration test on local transport client. We are investigating possibility to use integration tests from elasticsearch at the moment.

Meanwhile @olivergierke is Spring CI supports any virtual environment like docker where we can start elasticsearch and build the project ? just a thought.

[mp911de]

That's great news. Please make sure to pick up the Java 8 changes we pushed yesterday (it's on master right now). @trevormarshall is the right person to ask about Bamboo. We use additionally TravisCI in several projects for integration testing as we don't rely on a 3rd party with Travis.

[flefebure]

Hi,

We at www.softbridge.fr are dependant of DATAES (and so DATAMongo and DATAJPA)
We maintain a catch-all branch "performance-features" in https://github.com/flefebure/spring-data-elasticsearch where we've implemented our needs.

I tried some PR in the past but without any success so I gave up.

In this catch-all branch, we can find useful (at least for us) features like :

• response nested-hits mapping support
• timeout support on bulk requests
• node preference for paged queries (avoid duplicate or missing result for default _doc sorting in clustered env)
• from/size support (sometime the SD Paged responses does not fit. eg Vaadin lazy loading in table/grid)
• SPEL in @setting path
• bulk delete support
• indices refreshing after bulks is optional
• and a poorly implemented (but useful) index partitioning feature (a try to mimic SGBD partitioning features)

Now we have to migrate to ES5 and it's a bunch of work.
I started to work on it in my "5.x-partitioner" branch

The first problems I've seen :

• not a bug : missing Field(type=keyword) support

• one tricky : With ES5, in case of Parent/Child mappings, you have to declare the mappings at index creation time (if you try later you fail on a "can't add a _parent field that points to an already existing type, that isn't already a parent" exception)
  this is inconsistent with the DATAES indexCreation and putMapping workflow.
  Actually, I'm enriching the @document interface with a field "Class[] mappingAtIndexCreation"
  Some I can enrich the index creation query with some mappings

I will inform of other problems I see along the way

I would be happy to contribute to the version but I have the feeling that most of the PR are ignored so I don't want to waste my time in useless code repackaging

Mohsinh, for the integration test embedded server problem, may be you saw that Hadoop people still continue to use embedded server with the 5.x (https://github.com/elastic/elasticsearch-hadoop/blob/fefcf8b191d287aca93a04144c67b803c6c81db5/mr/src/itest/java/org/elasticsearch/hadoop/EsEmbeddedServer.java)
But it is not encouraged by ES folks...

Franck

[cemo]

@mohsinh gentle ping. What is the status of your branch?

[mrkpatchaa]

Hi all, any update on this ?

[mohsinh]

Hello, thanks to @akonczak we have working branch (compile & test) for elasticsearch 5.x

Also thanks @flefebure for pointing to embedded server in es-hadoop.

over few days we will polish/refactor some minor code and soon its going to be merged with master.

[mohsinh]

changes are merged to master, closing this thread.

[maxtuzz]

@mohsinh When do you plan to do a maven artifact release? Cheers

[odrotbohm]

@maxtuzz — You can find the general release plan in our wiki. The GA release (i.e. the first artifact version that appears on Maven central) is scheduled for mid July. However, we encourage users to try the milestone builds available from https://repo.spring.io/libs-milestone. In case of Kay M4, they will be available roundabout mid next week.

[maxtuzz]

Thank you very much @olivergierke I'll give M4 a go when it's available.

[mbedna]

Is it possible to use M4 with Spring Boot? Currently I get java.lang.AbstractMethodError probably because of spring-data-commons version difference.

[odrotbohm]

Please make sure you use the Spring Data version that ships with Spring Boot and not accidentally have customized it to a different one. If that still fails, please open a ticket in Boot containing more information (stack trace, sample etc.)

[mbedna]

Ok, Thanks for info.

[tsachev]

@mbedna you will need to use spring boot 2 milestonse (scheduled for november)or disable all elastic auto configurations.

[mbedna]

@tsachev I didn't think about disabling auto configuration, Thank you for sharing this idea 👍

[odrotbohm]

Just to clarify: there is a working milestone of Spring Boot that works OOTB with the current Spring Data milestone and thus ES 5. Boot 2.0 GA is scheduled for late this year, correct.

[maxtuzz]

@olivergierke @mohsinh I've been working a lot on upgrading my project to support Elasticsearch 5.5.0 and Spring Boot M4. I've noticed a few problems though, they seem to be more related to spring data than anything else:

1. Here is a snippet on my Page metadata when doing a search with the API I'm writing with Pageable (page=0, size=1, and sort=name,asc):

  "last": true,
  "numberOfElements": 1,
  "totalElements": 46,
  "totalPages": 1,
  "sort": {
    "sorted": false,
    "unsorted": true
  },
  "first": true,

As you can see, Sort will always be unsorted regardless of what value I enter. Total pages will always be 1 regardless of if total elements are greater than 1 or not.

I can request with (size=46) and get all 46 results fine though, however, it looks like the org.springframework.data.domain.Page `hasNext()' method will always return false with Page objects fetched from my Elasticsearch queries. This is a problem as I am using client uses hypermedia links to next element in order to achieve infinite scroll.

2. I've also found that anything implementing the Page interface will get an IndexOutOfBounds exception during serialization (possible due to override of the required map method). I had to remove the implementation of Page in my custom PagedResources class due to this to avoid the problem. I can fetch more info on this if you're interested.

3. When using the Spring Hateoas implementation of PagedResources, I get similar problems due to errors within how my Pages are acting.

4. When using AutoConfiguration, the fact that I have spring.data.* referenced in properties seems to require the use of a datasource and data driver regardless of whether I am using Spring Data Elasticsearch or not. I had to add a HSQL dependency to my classpath to avoid this error.

Wasn't really sure where the best forum to post this was, however since it seems to all come from the Spring Data Elasticsearch upgrade path I thought here was alright.

Any advice on where to look at this further? I'd love to be able to help out with a PR
Will fork and take a look.

[eacdy]

same problem. @maxtuzz Have you solved it?
the totalPages returns always 1 .
the version is

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
    <version>2.0.0.M5</version>
</dependency>