Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compatibility for Elasticsearch 7.1.1 #10003

Open
mvasil15 opened this issue Jun 28, 2019 · 8 comments
Open

Compatibility for Elasticsearch 7.1.1 #10003

mvasil15 opened this issue Jun 28, 2019 · 8 comments

Comments

@mvasil15
Copy link

@mvasil15 mvasil15 commented Jun 28, 2019

Overview of the issue

Me and my team are trying to setup and EFK stack for the monitoring of our application and were trying to put a security up for the Elasticsearch (version 7.1.1). However when attempting to authenticate into the elasticsearch through the spring-data-jest configuration available in jhipster we would get a 401 unauthorized or 403 forbidden despite being sure that the accounts used had the right roles and access. I soon started to suspect an incompatability between ES 7.1.1 and jhipster.

In order to try and isolate the issue i did the following:

I've generated a basic application with the jhipster generator, connected it to an Elasticsearch 7.1.1 and it fails with the following error:

Caused by: com.github.vanroy.springdata.jest.exception.JestElasticsearchException: Cannot execute jest action , response code : 400 , error : {"root_cause":[{"type":"parse_exception","reason":"unknown key [index.refresh_interval] for create index"}],"type":"parse_exception","reason":"unknown key [index.refresh_interval] for create index"} , message : null
at com.github.vanroy.springdata.jest.mapper.DefaultErrorMapper.mapError(DefaultErrorMapper.java:30)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.execute(JestElasticsearchTemplate.java:1126)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.executeWithAcknowledge(JestElasticsearchTemplate.java:1136)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.createIndex(JestElasticsearchTemplate.java:197)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.createIndexWithSettings(JestElasticsearchTemplate.java:1345)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.createIndexIfNotCreated(JestElasticsearchTemplate.java:1330)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.createIndex(JestElasticsearchTemplate.java:178)
at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.createIndex(AbstractElasticsearchRepository.java:96)
at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.<init>(AbstractElasticsearchRepository.java:87)
at org.springframework.data.elasticsearch.repository.support.NumberKeyedRepository.<init>(NumberKeyedRepository.java:36)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:172)
... 46 common frames omitted
Motivation for or Use Case

I'm trying to setup an EFK stack for the monitoring of my application and the elasticsearch version i wanted to use was 7.1.1. As no particular version of elasticsearch in requested whilst generating the app i guessed that it would be compatible with any version.

Reproduce the error

Prerequisites:

  • docker
  • jhipster

Step one: Generate a jhipster application with elasticsearch option with jhipster generate
Step two: Create an elasticsearch, docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.1.1
Step three: ./mvnw

And the bug should appear.

Note that if you change the image tag from 7.1.1 to 6.4.3, the error doesn't appear and elasticsearch works without any issue

Related issues

I did not find any similar issues.

Suggest a Fix

I'm not sure at all, i'm guessing it has to do with the ES version and an incompatability of the spring data jest resource. On my side i tried to create my application's indices manually through ES' rest api in order to see if the application would work. However i noticed that the mapping required to initialize an index is different from ES 6.4.3 to 7.1.1. So even if i do create the index in es 7.1.1 the application then throws a different error:

Caused by: org.springframework.data.elasticsearch.ElasticsearchException: Failed to build mapping for user:user
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.putMapping(JestElasticsearchTemplate.java:265)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.putMapping(JestElasticsearchTemplate.java:277)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.putMapping(JestElasticsearchTemplate.java:237)
at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.putMapping(AbstractElasticsearchRepository.java:100)
at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.<init>(AbstractElasticsearchRepository.java:88)
at org.springframework.data.elasticsearch.repository.support.NumberKeyedRepository.<init>(NumberKeyedRepository.java:36)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:172)
... 46 common frames omitted
Caused by: com.github.vanroy.springdata.jest.exception.JestElasticsearchException: Cannot execute jest action , response code : 400 , error : {"root_cause":[{"type":"illegal_argument_exception","reason":"Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."}],"type":"illegal_argument_exception","reason":"Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."} , message : null
at com.github.vanroy.springdata.jest.mapper.DefaultErrorMapper.mapError(DefaultErrorMapper.java:30)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.execute(JestElasticsearchTemplate.java:1126)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.executeWithAcknowledge(JestElasticsearchTemplate.java:1136)
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.putMapping(JestElasticsearchTemplate.java:262)
... 57 common frames omitted

The error seems to be indicating that the indexes are initialized in a way that is compatible with ES 6.4.3 and not 7.1.1.

JHipster Version(s)
testapp@0.0.0 /Users/MARTIN/workspace/testapp
└── generator-jhipster@6.1.2

JHipster configuration
{
  "generator-jhipster": {
    "promptValues": {
      "packageName": "com.mycompany.myapp",
      "nativeLanguage": "en"
    },
    "jhipsterVersion": "6.1.2",
    "applicationType": "monolith",
    "baseName": "testapp",
    "packageName": "com.mycompany.myapp",
    "packageFolder": "com/mycompany/myapp",
    "serverPort": "8080",
    "authenticationType": "jwt",
    "cacheProvider": "ehcache",
    "enableHibernateCache": true,
    "websocket": false,
    "databaseType": "sql",
    "devDatabaseType": "h2Disk",
    "prodDatabaseType": "mysql",
    "searchEngine": "elasticsearch",
    "messageBroker": false,
    "serviceDiscoveryType": false,
    "buildTool": "maven",
    "enableSwaggerCodegen": false,
    "jwtSecretKey": "NGEyN2VhNDQ4MGY5NDE1ZjViNWVhNjk0ZmQxNzM0NGNiMzg3ODlmNDhiYjMzMmIwYmFjZjQ2MTNiMzZiMGNlYTljYTQ5ZDg5ZWU3MjgzOGJjMzEzNzAyZjRkNTExMmFiMGI3OTNiYWEzMTJlNzRhN2FiNzM0ODNmNjVhMmRlYmQ=",
    "useSass": true,
    "clientPackageManager": "npm",
    "clientFramework": "angularX",
    "clientTheme": "none",
    "clientThemeVariant": "",
    "testFrameworks": [],
    "jhiPrefix": "jhi",
    "entitySuffix": "",
    "dtoSuffix": "DTO",
    "otherModules": [],
    "enableTranslation": true,
    "nativeLanguage": "en",
    "languages": [
      "en"
    ]
  }
Entity configuration(s) entityName.json files generated in the .jhipster directory
{
    "fluentMethods": true,
    "clientRootFolder": "",
    "relationships": [],
    "fields": [
        {
            "fieldName": "verbose",
            "fieldType": "String"
        }
    ],
    "changelogDate": "20190627140754",
    "dto": "no",
    "searchEngine": "elasticsearch",
    "service": "no",
    "entityTableName": "sentencce",
    "databaseType": "sql",
    "jpaMetamodelFiltering": false,
    "pagination": "infinite-scroll"
}
Browsers and Operating System

MacOS Mojave 10.14.5

  • Checking this box is mandatory (this is just to show you read everything)
@pascalgrimaud

This comment has been minimized.

Copy link
Member

@pascalgrimaud pascalgrimaud commented Jun 28, 2019

For now, we use Elasticsearch 6.4.3
See https://github.com/jhipster/generator-jhipster/blob/master/generators/generator-constants.js#L33

We should migrate to Elasticsearch 7+
So thanks for opening this ticket with details. Don't hesitate to PR if you find the fix

@pascalgrimaud

This comment has been minimized.

Copy link
Member

@pascalgrimaud pascalgrimaud commented Jun 28, 2019

Maybe related to VanRoy/spring-data-jest#116

@pascalgrimaud pascalgrimaud changed the title Compatability for Elasticsearch 7.1.1 Compatibility for Elasticsearch 7.1.1 Jun 28, 2019
@jdubois

This comment has been minimized.

Copy link
Member

@jdubois jdubois commented Jul 9, 2019

I'm adding a bug bounty here, this is important and should be upgraded

@DanielFran

This comment has been minimized.

Copy link
Member

@DanielFran DanielFran commented Jul 9, 2019

This is not yet even added in SDE, see spring-projects/spring-data-elasticsearch#284, and will be added only to v4, so it will take some time, problably available only at the end of the year.
Spring-data Moore has been delayed to september because Spring Framework 5.2 has moved its GA date to September 4 and is supporting only 6.7 (https://github.com/spring-projects/spring-data-elasticsearch/blob/master/pom.xml#L22).

@DanielFran

This comment has been minimized.

Copy link
Member

@DanielFran DanielFran commented Jul 10, 2019

Next SDE Moore 3.2.RC2 will now support ES 6.8.1! See spring-projects/spring-data-elasticsearch@a9d5793

@DanielFran

This comment has been minimized.

Copy link
Member

@DanielFran DanielFran commented Jul 18, 2019

@mvasil15 Good news, next SDE version will support elastic stack v.6.8.1 that also include free security support.
See spring-projects/spring-data-elasticsearch#284 (comment)

@rhoiyds

This comment has been minimized.

Copy link

@rhoiyds rhoiyds commented Dec 10, 2019

I am currently experiencing this issue whilst following the JHipster 6 introduction video .
If I generate the JHipster project (selecting Elasticsearch for searching), in the final step of the tutorial when you deploy to Heroku - You receive the error message from the original issue post.
As stated, this seems to be because of a compatibility issue with indexing on Elasticsearch 7.1.1+.

During deployment, Heroku will automatically configure a Bonsai add-on running the latest version of Elasticsearch (7.2.0) - The free tier of Bonsai only supports running the latest version of elasticsearch - You can downgrade to Elasticsearch 6.5.4 (which should fix the error, as suggested in the comments above) but only at a cost ($10USD/month), which isn't great for tinkerers starting out on their first project.

Edit: Also attempted to use SearchBox as an alternative to Bonsai which allows version selection on it's free tier - However, encountering a different problem.

Are there any updates as to when/how this problem can be fixed?

@DanielFran

This comment has been minimized.

Copy link
Member

@DanielFran DanielFran commented Dec 11, 2019

@rhoiyds Actually Spring-boot 2.2 is only managing ES till version 6.8.4 (Spring-data Moore), so there is no way using spring-boot for now to use ES 7+:

Only with Spring-data Neumann version we will have support for ES 7:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.