New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to elasticsearch 6.x compatibility #5020

Merged
merged 11 commits into from Sep 4, 2018

Conversation

Projects
None yet
2 participants
@kroepke
Member

kroepke commented Aug 24, 2018

Primarily adapt index mapping template changes for 6.x backends.
Updated jest+jackson dependency for minor URL compatibility with 6.x servers.

Add elasticsearch request logger for debugging request/response layer: enable with TRACE on org.graylog2.indexer.cluster.jest.RequestResponseLogger

fixes #4957

kroepke added some commits Jul 31, 2018

create field with random geo location
location is based on bounding boxes of the world's countries to get a bit more realistic looking data
otherwise a third is going to be in the pacific...
index mapping tweaks for es 6
add request logging to elasticsearch client for easier debugging
add tools, and http/jq commands to verify state
simplify configuration files
@kroepke

This comment has been minimized.

Member

kroepke commented Aug 24, 2018

This also enhances the random http input to provide geolocation information for easier testing of map features.

@kroepke kroepke requested a review from bernd Aug 24, 2018

@bernd bernd self-assigned this Aug 24, 2018

final Map<String, Object> settings = ImmutableMap.of("analysis", analysis);
// mapping types are deprecated, 7.0 and later will only allow `_doc` here, so we start using that
// see https://www.elastic.co/guide/en/elasticsearch/reference/6.3/removal-of-types.html#_schedule_for_removal_of_mapping_types
final Map<String, Object> mappings = ImmutableMap.of(IndexMapping.TYPE_MESSAGE, messageMapping(analyzer));

This comment has been minimized.

@bernd

bernd Aug 24, 2018

Member

Is the comment or the code wrong here? The comment says that we are using _doc but the code is actually still using message.

@bernd

This comment has been minimized.

Member

bernd commented Aug 24, 2018

@kroepke Please add the missing license headers.

@bernd

This comment has been minimized.

Member

bernd commented Aug 24, 2018

I found this log message:

[2018-08-24T18:54:25,832][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]

Are we ignoring this warning for now?

@kroepke

This comment has been minimized.

Member

kroepke commented Aug 28, 2018

Yes, that deprecation warning we'll likely ignore for now because we'll fix it in 3.0 anyway.

kroepke added some commits Aug 28, 2018

@bernd bernd added this to the 2.5.0 milestone Aug 29, 2018

@bernd

bernd approved these changes Sep 4, 2018

I did some basic testing with ES 2.4, ES 5.6 and ES 6.4.

  • Message ingest
  • Using some widgets
  • Manual index rotation
  • Archiving

@bernd bernd merged commit 5f00f3d into 2.5 Sep 4, 2018

2 of 3 checks passed

ci-web-linter Jenkins build graylog-pr-linter-check 2703 has failed
Details
graylog-project/pr Jenkins build graylog-project-pr-snapshot 1786 has succeeded
Details
license/cla Contributor License Agreement is signed.
Details

@bernd bernd deleted the es6-mapping branch Sep 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment