core Event class Java rewrite #4191

colinsurprenant · 2015-11-13T18:49:36Z

Overview

Rewrite the Event class in pure Java and its supporting classes like Timestamp and Accessors and keep 100% compatibility with current implementation of other core logstash components and all plugins.

Purpose

Eventis the main object which encapsulates data and provides and API for the plugins to perform processing on the event content. Having Event implemented in pure Java will improve performance, make possible faster serialization by avoiding costly type conversion between JRuby and Java which in turn will help with an efficient persistence implementation.

Also, these critical and costly operations are related to the Event where the Java implementation should provide performance improvement:

fields reference getters and setters: All Event data manipulation in the config is done using the logstash field reference syntax (ex. [foo][bar][baz]). This is implemented in the Accessors class. an optimized implementation should improve performance.
string interpolation: (Event#sprintf()) The config syntax provides string interpolation in field references. The Ruby implementation was known to be slow and the Java implementation should provides a noticeable improvement. String interpolation is heavily used in all configs.

Strategy

Extract the current Event implementation from logstash-core and make it a proper self-contained gem which should be drop-in replaceable with an alternate implementation. This is completed in logstash-core, logstash-core-event, logstash-core-event-java #4123 and the gem is logstash-core-event.
Create the Java implementation gem logstash-core-event-java. This is in progress in logstash-core, logstash-core-event, logstash-core-event-java #4123.
provide benchmarks
have all core specs pass
have all core plugins spec pass
have all default plugins spec pass

The text was updated successfully, but these errors were encountered:

wip initial Accessors, Event, EventImpl, Path, Timestamp and PathTest wip wip FieldReference and Accessors implementation rename targetCache to lut and set it protected initial Accessors tests todo comment more tests Timestamp implementation Timestamp tests fix method name add Long constructor event initialization, timestamp handling and json serialization add <> type information custom json serializer for Timestamp remove toJson test initial Event test more tests comments debug traces initial jruby Event wrapper and specs added PathCache implemented includes added clone wrap all Event methods Rakefile to build and jar missing getters and implement overwrite support Date conversion proper cast and coercion replace Ruby Event with Java Event test for field reference setter type coercion disable specs timestap setter should also set in map, accept more timestamp types pre cache timestamp and expose isTimestamp constructor from DateTime expose proper Ruby Timestamp object Ruby Timestamp basic specs also load JRuby Timestamp transpose Java<->Ruby Timestamp fix timestamp specs new jar cleanup object construction

This implementation is different from the master version since instead of doing JIT Template compilation on every events we create a data structure of a compiled templates. This template is cached in a concurrent hashmap and used on other requests. fixing conflict

First the case with single fieldref Add support for +%s => epoch Small comments concerning the caching mechanism and the memory usage Make the #sprintf method work with Array and Hash

event clone append in Ruby tag & append add timestamp assignment spec comment out original classes fix at signature implemented del remove from array del test

more event java updates - updated metadata - moved #append to java - other small test fixes commit gradle wrapper jar

…re-event-java impl, relates to elastic#4191 fixed timezone issue extracted logstash-core and reorganized specs extracted logstash-core-event extract java Event into logstash-core-event-java in a proper gem remove obsolete jruby_event bootstrapping fix require path add java code bootstrap use logstash-core-event/logstash-core-event.rb remove obsolete files basic instructions LogStash::Json need to be initialized from event update jruby and gradle versions update compile:logstash-core-event-java rake task WIP tasks refactor fix gem.files skip test if class is not defined fix gem related tasks for new structure add gem spec dirs in core tests bootstrap java implementation when requiring timestamp new Cloner class and Event clone impl fix array fields assignments, see elastic#4140 don't rely on json implementation ordering fix skipped last iterpolation char remove implementation specific unnecessary check also require ruby classes define error class in ruby raise exception on invalid format remove implementation specific tests and extract and put logger related test in pending missing bits for having all core timestamp specs pass run all core specs remove leftover comment regex missing encoding header revert to logstash-core-event by default finished proper gemification useless require dynamically pick specs depending on logstash-core-event-* implementation logstash root package version missing file for proper gemification do not build java event by default always check for root logstash lib dir fix concurrent-ruby version confict fix rebase conflict re-enable specs user vars instead of constants move non core code in bootstrap document version files move version file remove useless code use version in logstash-core fix gem files list put back concurrent-ruby version constrain as in master add dependency on logstash-core-event remove dependency on logstash-core to avoid circular dependency fix rebase conflict remove circular dependency fix specs update README

wip initial Accessors, Event, EventImpl, Path, Timestamp and PathTest wip wip FieldReference and Accessors implementation rename targetCache to lut and set it protected initial Accessors tests todo comment more tests Timestamp implementation Timestamp tests fix method name add Long constructor event initialization, timestamp handling and json serialization add <> type information custom json serializer for Timestamp remove toJson test initial Event test more tests comments debug traces initial jruby Event wrapper and specs added PathCache implemented includes added clone wrap all Event methods Rakefile to build and jar missing getters and implement overwrite support Date conversion proper cast and coercion replace Ruby Event with Java Event test for field reference setter type coercion disable specs timestap setter should also set in map, accept more timestamp types pre cache timestamp and expose isTimestamp constructor from DateTime expose proper Ruby Timestamp object Ruby Timestamp basic specs also load JRuby Timestamp transpose Java<->Ruby Timestamp fix timestamp specs new jar cleanup object construction

This implementation is different from the master version since instead of doing JIT Template compilation on every events we create a data structure of a compiled templates. This template is cached in a concurrent hashmap and used on other requests. fixing conflict

First the case with single fieldref Add support for +%s => epoch Small comments concerning the caching mechanism and the memory usage Make the #sprintf method work with Array and Hash

event clone append in Ruby tag & append add timestamp assignment spec comment out original classes fix at signature implemented del remove from array del test

more event java updates - updated metadata - moved #append to java - other small test fixes commit gradle wrapper jar

…re-event-java impl, relates to elastic#4191 fixed timezone issue extracted logstash-core and reorganized specs extracted logstash-core-event extract java Event into logstash-core-event-java in a proper gem remove obsolete jruby_event bootstrapping fix require path add java code bootstrap use logstash-core-event/logstash-core-event.rb remove obsolete files basic instructions LogStash::Json need to be initialized from event update jruby and gradle versions update compile:logstash-core-event-java rake task WIP tasks refactor fix gem.files skip test if class is not defined fix gem related tasks for new structure add gem spec dirs in core tests bootstrap java implementation when requiring timestamp new Cloner class and Event clone impl fix array fields assignments, see elastic#4140 don't rely on json implementation ordering fix skipped last iterpolation char remove implementation specific unnecessary check also require ruby classes define error class in ruby raise exception on invalid format remove implementation specific tests and extract and put logger related test in pending missing bits for having all core timestamp specs pass run all core specs remove leftover comment regex missing encoding header revert to logstash-core-event by default finished proper gemification useless require dynamically pick specs depending on logstash-core-event-* implementation logstash root package version missing file for proper gemification do not build java event by default always check for root logstash lib dir fix concurrent-ruby version confict fix rebase conflict re-enable specs user vars instead of constants move non core code in bootstrap document version files move version file remove useless code use version in logstash-core fix gem files list put back concurrent-ruby version constrain as in master add dependency on logstash-core-event remove dependency on logstash-core to avoid circular dependency fix rebase conflict remove circular dependency fix specs update README

colinsurprenant · 2015-12-09T16:13:38Z

All default plugins are now passing specs, all related PRs are tracked in #4264. Merge fest can now start.

colinsurprenant · 2015-12-15T22:12:53Z

quick benchmark update, I tested the config defined in https://gist.github.com/colinsurprenant/8143a0dfd30830d57b03 using 4 workers -w 4 and default batch size of 125 (not specifying -b) on master pre-g-pipeline, ng-pipeline and ng-pipeline+java-event:

TPS	pipeline
4.86k	pre ng-pipeline
6.29k	ng-pipeline
7.50k	ng-pipeline+java-event

untergeek · 2015-12-15T23:33:33Z

That's an impressive increase!

…re-event-java impl, relates to elastic#4191 fixed timezone issue extracted logstash-core and reorganized specs extracted logstash-core-event extract java Event into logstash-core-event-java in a proper gem remove obsolete jruby_event bootstrapping fix require path add java code bootstrap use logstash-core-event/logstash-core-event.rb remove obsolete files basic instructions LogStash::Json need to be initialized from event update jruby and gradle versions update compile:logstash-core-event-java rake task WIP tasks refactor fix gem.files skip test if class is not defined fix gem related tasks for new structure add gem spec dirs in core tests bootstrap java implementation when requiring timestamp new Cloner class and Event clone impl fix array fields assignments, see elastic#4140 don't rely on json implementation ordering fix skipped last iterpolation char remove implementation specific unnecessary check also require ruby classes define error class in ruby raise exception on invalid format remove implementation specific tests and extract and put logger related test in pending missing bits for having all core timestamp specs pass run all core specs remove leftover comment regex missing encoding header revert to logstash-core-event by default finished proper gemification useless require dynamically pick specs depending on logstash-core-event-* implementation logstash root package version missing file for proper gemification do not build java event by default always check for root logstash lib dir fix concurrent-ruby version confict fix rebase conflict re-enable specs user vars instead of constants move non core code in bootstrap document version files move version file remove useless code use version in logstash-core fix gem files list put back concurrent-ruby version constrain as in master add dependency on logstash-core-event remove dependency on logstash-core to avoid circular dependency fix rebase conflict remove circular dependency fix specs update README

colinsurprenant · 2016-01-13T19:01:58Z

I am calling this issue closed, Java Event is merged and all core and plugins specs are passing. I will followup with a post-merge review process issue and we can create new issues as needed.

colinsurprenant added discuss v2.2.0 java event labels Nov 13, 2015

colinsurprenant self-assigned this Nov 13, 2015

suyograo added the performance improvements label Nov 15, 2015

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

Timestamp.java reformatting, relates to elastic#4191

a8185a2

colinsurprenant pushed a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

refactor to gradle project, relates to elastic#4191

e28f188

more event java updates - updated metadata - moved #append to java - other small test fixes commit gradle wrapper jar

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

Timestamp.java reformatting, relates to elastic#4191

2ce84bb

colinsurprenant pushed a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

refactor to gradle project, relates to elastic#4191

e8880d0

more event java updates - updated metadata - moved #append to java - other small test fixes commit gradle wrapper jar

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

updated lock file, relates to elastic#4191

f2077ec

colinsurprenant mentioned this issue Nov 16, 2015

new lock file for #4191 #4201

Merged

This was referenced Dec 9, 2015

fix string interpolation cache api and related specs #4325

Closed

new Event API for internal data hash mutation #4326

Closed

using symbols in Event data does not make sense #4341

Closed

suyograo added v2.3.0 and removed v2.2.0 labels Dec 16, 2015

colinsurprenant mentioned this issue Jan 11, 2016

remove implementation specific timestamp check logstash-plugins/logstash-input-graphite#10

Merged

colinsurprenant closed this as completed Jan 13, 2016

colinsurprenant mentioned this issue Jan 13, 2016

[meta] Java Event post-merge review process #4478

Closed

suyograo mentioned this issue Jan 18, 2016

Features for v2.3 #4515

Closed

2 tasks

suyograo mentioned this issue Mar 6, 2016

Pipeline changes for resiliency, phase1 #4767

Closed

11 tasks

dedemorton mentioned this issue Apr 1, 2016

Add a note concerning the ruby filter for the change in the java event #4974

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core Event class Java rewrite #4191

core Event class Java rewrite #4191

colinsurprenant commented Nov 13, 2015

colinsurprenant commented Dec 9, 2015

colinsurprenant commented Dec 15, 2015

untergeek commented Dec 15, 2015

colinsurprenant commented Jan 13, 2016

core Event class Java rewrite #4191

core Event class Java rewrite #4191

Comments

colinsurprenant commented Nov 13, 2015

Overview

Purpose

Strategy

colinsurprenant commented Dec 9, 2015

colinsurprenant commented Dec 15, 2015

untergeek commented Dec 15, 2015

colinsurprenant commented Jan 13, 2016