New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core Event class Java rewrite #4191

Closed
colinsurprenant opened this Issue Nov 13, 2015 · 4 comments

Comments

Projects
None yet
3 participants
@colinsurprenant
Contributor

colinsurprenant commented Nov 13, 2015

Overview

Rewrite the Event class in pure Java and its supporting classes like Timestamp and Accessors and keep 100% compatibility with current implementation of other core logstash components and all plugins.

Purpose

Eventis the main object which encapsulates data and provides and API for the plugins to perform processing on the event content. Having Event implemented in pure Java will improve performance, make possible faster serialization by avoiding costly type conversion between JRuby and Java which in turn will help with an efficient persistence implementation.

Also, these critical and costly operations are related to the Event where the Java implementation should provide performance improvement:

  • fields reference getters and setters: All Event data manipulation in the config is done using the logstash field reference syntax (ex. [foo][bar][baz]). This is implemented in the Accessors class. an optimized implementation should improve performance.
  • string interpolation: (Event#sprintf()) The config syntax provides string interpolation in field references. The Ruby implementation was known to be slow and the Java implementation should provides a noticeable improvement. String interpolation is heavily used in all configs.

Strategy

  • Extract the current Event implementation from logstash-core and make it a proper self-contained gem which should be drop-in replaceable with an alternate implementation. This is completed in #4123 and the gem is logstash-core-event.
  • Create the Java implementation gem logstash-core-event-java. This is in progress in #4123.
  • provide benchmarks
  • have all core specs pass
  • have all core plugins spec pass
  • have all default plugins spec pass

@colinsurprenant colinsurprenant self-assigned this Nov 13, 2015

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

logstash-core-event-java initial impl, relates to elastic#4191
wip initial Accessors, Event, EventImpl, Path, Timestamp and PathTest

wip

wip

FieldReference and Accessors implementation

rename targetCache to lut and set it protected

initial Accessors tests

todo comment

more tests

Timestamp implementation

Timestamp tests

fix method name

add Long constructor

event initialization, timestamp handling and json serialization

add <> type information

custom json serializer for Timestamp

remove toJson test

initial Event test

more tests

comments

debug traces

initial jruby Event wrapper and specs

added PathCache

implemented includes

added clone

wrap all Event methods

Rakefile to build and jar

missing getters and implement overwrite

support Date conversion

proper cast and coercion

replace Ruby Event with Java Event

test for field reference setter type coercion

disable specs

timestap setter should also set in map, accept more timestamp types

pre cache timestamp and expose isTimestamp

constructor from DateTime

expose proper Ruby Timestamp object

Ruby Timestamp basic specs

also load JRuby Timestamp

transpose Java<->Ruby Timestamp

fix timestamp specs

new jar

cleanup object construction

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

Native java implementation of Event#sprintf, relates to elastic#4191
This implementation is different from the master version since instead
of doing JIT Template compilation on every events we create a data
structure of a compiled templates. This template is cached in a
concurrent hashmap and used on other requests.

fixing conflict

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

more sprintf java impl, relates to elastic#4191
First the case with single fieldref

Add support for +%s => epoch

Small comments concerning the caching mechanism and the memory usage

Make the #sprintf method work with Array and Hash

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

more Event java impl, relates to elastic#4191
event clone

append in Ruby

tag & append

add timestamp assignment spec

comment out original classes

fix at signature

implemented del

remove from array

del test

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

refactor to gradle project, relates to elastic#4191
more event java updates

- updated metadata
- moved #append to java
- other small test fixes

commit gradle wrapper jar

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

logstash-core & logstash-core-event extraction to support logstash-co…
…re-event-java impl, relates to elastic#4191

fixed timezone issue

extracted logstash-core and reorganized specs

extracted logstash-core-event

extract java Event into logstash-core-event-java in a proper gem

remove obsolete jruby_event bootstrapping

fix require path

add java code bootstrap

use logstash-core-event/logstash-core-event.rb

remove obsolete files

basic instructions

LogStash::Json need to be initialized from event

update jruby and gradle versions

update compile:logstash-core-event-java rake task

WIP tasks refactor

fix gem.files

skip test if class is not defined

fix gem related tasks for new structure

add gem spec dirs in core tests

bootstrap java implementation when requiring timestamp

new Cloner class and Event clone impl

fix array fields assignments, see elastic#4140

don't rely on json implementation ordering

fix skipped last iterpolation char

remove implementation specific unnecessary check

also require ruby classes

define error class in ruby

raise exception on invalid format

remove implementation specific tests and extract and put logger related test in pending

missing bits for having all core timestamp specs pass

run all core specs

remove leftover

comment regex

missing encoding header

revert to logstash-core-event by default

finished proper gemification

useless require

dynamically pick specs depending on logstash-core-event-* implementation

logstash root package version

missing file for proper gemification

do not build java event by default

always check for root logstash lib dir

fix concurrent-ruby version confict

fix rebase conflict

re-enable specs

user vars instead of constants

move non core code in bootstrap

document version files

move version file

remove useless code

use version in logstash-core

fix gem files list

put back concurrent-ruby version constrain as in master

add dependency on logstash-core-event

remove dependency on logstash-core to avoid circular dependency

fix rebase conflict

remove circular dependency

fix specs

update README

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

logstash-core-event-java initial impl, relates to elastic#4191
wip initial Accessors, Event, EventImpl, Path, Timestamp and PathTest

wip

wip

FieldReference and Accessors implementation

rename targetCache to lut and set it protected

initial Accessors tests

todo comment

more tests

Timestamp implementation

Timestamp tests

fix method name

add Long constructor

event initialization, timestamp handling and json serialization

add <> type information

custom json serializer for Timestamp

remove toJson test

initial Event test

more tests

comments

debug traces

initial jruby Event wrapper and specs

added PathCache

implemented includes

added clone

wrap all Event methods

Rakefile to build and jar

missing getters and implement overwrite

support Date conversion

proper cast and coercion

replace Ruby Event with Java Event

test for field reference setter type coercion

disable specs

timestap setter should also set in map, accept more timestamp types

pre cache timestamp and expose isTimestamp

constructor from DateTime

expose proper Ruby Timestamp object

Ruby Timestamp basic specs

also load JRuby Timestamp

transpose Java<->Ruby Timestamp

fix timestamp specs

new jar

cleanup object construction

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

Native java implementation of Event#sprintf, relates to elastic#4191
This implementation is different from the master version since instead
of doing JIT Template compilation on every events we create a data
structure of a compiled templates. This template is cached in a
concurrent hashmap and used on other requests.

fixing conflict

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

more sprintf java impl, relates to elastic#4191
First the case with single fieldref

Add support for +%s => epoch

Small comments concerning the caching mechanism and the memory usage

Make the #sprintf method work with Array and Hash

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

more Event java impl, relates to elastic#4191
event clone

append in Ruby

tag & append

add timestamp assignment spec

comment out original classes

fix at signature

implemented del

remove from array

del test

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

refactor to gradle project, relates to elastic#4191
more event java updates

- updated metadata
- moved #append to java
- other small test fixes

commit gradle wrapper jar

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

logstash-core & logstash-core-event extraction to support logstash-co…
…re-event-java impl, relates to elastic#4191

fixed timezone issue

extracted logstash-core and reorganized specs

extracted logstash-core-event

extract java Event into logstash-core-event-java in a proper gem

remove obsolete jruby_event bootstrapping

fix require path

add java code bootstrap

use logstash-core-event/logstash-core-event.rb

remove obsolete files

basic instructions

LogStash::Json need to be initialized from event

update jruby and gradle versions

update compile:logstash-core-event-java rake task

WIP tasks refactor

fix gem.files

skip test if class is not defined

fix gem related tasks for new structure

add gem spec dirs in core tests

bootstrap java implementation when requiring timestamp

new Cloner class and Event clone impl

fix array fields assignments, see elastic#4140

don't rely on json implementation ordering

fix skipped last iterpolation char

remove implementation specific unnecessary check

also require ruby classes

define error class in ruby

raise exception on invalid format

remove implementation specific tests and extract and put logger related test in pending

missing bits for having all core timestamp specs pass

run all core specs

remove leftover

comment regex

missing encoding header

revert to logstash-core-event by default

finished proper gemification

useless require

dynamically pick specs depending on logstash-core-event-* implementation

logstash root package version

missing file for proper gemification

do not build java event by default

always check for root logstash lib dir

fix concurrent-ruby version confict

fix rebase conflict

re-enable specs

user vars instead of constants

move non core code in bootstrap

document version files

move version file

remove useless code

use version in logstash-core

fix gem files list

put back concurrent-ruby version constrain as in master

add dependency on logstash-core-event

remove dependency on logstash-core to avoid circular dependency

fix rebase conflict

remove circular dependency

fix specs

update README

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Nov 16, 2015

@colinsurprenant

This comment has been minimized.

Show comment
Hide comment
@colinsurprenant

colinsurprenant Dec 9, 2015

Contributor

All default plugins are now passing specs, all related PRs are tracked in #4264. Merge fest can now start.

Contributor

colinsurprenant commented Dec 9, 2015

All default plugins are now passing specs, all related PRs are tracked in #4264. Merge fest can now start.

@colinsurprenant

This comment has been minimized.

Show comment
Hide comment
@colinsurprenant

colinsurprenant Dec 15, 2015

Contributor

quick benchmark update, I tested the config defined in https://gist.github.com/colinsurprenant/8143a0dfd30830d57b03 using 4 workers -w 4 and default batch size of 125 (not specifying -b) on master pre-g-pipeline, ng-pipeline and ng-pipeline+java-event:

TPS pipeline
4.86k pre ng-pipeline
6.29k ng-pipeline
7.50k ng-pipeline+java-event
Contributor

colinsurprenant commented Dec 15, 2015

quick benchmark update, I tested the config defined in https://gist.github.com/colinsurprenant/8143a0dfd30830d57b03 using 4 workers -w 4 and default batch size of 125 (not specifying -b) on master pre-g-pipeline, ng-pipeline and ng-pipeline+java-event:

TPS pipeline
4.86k pre ng-pipeline
6.29k ng-pipeline
7.50k ng-pipeline+java-event
@untergeek

This comment has been minimized.

Show comment
Hide comment
@untergeek

untergeek Dec 15, 2015

Member

That's an impressive increase!

Member

untergeek commented Dec 15, 2015

That's an impressive increase!

@suyograo suyograo added v2.3.0 and removed v2.2.0 labels Dec 16, 2015

colinsurprenant added a commit to colinsurprenant/logstash that referenced this issue Jan 12, 2016

logstash-core & logstash-core-event extraction to support logstash-co…
…re-event-java impl, relates to elastic#4191 fixed timezone issue

extracted logstash-core and reorganized specs

extracted logstash-core-event

extract java Event into logstash-core-event-java in a proper gem

remove obsolete jruby_event bootstrapping

fix require path

add java code bootstrap

use logstash-core-event/logstash-core-event.rb

remove obsolete files

basic instructions

LogStash::Json need to be initialized from event

update jruby and gradle versions

update compile:logstash-core-event-java rake task

WIP tasks refactor

fix gem.files

skip test if class is not defined

fix gem related tasks for new structure

add gem spec dirs in core tests

bootstrap java implementation when requiring timestamp

new Cloner class and Event clone impl

fix array fields assignments, see elastic#4140

don't rely on json implementation ordering

fix skipped last iterpolation char

remove implementation specific unnecessary check

also require ruby classes

define error class in ruby

raise exception on invalid format

remove implementation specific tests and extract and put logger related test in pending

missing bits for having all core timestamp specs pass

run all core specs

remove leftover

comment regex

missing encoding header

revert to logstash-core-event by default

finished proper gemification

useless require

dynamically pick specs depending on logstash-core-event-* implementation

logstash root package version

missing file for proper gemification

do not build java event by default

always check for root logstash lib dir

fix concurrent-ruby version confict

fix rebase conflict

re-enable specs

user vars instead of constants

move non core code in bootstrap

document version files

move version file

remove useless code

use version in logstash-core

fix gem files list

put back concurrent-ruby version constrain as in master

add dependency on logstash-core-event

remove dependency on logstash-core to avoid circular dependency

fix rebase conflict

remove circular dependency

fix specs

update README
@colinsurprenant

This comment has been minimized.

Show comment
Hide comment
@colinsurprenant

colinsurprenant Jan 13, 2016

Contributor

I am calling this issue closed, Java Event is merged and all core and plugins specs are passing. I will followup with a post-merge review process issue and we can create new issues as needed.

Contributor

colinsurprenant commented Jan 13, 2016

I am calling this issue closed, Java Event is merged and all core and plugins specs are passing. I will followup with a post-merge review process issue and we can create new issues as needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment