Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kinesis Elasticsearch Sink: cache the mapping of field name to field type #2090

Closed
fblundun opened this issue Oct 2, 2015 · 2 comments
Closed
Assignees

Comments

@fblundun
Copy link
Contributor

fblundun commented Oct 2, 2015

The way this currently works, for every event, for every field, the code checks based on the name of the field how that field should be handled using the converter function. This involves checking whether the string name of the field is a member of various sets. It would be faster to precompute and cache the correspondence between field names and handler functions. (Crude benchmarking appears to show that this roughly doubles the speed of the jsonifyGoodEvent function.)

@fblundun fblundun self-assigned this Oct 2, 2015
@fblundun fblundun added this to the Kinesis #7 milestone Oct 2, 2015
@fblundun
Copy link
Contributor Author

fblundun commented Oct 2, 2015

My benchmark was flawed because it didn't take JIT optimization into account - the actual performance increase is a lot smaller. Still, I think this design change is an improvement.

@fblundun fblundun changed the title Kinesis Elasticsearch Sink: optimize transformation of TSV to JSON Kinesis Elasticsearch Sink: cache the mapping of field name to field type Oct 2, 2015
@alexanderdean
Copy link
Member

Agree!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants