Use 'oj' for performance improvement when oj is installed #748

repeatedly · 2015-12-16T10:09:27Z

oj is faster json library and it improves JSONParser performance.
I tested in_tail with following json.

{"_id":"5671172e7ba230441460f011","index":0,"guid":"3d18838a-ae85-4986-83c7-e4a4acdd7f01","isActive":false,"balance":"$3,929.20","picture":"http://placehold.it/32x32","age":40,"eyeColor":"green","company":"MEDMEX","email":"hardin.sargent@medmex.org","phone":"+1 (865) 468-2885","address":"302 Buffalo Avenue, Bend, Louisiana, 635","latitude":"89.385551","longitude":"75.132185","greeting":"Hello, Hardin! You have 5 unread messages.","favoriteFruit":"banana"}

Result is

Yajl

2015-12-16 18:23:37 +0900 [info]: following tail of /Users/repeatedly/tmp/fluentd/json.log
2015-12-16 18:23:38 +0900 [info]: plugin:out_flowcounter_simple count:2003      indicator:num   unit:second
2015-12-16 18:23:39 +0900 [info]: plugin:out_flowcounter_simple count:21037     indicator:num   unit:second
2015-12-16 18:23:40 +0900 [info]: plugin:out_flowcounter_simple count:19033     indicator:num   unit:second
2015-12-16 18:23:41 +0900 [info]: plugin:out_flowcounter_simple count:20034     indicator:num   unit:second
2015-12-16 18:23:42 +0900 [info]: plugin:out_flowcounter_simple count:19033     indicator:num   unit:second
2015-12-16 18:23:43 +0900 [info]: plugin:out_flowcounter_simple count:21037     indicator:num   unit:second
2015-12-16 18:23:44 +0900 [info]: plugin:out_flowcounter_simple count:18031     indicator:num   unit:second
2015-12-16 18:23:45 +0900 [info]: plugin:out_flowcounter_simple count:20035     indicator:num   unit:second
2015-12-16 18:23:46 +0900 [info]: plugin:out_flowcounter_simple count:19033     indicator:num   unit:second
2015-12-16 18:23:47 +0900 [info]: plugin:out_flowcounter_simple count:20035     indicator:num   unit:second
2015-12-16 18:23:48 +0900 [info]: plugin:out_flowcounter_simple count:20035     indicator:num   unit:second
2015-12-16 18:23:49 +0900 [info]: plugin:out_flowcounter_simple count:20034     indicator:num   unit:second
2015-12-16 18:23:50 +0900 [info]: plugin:out_flowcounter_simple count:19033     indicator:num   unit:second
2015-12-16 18:23:51 +0900 [info]: plugin:out_flowcounter_simple count:20035     indicator:num   unit:second
2015-12-16 18:23:52 +0900 [info]: plugin:out_flowcounter_simple count:19033     indicator:num   unit:second
2015-12-16 18:23:53 +0900 [info]: plugin:out_flowcounter_simple count:20035     indicator:num   unit:second
2015-12-16 18:23:54 +0900 [info]: plugin:out_flowcounter_simple count:21037     indicator:num   unit:second
2015-12-16 18:23:55 +0900 [info]: plugin:out_flowcounter_simple count:19033     indicator:num   unit:second
2015-12-16 18:23:56 +0900 [info]: plugin:out_flowcounter_simple count:19033     indicator:num   unit:second
2015-12-16 18:23:57 +0900 [info]: plugin:out_flowcounter_simple count:18031     indicator:num   unit:second
2015-12-16 18:23:58 +0900 [info]: plugin:out_flowcounter_simple count:20035     indicator:num   unit:second
2015-12-16 18:23:59 +0900 [info]: plugin:out_flowcounter_simple count:21036     indicator:num   unit:second
2015-12-16 18:24:00 +0900 [info]: plugin:out_flowcounter_simple count:18032     indicator:num   unit:second
2015-12-16 18:24:01 +0900 [info]: plugin:out_flowcounter_simple count:20034     indicator:num   unit:second
2015-12-16 18:24:02 +0900 [info]: plugin:out_flowcounter_simple count:18032     indicator:num   unit:second

Oj

2015-12-16 18:25:09 +0900 [info]: following tail of /Users/repeatedly/tmp/fluentd/json.log
2015-12-16 18:25:10 +0900 [info]: plugin:out_flowcounter_simple count:19033     indicator:num   unit:second
2015-12-16 18:25:11 +0900 [info]: plugin:out_flowcounter_simple count:28048     indicator:num   unit:second
2015-12-16 18:25:12 +0900 [info]: plugin:out_flowcounter_simple count:33058     indicator:num   unit:second
2015-12-16 18:25:13 +0900 [info]: plugin:out_flowcounter_simple count:25043     indicator:num   unit:second
2015-12-16 18:25:14 +0900 [info]: plugin:out_flowcounter_simple count:31054     indicator:num   unit:second
2015-12-16 18:25:15 +0900 [info]: plugin:out_flowcounter_simple count:29050     indicator:num   unit:second
2015-12-16 18:25:16 +0900 [info]: plugin:out_flowcounter_simple count:30053     indicator:num   unit:second
2015-12-16 18:25:17 +0900 [info]: plugin:out_flowcounter_simple count:33057     indicator:num   unit:second
2015-12-16 18:25:18 +0900 [info]: plugin:out_flowcounter_simple count:27047     indicator:num   unit:second
2015-12-16 18:25:19 +0900 [info]: plugin:out_flowcounter_simple count:32056     indicator:num   unit:second
2015-12-16 18:25:20 +0900 [info]: plugin:out_flowcounter_simple count:29050     indicator:num   unit:second
2015-12-16 18:25:21 +0900 [info]: plugin:out_flowcounter_simple count:31054     indicator:num   unit:second
2015-12-16 18:25:22 +0900 [info]: plugin:out_flowcounter_simple count:32056     indicator:num   unit:second
2015-12-16 18:25:23 +0900 [info]: plugin:out_flowcounter_simple count:26045     indicator:num   unit:second
2015-12-16 18:25:24 +0900 [info]: plugin:out_flowcounter_simple count:32056     indicator:num   unit:second
2015-12-16 18:25:25 +0900 [info]: plugin:out_flowcounter_simple count:34059     indicator:num   unit:second
2015-12-16 18:25:26 +0900 [info]: plugin:out_flowcounter_simple count:37064     indicator:num   unit:second
2015-12-16 18:25:27 +0900 [info]: plugin:out_flowcounter_simple count:32056     indicator:num   unit:second
2015-12-16 18:25:28 +0900 [info]: plugin:out_flowcounter_simple count:37064     indicator:num   unit:second
2015-12-16 18:25:29 +0900 [info]: plugin:out_flowcounter_simple count:34059     indicator:num   unit:second
2015-12-16 18:25:30 +0900 [info]: plugin:out_flowcounter_simple count:35061     indicator:num   unit:second
2015-12-16 18:25:31 +0900 [info]: plugin:out_flowcounter_simple count:35061     indicator:num   unit:second
2015-12-16 18:25:32 +0900 [info]: plugin:out_flowcounter_simple count:33057     indicator:num   unit:second

Oj result is better than Yajl so using oj is better when oj is installed.

repeatedly · 2015-12-16T10:48:08Z

@tagomoris @sonots How about this?

mr-salty · 2015-12-16T15:17:23Z

Should we use multi_json? It does exactly what we're doing here (selects the best available json library).

repeatedly · 2015-12-16T15:47:14Z

@mr-salty It is one idea. I considered multi_json first but multi_json adds a overhead.
Parser is called frequently in input plugin so I want to reduce such overhead.
This is why I don't use multi_json in this patch.

repeatedly · 2015-12-16T16:56:48Z

I added oj support in formatter.

In addition, I added disable_oj parameter for existing users.
If hit dump / load errors with oj, user can use this parameter to avoid oj specific error.
If there is no problem, we can remove this parameter in the future.

sonots · 2015-12-16T19:46:19Z

Very interesting. Let us go ahead.

sonots · 2015-12-16T19:47:58Z

lib/fluent/formatter.rb

@@ -131,8 +131,22 @@ class JSONFormatter < Formatter
      include HandleTagAndTimeMixin
      include StructuredFormatMixin

+      config_param :disable_oj, :bool, :default => false


Option like json_parser oj or json_parser yajl looks better because there exists more parsers (cf. https://github.com/intridea/multi_json#supported-json-engines)

I don't want to support more parsers and I want to remove this option in the feature.
So I don't want to use more better name because users may misunderstands fluentd will support another parsers.

Okay, please leave the comment in codes.

Use 'oj' for performance improvement when oj is installed

repeatedly added 2 commits December 16, 2015 18:26

Use 'oj' for performance improvement when oj is installed

220600b

Fix ParserTestDriver

2acac62

repeatedly added 3 commits December 17, 2015 01:46

Fix FormatterTestDriver

b8bd091

Use 'oj' in JSONFormatter when 'oj' is installed

a2a2379

Add disable_oj option to use yajl for existing users

f3b5474

sonots reviewed Dec 16, 2015
View reviewed changes

repeatedly added 4 commits December 17, 2015 05:31

Use 'json_parser' instead of 'oj'

665575d

Use compat mode in oj dump

d1f34e9

Add test for both oj and yajl

8d0d189

Fix out_stdout test with json format

6646a73

repeatedly added a commit that referenced this pull request Dec 18, 2015

Merge pull request #748 from fluent/support-oj-in-json-parser

955f24d

Use 'oj' for performance improvement when oj is installed

repeatedly merged commit 955f24d into master Dec 18, 2015

repeatedly deleted the support-oj-in-json-parser branch December 18, 2015 11:50

repeatedly added a commit that referenced this pull request Dec 21, 2015

Merge pull request #748 from fluent/support-oj-in-json-parser

d179ee9

Use 'oj' for performance improvement when oj is installed

repeatedly added v0.12 v0.14 labels Dec 22, 2015

This was referenced Apr 22, 2016

Detect JSON by checking each line, in all ingested logs GoogleCloudPlatform/fluent-plugin-google-cloud#65

Closed

Switch to a more performant JSON parsing library GoogleCloudPlatform/fluent-plugin-google-cloud#67

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use 'oj' for performance improvement when oj is installed #748

Use 'oj' for performance improvement when oj is installed #748

repeatedly commented Dec 16, 2015

repeatedly commented Dec 16, 2015

mr-salty commented Dec 16, 2015

repeatedly commented Dec 16, 2015

repeatedly commented Dec 16, 2015

sonots commented Dec 16, 2015

sonots Dec 16, 2015

repeatedly Dec 16, 2015

sonots Dec 16, 2015

Use 'oj' for performance improvement when oj is installed #748

Use 'oj' for performance improvement when oj is installed #748

Conversation

repeatedly commented Dec 16, 2015

repeatedly commented Dec 16, 2015

mr-salty commented Dec 16, 2015

repeatedly commented Dec 16, 2015

repeatedly commented Dec 16, 2015

sonots commented Dec 16, 2015

sonots Dec 16, 2015

Choose a reason for hiding this comment

repeatedly Dec 16, 2015

Choose a reason for hiding this comment

sonots Dec 16, 2015

Choose a reason for hiding this comment