New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON Extractor fails to convert to fields when there are fields with null value #1475

Closed
gvagenas opened this Issue Oct 10, 2015 · 10 comments

Comments

Projects
None yet
4 participants
@gvagenas

gvagenas commented Oct 10, 2015

Working with the latest 1.2.1.

Even though when testing with "Try" I can see all the fields, even the ones with null value, the extractor doesn't convert the string to fields.

I believe the fields with null value should be extracted with string "null" instead of null value

gvagenas added a commit to gvagenas/graylog2-server that referenced this issue Oct 10, 2015

@joschi

This comment has been minimized.

Contributor

joschi commented Oct 10, 2015

@gvagenas Could you please elaborate on the problem and maybe add a concrete example to the issue?

@gvagenas

This comment has been minimized.

gvagenas commented Oct 10, 2015

@joschi see bellow a json message that contains null values:

{  
  "1xx-processed":0,
  "1xx-sent":0,
  "2xx-processed":0,
  "2xx-sent":0,
  "3xx-processed":0,
  "3xx-sent":0,
  "4xx-processed":0,
  "4xx-sent":0,
  "5xx-processed":0,
  "5xx-sent":0,
  "6xx-processed":0,
  "6xx-sent":0,
  "7xx-processed":0,
  "7xx-sent":0,
  "8xx-processed":0,
  "8xx-sent":0,
  "9xx-processed":0,
  "9xx-sent":0,
  "ack-processed":0,
  "ack-sent":0,
  "additional-parameterable-headers":null,
  "app-dispatcher-class":"org.mobicents.servlet.sip.core.SipApplicationDispatcherImpl",
  "application-router":"configuration/dars/mobicents-dar.properties",
  "back-to-normal-memory-threshold":75,
  "base-timer-interval":500,
  "bye-processed":0,
  "bye-sent":0,
  "cancel-processed":0,
  "cancel-sent":0,
  "canceled-timer-tasks-purge-period":-1,
  "concurrency-control-mode":"SipApplicationSession",
  "congestion-control-interval":"-1",
  "congestion-control-policy":"ErrorResponse",
  "dialog-pending-request-checking":false,
  "dns-resolver-class":null,
  "dns-server-locator-class":null,
  "dns-timeout":1,
  "gather-statistics":false,
  "info-processed":0,
  "info-sent":0,
  "instance-id":null,
  "invite-processed":0,
  "invite-sent":0,
  "memory-threshold":85,
  "message-processed":0,
  "message-sent":0,
  "notify-processed":0,
  "notify-sent":0,
  "number-client-transaction":0,
  "number-dialog":0,
  "number-early-dialog":0,
  "number-server-transaction":0,
  "options-processed":0,
  "options-sent":0,
  "outbound-proxy":null,
  "path-name":"org.mobicents.ext",
  "prack-processed":0,
  "prack-sent":0,
  "publish-processed":0,
  "publish-sent":0,
  "refer-processed":0,
  "refer-sent":0,
  "register-processed":0,
  "register-sent":0,
  "stack-properties":"configuration/mss-sip-stack.properties",
  "subscribe-processed":0,
  "subscribe-sent":0,
  "t2-interval":4000,
  "t4-interval":5000,
  "timer-d-interval":32000,
  "update-processed":0,
  "update-sent":0,
  "use-pretty-encoding":true,
  "connector":{  
    "sip-udp":null,
    "sip-tcp":null,
    "sip-tls":null,
    "sip-ws":null
  }
}

When trying to extract fields with a JSON extractor the "Try!" button gives me all the fields as expected and the ones with null values comes with just the key:

cancel-processed
    0
*** connector.sip-udp ***
application-router
    configuration/dars/mobicents-dar.properties
message-sent
    0
path-name
    org.mobicents.ext
subscribe-processed
    0
base-timer-interval
    500
update-sent
    0
...

Noticed the property "connector" : { "sip-udp" : null => connector.sip-udp ?

Now even like this when I save the extractor, I see no extracted fields, not even the one that have values.

After applying the patch #1476 it will come with:
connector.sip-udp = null and all the fields will be available.

Thanks
George

@joschi

This comment has been minimized.

Contributor

joschi commented Oct 10, 2015

Mapping a JSON null value to null is intentional and mapping null to "null" would change this semantic.

Please check your Graylog server logs for error messages (maybe a NullPointerException) around the time you've received those messages.

@gvagenas

This comment has been minimized.

gvagenas commented Oct 10, 2015

@joschi, there are no NPE at server log.
I changed to DEBUG log level and created a gist, please check here:
https://gist.github.com/gvagenas/7903010bbe4912ca2d7f

George

@gvagenas

This comment has been minimized.

gvagenas commented Oct 10, 2015

Also, I agree on the semantics of null, the patch I provided was a quick hack that maybe can provide hints for where is the problem

@chiefy

This comment has been minimized.

chiefy commented Oct 13, 2015

Currently evaluating GL as we're not happy with CloudWatch logging & JSON, and this is kind of a blocker. 😞

@bernd bernd self-assigned this Oct 22, 2015

@bernd bernd added the bug label Oct 22, 2015

@bernd bernd added this to the 1.2.2 milestone Oct 22, 2015

bernd added a commit that referenced this issue Oct 23, 2015

bernd added a commit that referenced this issue Oct 23, 2015

Ingore fields with null values in JsonExtractor
Fixes #1475

(cherry picked from commit a6f0157)

@bernd bernd closed this in 0b1a2b9 Oct 23, 2015

@bernd

This comment has been minimized.

Member

bernd commented Oct 23, 2015

The issue has been fixed and will be in the upcoming Graylog 1.2.2 release.

@gvagenas

This comment has been minimized.

gvagenas commented Oct 26, 2015

Thanks @bernd . Do you have a ETA for the 1.2.2 release? I have a blog post ready for how to use Graylog with Restcomm (www.restcomm.com) and waiting the 1.2.2 release.

Regards
George

@joschi

This comment has been minimized.

Contributor

joschi commented Oct 26, 2015

@gvagenas Graylog 1.2.2 is being released as we speak.

@gvagenas

This comment has been minimized.

gvagenas commented Oct 26, 2015

Thats great :) Congrats guys

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment