Filebeat unmarshals JSON integers as float64

[andrewkroh]

Filebeat can unmarshal arbitrary JSON data, and when it unmarshals numbers they are of type float64. If you try to use a conditional filter with equals to match against a number read from JSON you get the following error.

condition.go:240: WARN unexpected type float64 in equals condition as it accepts only integers and strings.

Filebeat version: 5.0.0 git c54b7c4

Config:

$ cat filebeat.dev.yml 
filebeat.prospectors:
  - paths:
    - "./input.json"
    json.keys_under_root: false

processors:
  - drop_event:
      when:
        and:
          - contains:
              json.http_user_agent: "ELB-HealthChecker"
          - equals:
              json.status: 200

output.console:
  pretty: true

logging.level: debug

Input file:

$ cat input.json 
{"http_user_agent": "ELB-HealthChecker/1.0", "status": 200}

[tsg]

The fact that JSON unmarshals all numbers to float is quite unfortunate. Maybe we can fix that..

[monicasarbu]

This is the expected behavior. "equals" excepts only integers and strings as it doesn't really make sense for floats. If you want to compare a float, then the "range" operation is needed.

[monicasarbu]

@andrewkroh How the numbers look like in the json file? Do they look like integers?

[andrewkroh]

@monicasarbu Yes, the value is an integer in the JSON. I included an example in the description.

I updated the title to reflect the fact that the focus should probably be on the unmarshaling behavior instead of the processor condition since equals is designed for ints and strings.

[tsg]

Unfortunately that's just how Unmarshal works in Go: if it's a number, put it in float64. One option that I'm thinking is that after unmarshalling, we do a pass through the result and for all float64s we check if they are within ε of an integer, and if they are we convert to int64. We make ε configurable with a default value of 1e-6 or something like that.

@andrewkroh @monicasarbu Thoughts? Alternatives?