Problem parsing JSON

[alvassin]

I have xunit.json file i want to parse & to use its keys as fields names & values as field values.
I tried a couple examples over internet, but anyway i runs to following problem. Could you please assist?

Trouble parsing json {:source=>"message", :raw=>"{", :exception=>#<JSON::ParserError: A JSON text must at least contain two octets!>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"  \"time\" : {", :exception=>#<JSON::ParserError: unexpected token at '"time" : {'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"    \"start\" : 1419621623182,", :exception=>#<JSON::ParserError: unexpected token at '"start" : 1419621623182,'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"    \"stop\" : 1419621640826,", :exception=>#<JSON::ParserError: unexpected token at '"stop" : 1419621640826,'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"    \"duration\" : 17644", :exception=>#<JSON::ParserError: unexpected token at '"duration" : 17644'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"  },", :exception=>#<JSON::ParserError: unexpected token at '},'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"  \"testSuites\" : [ {", :exception=>#<JSON::ParserError: unexpected token at '"testSuites" : [ {'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"    \"uid\" : \"ac7efb112b3b91cc\",", :exception=>#<JSON::ParserError: unexpected token at '"uid" : "ac7efb112b3b91cc",'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"    \"name\" : \"tests.buylinks\",", :exception=>#<JSON::ParserError: unexpected token at '"name" : "tests.buylinks",'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"    \"title\" : \"Buylinks\",", :exception=>#<JSON::ParserError: unexpected token at '"title" : "Buylinks",'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"    \"time\" : {", :exception=>#<JSON::ParserError: unexpected token at '"time" : {'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"      \"start\" : 1419621623182,", :exception=>#<JSON::ParserError: unexpected token at '"start" : 1419621623182,'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"      \"stop\" : 1419621640826,", :exception=>#<JSON::ParserError: unexpected token at '"stop" : 1419621640826,'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"      \"duration\" : 17644", :exception=>#<JSON::ParserError: unexpected token at '"duration" : 17644'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"    },", :exception=>#<JSON::ParserError: unexpected token at '},'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"    \"statistic\" : {", :exception=>#<JSON::ParserError: unexpected token at '"statistic" : {'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"      \"total\" : 3,", :exception=>#<JSON::ParserError: unexpected token at '"total" : 3,'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"      \"passed\" : 0,", :exception=>#<JSON::ParserError: unexpected token at '"passed" : 0,'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"      \"pending\" : 0,", :exception=>#<JSON::ParserError: unexpected token at '"pending" : 0,'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"      \"canceled\" : 0,", :exception=>#<JSON::ParserError: unexpected token at '"canceled" : 0,'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"      \"failed\" : 3,", :exception=>#<JSON::ParserError: unexpected token at '"failed" : 3,'>, :level=>:warn}
Trouble parsing json {:source=>"message", :raw=>"      \"broken\" : 0", :exception=>#<JSON::ParserError: unexpected token at '"broken" : 0'>, :level=>:warn}

Xunit looks like the following:

{
  "time" : {
    "start" : 1419621623182,
    "stop" : 1419621640826,
    "duration" : 17644
  },
  "testSuites" : [ {
    "uid" : "ac7efb112b3b91cc",
    "name" : "tests.buylinks",
    "title" : "Buylinks",
    "time" : {
      "start" : 1419621623182,
      "stop" : 1419621640826,
      "duration" : 17644
    },
    "statistic" : {
      "total" : 3,
      "passed" : 0,
      "pending" : 0,
      "canceled" : 0,
      "failed" : 3,
      "broken" : 0
    },
    "description" : null,
    "testCases" : [ {
      "uid" : "441d1d1dd296fe60",
      "name" : "test_buylinks",
      "title" : "Test buylinks",
      "time" : {
        "start" : 1419621623182,
        "stop" : 1419621640491,
        "duration" : 17309
      },
      "severity" : "NORMAL",
      "status" : "FAILED"
    }, {
      "uid" : "a88c89b377aca0c9",
      "name" : "test_buylinks",
      "title" : "Test buylinks",
      "time" : {
        "start" : 1419621623182,
        "stop" : 1419621640634,
        "duration" : 17452
      },
      "severity" : "NORMAL",
      "status" : "FAILED"
    }, {
      "uid" : "32c3f8b52386c85c",
      "name" : "test_buylinks",
      "title" : "Test buylinks",
      "time" : {
        "start" : 1419621623185,
        "stop" : 1419621640826,
        "duration" : 17641
      },
      "severity" : "NORMAL",
      "status" : "FAILED"
    } ]
  } ]
}

My config:

input {
  file {
    type => "testcases"
    path => [ "/home/logstash/xunit.json" ]
    start_position => "beginning"
    sincedb_path => "/home/logstash/sincedb"
  }
}

filter {
  json {
    source => "message"
    target => "doc"
    add_field => [ "testsuite", "%{[message][testSuites][name]}" ]
    add_field => [ "testcase", "%{[message][testSuites][testCases][name]}" ]
  }
}
output {
  elasticsearch {
    host => "elasticsearch.dev"
  }
}

P.S. I am using latest logstash i've downloaded from website 1.4.2.

[talevy]

@alvassin it looks like you are attempting to reference the "message" field as the target of your json document during the add_field phase. The parsed JSON is placed in the target field by the json filter.

maybe try:

add_field => [ "testsuite", "%{[doc][testSuites][name]}" ]
add_field => [ "testcase", "%{[doc][testSuites][testCases][name]}" ]

[alvassin]

@talevy thanks a lot for your response. This helps, but i also have to make JSON without any newlines/whitespace, otherwise logstash tries to parse line-by-line, not whole json. And i still see Trouble parsing json error.

Is it possible to pre-filter the whole file to remove all whitespace and newlines, or to ignore them and to recognize json from whole file, not per line?

[talevy]

Oh, I did not realize that your json was spread out in separate files. For messages that span multiple lines. I suggest taking a look at the multiline codec

[alvassin]

@talevy thanks a lot, i can see my data is loading in json now.

[wanghaisheng]

could you share your solution? @alvassin

[amiya-elear]

input {
tcp {
port => 5001
type => "tomcat"
}

tcp {
port => 5000
type => "nginx"
codc=> "json"
}

}

Add your filters / logstash plugins configuration here

filter {
json {
source => "message"
target => "parsedjson"
}
mutate {
add_field => { "Source_IP" => "%{[_source][SOURCEIP]}" }
}
}

output {

if [type] == "tomcat" {
    elasticsearch {
        hosts => "elasticsearch:9200"
        index => "tomcat_index"
    }
} else if [type] == "nginx"{
    elasticsearch {
        hosts => "elasticsearch:9200"
        index => "nginx_index"
    }
} else {
     elasticsearch {
                   hosts => "elasticsearch:9200"
                   index => "default"
          }
     }

}

this is my logstash config file and i am having the same issue. kindly help me o this.
error file:

[2019-02-18T11:28:04,248][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"<30>Feb 18 16:58:04 nginx[2170]: 172.17.0.1 - - [18/Feb/2019:11:28:04 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0" "-"", :exception=>#<LogStash::Json::ParserError: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

thanks in advance

[jsvd]

@amiya-elear please open an issue if you want to report a problem.
Looking at the text of that error message the json filter is being applied to a field whose message contains a line of text that isn't json, so the error is normal/expected.