Filebeat httpjson input response.split fails on an array of arrays #30345
Comments
botelastic
bot
added
the
needs_team
|
Pinging @elastic/security-external-integrations (Team:Security-External Integrations) |
botelastic
bot
removed
the
needs_team
|
Same issue happens if u have an array of strings. Its caused by the forced casting here, . I added the below to the Switch and it seems to work.Using the example above it produces |
|
Was annoyingly hit with this issue too, with json with this structure: {
"row_headers":[
"Date",
"Time Spent (seconds)"
],
"rows":[
[
"2022-06-18T00:00:00",
123
],
[
"2022-06-18T00:00:00",
456
]
]
}Was banging my head for a while, as first time using go templates, but eventually I managed to come up with this as a workaround. - set:
target: body.rows
value_type: json
value: '[
[[- $last_row := len (slice .last_response.body.rows 1) -]]
[[- $headers := .last_response.body.row_headers -]]
[[- range $row_idx, $row := .last_response.body.rows -]]
{
[[- $last_cell := len (slice $row 1) -]]
[[- range $cell_idx, $cell := $row -]]
[[- sprintf "%q : %q" (js (index $headers $cell_idx)) (js $cell) -]]
[[- if lt $cell_idx $last_cell -]][[- sprintf ","]][[end]]
[[end]]
}
[[- if lt $row_idx $last_row -]][[- sprintf ","]][[end]]
[[end]]
]'Which converts each element in the row array to json objects, with their values keyed by the appropriate header. I ran into some encoding issues on some of the data, hence the js call to encode the values. Resulting in body.rows looking something like this [
{
"Date" : "2022-06-18T00:00:00",
"Time Spent (seconds)" : "123"
},
{
"Date" : "2022-06-18T00:00:00",
"Time Spent (seconds)" : "456"
}
]Hope it helps others before the fix above gets merged and becomes available (or someone points out a better approach 😉 )
Here is a slightly modified (and not perfect) version, which appears to work with your example @wasserman - do I win the bonus points? 🏆 😄 |
@tompipe thanks for your workaround! Could you share the rest of the filebeat definition? I understand you use your snippet as a |
When trying to split a response of arrays I get the following error:
error(*errors.errorString) *{s: "split was expecting field to be an object"}It happens here:
beats/x-pack/filebeat/input/httpjson/internal/v2/split.go
Line 145 in 2c09708
Returned from here:
beats/x-pack/filebeat/input/httpjson/internal/v2/split.go
Line 214 in 2c09708
Keep in mind that the default should be
arrayand the flow does go down this path.My scenario is similar to the example I'm providing here, but I just used the Elastic demo site to make it easy to test and reproduce.
A sample input generated by the following from https://demo.elastic.co/app/dev_tools#/console:
Saved the output and served as a static file from local Web server for simplicity:
Filebeat input config as follows:
Tested on Filebeat 7.16.2.
I hope that this is a simple thing to fix.
Bonus points for some smart way to map the
columnstorowsafter such a split so Filebeat can output documents!Thanks!
The text was updated successfully, but these errors were encountered: