How to skip non-json lines?

[luckydonald]

The situation:
I am getting input, piped in from another program.
Sometimes there are error messages, which are no valid json.
The next line there will be valid json (or another error) again.
So how can I say "discard the line (ending with \n) and continue with the next one"?

Example how piped input looks like:

Program started...
{"example":{"valid-json": true}}
ERROR LOL
{"valid-json": "yes, it is.","parse me": ["please", "i beg you"]}
["sometimes", "I", "get", "lists", "too"]

How can I ignore all lines which

failes to parse?
The closest I got was
.|split("\n")[]| fromjson.example with Raw Input enabled.
It at least processed all lines, and only raises errors on the non-json lines.
Still I don't found a way how to surpress the errors, and get the correct lines printed again.

[luckydonald]

Ugly workaround, grep to use only lines starting with "{":

the_executable | grep '^{' | jq '.'

[dtolnay]

the_executable | jq -R 'fromjson?'

[dtolnay]

You can skip non-objects with:

the_executable | jq -R 'fromjson? | select(type == "object")'

[luckydonald]

dhcp232-052:.Trash luckydonald$ cat cli_temp.txt | jq -R 'fromjson? | select(type == "object")'

error: syntax error, unexpected '?', expecting $end
fromjson? | select(type == "object")        1 compile error

Latest version from homebrew (jq-1.4)

[luckydonald]

Also I realized, (in case of grep) it will only output, after the executable is terminated, while grep displayes instantly.

[dtolnay]

Ah it looks like ? was added just barely after the release of 1.4 in 7fce342 and 7d3a44a. Do you want to try jq-1.5rc2 which came out a week ago? brew install jq --devel

[pkoppstein]

@luckydonald wrote:

Ugly workaround ...

Perhaps in your case using "?" or a simple grep to select JSON entities will work, but both approaches are somewhat brittle in that they will only work reliably if every JSON entity is on a line by itself.

If on the other hand that assumption cannot be made, then focusing on weeding out the error messages might well be the way to go. The key here is that newlines are not allowed inside JSON strings. This makes it quite likely that "grep -v" will be your friend.

[luckydonald]

Version from brew install jq --devel works fine, thanks.

[chesshacker]

FWIW, I came up with this to display pretty JSON lines mixed with regular text lines:

the_executable | jq -R -r '. as $line | try fromjson catch $line'

[pkoppstein]

@chesshacker - Nice example. As far as the documentation is concerned, the use of try/catch in this context is already documented in the FAQ (https://github.com/stedolan/jq/wiki/FAQ) -- search for try/catch

[DaPutzy]

Also I realized, (in case of grep) it will only output, after the executable is terminated, while grep displayes instantly.

@luckydonald You need to use grep with --line-buffered if you pipe it into sth else, see: https://stackoverflow.com/a/37034112

[acheronfail]

If it's any help, this function works wonders for me:

# Use `jq` with both JSON and non-JSON lines.
function jjq {
    jq -R -r "${1:-.} as \$line | try fromjson catch \$line"
}

[skylarmb]

I took a simpler approach and just filtered bad lines out with grep

cat logfile | grep -e "^\[" -e "^{" | jq

[johnb85022]

Sed helped me, ansible facts input example, not same as the input talked about here, but is a good example.
ansible localhost -m setup -a "filter=date" |sed '1c {'|jq '.ansible_facts.ansible_date_time.weekday'''
and
ansible localhost -m setup -a "filter=date" | sed '1 s/^.*$/{/' |jq '.ansible_facts.ansible_date_time.weekday'