No indent option on --json and --log-json

[xeor]

It would be much easier for tools reading logs from stdout to parse the json if every line was a single entry (ie, no indention).

What would be the best way to implement this? Using another option, like --json-indent=0, or --json-singleline?

I can make a PR on this, but wanted to discuss it first.

[ThomasWaldmann]

"Easier"? Do you mean it would be faster? Did you measure that?

I don't think a json parser has bad feelings about repeatedly having to skip some whitespace. :-D

[xeor]

I feel sorry for the parsers that look at the stdout on all this json. They will have it much easier if they could knew that there was 1 entry per line, so they don't need to figure out where the json ends.

This might be a everything from a simple script to splunk. Much easier to use the \n, than using a lot of logic..

The json output option is there to make the output easier to parse for robots after all, why mix in the humanized layer (indention) when they don't care..? :)

[enkore]

stdout only gets one JSON object - so that one is easy, you just hand the entire stdout output to the parser.

stderr is already jsonlines.

[xeor]

oh...... I didn't notice that :) Problem solved! Thanks!

[xeor]

Sorry for reopening this. But even tho the stream of info, and the summary is separated into stdout and stderr, it makes it much harder than it need to be to parse the summary.

An example (in my case):

• borg backup runs every night inside a container inside a kubernetes cluster
• container output is picked up by a logger collector (fluentd), then sent to splunk
• splunk parses the stderr fine as default, being able to extract the key/values from the json
• the stdout however, contains multiple lines, and splunk sees the lines as a separate event.

This is an example with splunk, but other log-tools would probably also have the same problem.

It is fixable, but then we need a lot of extra log-logic configuration for what an "event" is, logic being separated by stderr/stdout, or something..

[ThomasWaldmann]

I'ld say if the output of borg is not what your tools can process, you just need to do some intermediate processing to make it fit.

[xeor]

Yes.. Some intermediate processing would fix this. But either doing that in fluentd or splunk would still feel wrong. Getting the json output option is awesome, but why not go all the way making it even easier for machine, making it obvious when the blob starts and ends, without extra configuration or intermediate steps.

Will a PR on this gets accepted?

[enkore]

I don't think the specific format of stdout is actually guaranteed in JSON mode so you can probably just change it.

[xeor]

Correct me if i'm wrong, but isnt

borg/src/borg/archive.py

Line 118 in 44e5365

msg = json.dumps(data)

the part that output what we discuss? I thought there where going to be an indent=4 here somewhere. Is this getting "prettified" somewhere else?

[xeor]

Sorry, based on the docs, it sais

An indent level of 0, negative, or "" will only insert newlines. None (the default) selects the most compact representation.

So I guess, it's hard to know... :)

[enkore]

Nope, that's progress. Stats are this:

borg/src/borg/archiver.py

Line 488 in 1922ae3

if args.stats:

[xeor]

aha.. yes...

So it is ultimately

borg/src/borg/helpers/parseformat.py

Line 925 in f83e5b7

return json.dumps(obj, sort_keys=True, indent=4, cls=BorgJsonEncoder)

that is the one that I want configurable..

[enkore]

You don't want it configurable, you just need to get rid of the indentation ;)

[xeor]

So will the logic of something like 0 if args.flatten_json else 4 be "ok" at that line?