Explore: Prettify json log

[ventris]

What would you like to be added:
Some way to pretty print the searchable json logs in the explore ui. Either through an option in the configuration menu or a checkbox in the explore view itself to enable it globally for all json data.

Why is this needed:
Reading big chunks of json data is hard and its easy to get lost in it. If there was a way to prettify the data it would be much easier to review and find the thing that you are searching for in the log line.

Example:

Non prettified data:

{"log":"{\"type\":\"response\",\"@timestamp\":\"2019-08-12T20:06:43Z\",\"tags\":[],\"pid\":1,\"method\":\"get\",\"statusCode\":200,\"req\":{\"url\":\"/app/kibana\",\"method\":\"get\",\"headers\":{\"user-agent\":\"curl/7.29.0\",\"host\":\"localhost:5601\",\"accept\":\"*/*\"},\"remoteAddress\":\"127.0.0.1\",\"userAgent\":\"127.0.0.1\"},\"res\":{\"statusCode\":200,\"responseTime\":45,\"contentLength\":9},\"message\":\"GET /app/kibana 200 45ms - 9.0B\"}\n","stream":"stdout","time":"2019-08-12T20:06:43.183719829Z"}

Prettified data:

{
  "log": "{\"type\":\"response\",\"@timestamp\":\"2019-08-12T20:06:43Z\",\"tags\":[],\"pid\":1,\"method\":\"get\",\"statusCode\":200,\"req\":{\"url\":\"/app/kibana\",\"method\":\"get\",\"headers\":{\"user-agent\":\"curl/7.29.0\",\"host\":\"localhost:5601\",\"accept\":\"*/*\"},\"remoteAddress\":\"127.0.0.1\",\"userAgent\":\"127.0.0.1\"},\"res\":{\"statusCode\":200,\"responseTime\":45,\"contentLength\":9},\"message\":\"GET /app/kibana 200 45ms - 9.0B\"}\n",
  "stream": "stdout",
  "time": "2019-08-12T20:06:43.183719829Z"
}

One of the problems with this is that the log field has the log data with escaped " characters. This makes the inner json structure harder to prettify as one needs to take that in consideration when it is prettified.

Prettified data with unescaped ":

{  
  "log":{  
    "type":"response",
    "@timestamp":"2019-08-12T20:06:43Z",
    "tags":[  

    ],
    "pid":1,
    "method":"get",
    "statusCode":200,
    "req":{  
      "url":"/app/kibana",
      "method":"get",
      "headers":{  
        "user-agent":"curl/7.29.0",
        "host":"localhost:5601",
        "accept":"*/*"
      },
      "remoteAddress":"127.0.0.1",
      "userAgent":"127.0.0.1"
    },
    "res":{  
      "statusCode":200,
      "responseTime":45,
      "contentLength":9
    },
    "message":"GET /app/kibana 200 45ms - 9.0B"
  },
  "stream":"stdout",
  "time":"2019-08-12T20:06:43.183719829Z"
}

[milandamen]

The absence of this feature is the only reason we still use Kibana for reading logs. When this feature gets implemented it would mean we could use just Grafana, which would be great.

[slim-bean]

Would this issue also include a mechanism for selecting just certain fields out of the json object? Taking the above example being able to add to the query something like log.message and getting back just GET /app/kibana 200 45ms - 9.0B

[roidelapluie]

+1 @slim-bean would be awesome

[Fr33maan]

+1 Thie is actually pretty important for readability. I see that the TODO backlog is already pretty huge but hopes to see this landing soon

[jcdauchy]

Being table to select dispayed fields would be awesome. Don't know if this could be done also for Loki datasource ? One raison we cannot switch from Kibana to Explore/Logpanel on grafana is this missing fields selection feature :(

[Fr33maan]

I think that it would need a full text indexing which is not in the scope of Loki. When I upgraded to lastest grafana I finally get the json being pretty printed. Le dim. 19 janv. 2020 à 23:22, JC Dauchy <notifications@github.com> a écrit :

…

Being table to select dispayed fields would be awesome. Don't if this could be done also for Loki datasource ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#18522?email_source=notifications&email_token=AA2SDKONZ57UHPJSX7QB3E3Q6R43XA5CNFSM4ILEQHJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJKWE4I#issuecomment-576021105>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2SDKLP3S6TTJQPYPG2KM3Q6R43XANCNFSM4ILEQHJA> .

[jcdauchy]

I understand that loki uses only labels as "index". But why do you need to display all labels and fields as they seem to be already "parsed" as show in the details of each line ?

[Guchman]

There are two reasons why we still need to use Kibana:
selecting fields to display in log view and select fast filters by labels. It's really necessary to have before grafana will be good not only for metrics:(

[jcerrada]

+1 I'd be really good to have this feature.

[fredr]

Would this issue also include a mechanism for selecting just certain fields out of the json object? Taking the above example being able to add to the query something like log.message and getting back just GET /app/kibana 200 45ms - 9.0B

I made a POC/Proposal/discussion starter for something like that in pr #26178

[aocenas]

Also interesting issue with nested string fields that user also may want formatted #27964

[muhmud]

Any movement on this issue? It's makes debugging Java pretty painful...

[smbrd]

Lack of that feature is a deal breaker for our company to move from Sumo Logic.

[aciokler]

Any updates on this feature? This is really needed for our company. Reading java stack traces is almost impossible without some kind of formatting.

[zhagan]

I also need this ASAP!

[devnerCO]

Any updates on this? Reading Java logs in Grafana is not fun

[Paul414762]

Our team needs this! It's impossible to read the json output we have. We end up having to copy and paste the json into some other tool. We're wasting a ton of time and adding frustration!

[BitProcessor]

So how about this ?
Seems to have been on the roadmap for 7.0, we're at 7.4 now?
Roadmap: #15999

[Elfo404]

As a first step we can try to parse and prettify the first level of each log entry. We can then see what would be needed to better improve readability for log entries.

[ivanahuckova]

I think that we could, as a first step, just check if message is JSON.parsable. And if it is, we can use JSON.stringify(JSON.parse(message), undefined, 2) so it shows in the following way:

Not sure if any library would be faster than these native js functions. Also, not sure if we want to do it automatically for all parsable messages, or if we want to add button/switch.