JSON and regex in format?

[ammsa]

I have a log file that is not a complete JSON field

[2017-06-22 17:32:22.000287] INFO [topic.views.post:81] [{'user': 'a', 'client_time': 1498152742245, 'result': {'message': 'Topics create page visit.'}]

The first three parts is the [timestamp] [LEVEL] [code location] [JSON message]

How would I create a format for such log file? The documentation mentions that if the line is JSON compatible, then you shouldn't specify a regex.

[tstack]

You can construct the regex as usual and set the "kind" for the JSON part of the message to "json". For example:

{
  "ammsa_log": {
    "title" : "Ammsa's log format",
    "description" : "Log format for issue #458",
    "url" : "https://github.com/tstack/lnav/issues/458",
    "regex" : {
        "basic" : {
          "pattern": "^\\[(?<timestamp>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{6})\\] (?<level>\\w+) \\[(?<code_ns>[^:]+):(?<code_line>\\d+)\\] (?<jmsg>.*)$"
        }
    },
    "level-field" : "level",
    "value" : {
      "code_line" : {
        "kind" : "integer",
        "foreign-key": true 
      },
      "jmsg" : {
        "kind" : "json"
      }
    },
    "sample" : [
      {
        "line": "[2017-06-22 17:32:22.000287] INFO [topic.views.post:81] [{\"user\": \"a\", \"client_time\": 1498152742245, \"result\": {\"message\": \"Topics create page visit.\"}}]"
      }
    ]

  }
}

Note that the log message you supplied does not have valid JSON since it uses single quotes instead of double and is missing a closing curly brace. I fixed this formatting error and things seem to work.

[ammsa]

Thanks! I've tried the format file you provided but I keep getting

invalid json -- parse error: client cancelled parse via callback return value
          g" : {         "kind" : "json"       }     },     "sample" :
                     (right here) ------^
error:ammsa_log:no sample logs provided, all formats must have samples

It seems to work when I change kind to something other than json.

[tstack]

What version of lnav are you using? I don't remember when the "json" kind was added, try upgrading to the latest (v0.8.2).

[ammsa]

Thanks! it worked. Is there a way to customize the colouring of level and jmsg? And is it possible to hide jmsg and only toggle it using a shortcut?

[thetooth]

You can hide jmsg by giving it the hidden property under values. Customising colors isn't really possible unless you want to recompile, if so all the coloring is done by the view_curses.cc file.

[sadv1r]

I have the same situation, but my json contains the main log information. Can i now get some values from this json?
Something like:

"value" : {
  "jmsg" : {
    "kind" : "json",
    "value" : {
      "name" : "/root/request/name"
    }
  }
}

Or any other solutions maybe?

[tstack]

@sadv1r Sorry, I'm not quite sure what you're trying to do. Are you trying to extract certain values from the embedded JSON for display instead of hiding the whole JSON blob?

[sadv1r]

Yes, you are right. I wanna extract some fields from json to show them like fields from regex pattern and hide other staff in this huge json.

If it's not possible, maybe I can transform log files to json somehow and exclude this part '[2017-06-22 17:32:22.000287] INFO [topic.views.post:81] [ ]' totally to discribe it with "json" : true ?
I mean, the main part of log is this inner json. The outer part is not so important