v0.8.3b

[build] fixes...

v0.8.3a: [index] do a full sort if new lines have a timestamp before the last …

…indexed

lnav v0.8.3

Features:

• Support for the Bro Network Security Monitor (https://www.bro.org) log file format.
• Added an fstat() table-valued function for querying the local filesystem.
• Added readlink() and realpath() SQL functions.
• Highlights specified in log formats can now specify the colors to use for the highlighted parts of the log message.
• Added a ":quit" command.
• Added a "/ui/default-colors" configuration option to specify that the terminal's default background and foreground colors should be used instead of black and white.

Interface Changes:

• Pressing delete at a command-prompt will exit the prompt if there is no other input.

Fixes:

• The help view now includes all the command-help that would pop up as you entered commands and SQL queries.
• Hidden fields and lines hidden before/after times are now saved in the current session and restored.
• Unicode characters should now be displayed correctly (make sure you have LANG set to a UTF-8 locale).

v0.8.3-alpha: [perf] avoid an initial forced index rebuild

  Defect Number:
    Reviewed By:
   Testing Done:

lnav v0.8.2

Features:

• The timestamp format for JSON log files can be specified with the
  "timestamp-format" option in the "line-format" array.
• Added "min-width", "max-width", "align", and "overflow" options to the
  "line-format" in format definitions for JSON log files. These options
  give you more control over how the displayed line looks.
• Added a "hidden" option to log format values so that you can hide JSON
  log fields from being displayed if they are not in the line format.
• Added a "rewriter" field to log format value definitions that is a
  command used to rewrite the field in the pretty-printed version of a
  log message. For example, the HTTP access log format will rewrite the
  status code field to include the textual version (e.g. 200 (OK)).
• Log message fields can now be hidden using the :hide-fields" command or
  by setting the 'hidden' property in the log format. When hidden, the
  fields will be replaced with a yellow ellipsis when displayed. Hiding
  large fields that contain extra details can make the log easier to read.
  The 'x' hotkey can be used to quickly toggle whether these fields are
  displayed or not.
• Added a ':mark' command to bookmark the top line in the current view.
• Added an ':alt-msg' command that can be used to set the text to be
  displayed in the bottom right of the command line. This command is
  mostly intended for use by hotkey maps to set the help text.
• In lnav scripts, the first row of a SQL query result will now be turned
  into local variables that can be referenced in other commands or
  queries. For example, the following script will print the number one:
  ;SELECT 1 as foobar
  :eval :echo ${foobar}
• Added an 'lnav_view_stack' SQL table that gives access to the view
  stack.
• Added a 'top_time' column to the lnav_views table so that you can get
  the timestamp for the top line in views that are time-based as well as
  allowing you to move the view to a given time with an UPDATE statement.
• Added a 'search' column to the lnav_views table so that you can perform
  a text search programmatically.
• Added a 'regexp_capture(, )' table-valued function for
  getting detailed results from matching a regular expression against a
  string.
• Added a 'timediff(, )' SQL function for computing the
  difference between two relative or absolute timestamps.
• Log formats can now define a default set of highlights with the
  "highlights" property.
• Added a '|search-for ' built-in script that can be used to
  start a search from the command-line.
• Log format definitions can now specify the expected log level for a
  sample line. This check should make it easier to validate the
  definition.

Interface Changes:

• Command and SQL documentation is now displayed in a section at the
  bottom of the screen when a command or query is being entered. Some
  commands will also display a preview of the command results. For
  example, the ':open' command will display the first ten lines of the
  file to be opened and the ':filter-out' command will highlight text
  that matches in the current view. The preview pane can be shown/hidden
  by pressing CTRL-P.
• The color used for text colored via ":highlight" is now based on the
  the regex instead of randomly picked so that colors are consistent
  across invocations.
• The "graph" view has been removed since it's functionality has been
  obsoleted by other features, like ":create-search-table".
• When doing a search, if a hit is found within a second after hitting
  , the view will move to the matched line. The previous behavior
  was to stay on the current line, which tended to be a surprise to new
  users.
• Pressing 'n'/'N' to move through the next/previous search hit will now
  skip adjacent lines, up to the vertical size of the view. This should
  make scanning through clusters of hits much faster. Repeatedly
  pressing these keys within a short time will also accelerate scanning
  by moving the view at least a full page at a time.

Breaking Changes:

• The captured timestamp text in log files must fully match a known format
  or an error will be reported. The previous behavior was to ignore any
  text at the end of the line.

Fixes:

• You can now execute commands from the standard input by using a dash (-)
  with the '-f' command-line argument. Reading commands from a file
  descriptor should also work, for example, with the following bash
  syntax:

    $ lnav -f <(echo :open the-file-to-open)

• Programming language syntax highlighting should now only be applied to
  source code files instead of everywhere.

v0.8.2-alpha

[log_vtab_manager] fix lookup_impl()

lnav v0.8.1

Features:

• Added a spectrogram command and view that displays the values of a
  numeric field over time. The view works for log message fields or
  for database result columns.
• Log formats can now create SQL views and execute other statements
  by adding '.sql' files to their format directories. The SQL scripts
  will be executed on startup.
• Added 'json_group_object' and 'json_group_array' aggregate SQL
  functions that collects values from a GROUP BY query into a JSON
  object or array, respectively.
• The SQL view will now graph values found in JSON objects/arrays in
  addition to the regular columns in the result.
• Added an 'regexp_match(, )' SQL function that can be used to
  extract values from a string using a regular expression.
• Added an 'extract()' SQL function that extracts values using the
  same data discover/extraction parser used in the 'logline' table.
• Added a "summary" overlay line to the bottom of the log view that
  displays how long ago the last message was received, along with the
  total number of files and the error rate over the past five minutes.
• Pressing 'V' in the DB view will now check for a column with a
  timestamp and move to the corresponding time in the log view.
• Added 'a/A' hotkeys to restore a view previously popped with 'q/Q'.
• Added ":hide-lines-before", ":hide-lines-after", and
  ":show-lines-before-and-after" commands so that you can filter out
  log lines based on time.
• Scripts containing lnav commands/queries can now be executed using
  the pipe ('|') hotkey. See the documentation for more information.
• Added an ":eval" command that can be used to execute a command or
  query after performing environment variable substitution.
• Added an ":echo" command that can be useful for scripts to message
  the user.
• The "log_part" column can now be set with an SQL UPDATE statement.
• Added a "log_body" hidden column that returns the body of the log
  message.
• Added ":config", ":reset-config", and ":save-config" commands to change
  configuration options, reset to default, and save them for future
  executions.
• Added a "/ui/clock-format" configuration option that controls the time
  format in the top-left corner.
• Added a "/ui/dim-text" configuration option that controls the brightness
  of text in the UI.
• Added support for TAI64 timestamps (http://cr.yp.to/libtai/tai64.html).
• Added a safe execution mode. If the 'LNAVSECURE' environment variable is
  set before executing lnav, the following commands are disabled:
  • open
  • pipe-to
  • pipe-line-to
  • write-*-to
    This makes it easier to run lnav with escalated privileges in restricted
    environments, without the risk of users being able to use the above
    mentioned commands to gain privileged access.

Interface Changes:

• The 'o/O' hotkeys have been reassigned to navigate through log
  messages that have a matching "opid" field. The old action of
  moving forward and backward by 60 minutes can be simulated by
  using the ':goto' command with a relative time and the 'r/R'
  hotkeys.
• Log messages with timestamps that pre-date previous log messages will
  have the timestamp highlighted in yellow and underlined. These out-
  of-time-order messages will be assigned the time of the previous
  message for sorting purposes. You can press the 'p' hotkey to examine
  the 'Received Time' of the message as well as the time parsed from the
  original message. A "log_actual_time" hidden field has also been
  added to the SQLite virtual table so you can operate on the original
  message time from the file.
• The 'A/B' hotkeys for moving forward/backward by 10% line increments
  have been reassigned to '[' and ']'. The 'a' and 'A' hotkeys are now
  used to return to the previously popped view while trying to preserve
  the time range. For example, after leaving the spectrogram view with
  'q', you can press 'A' return to the view with the top time in the
  spectrogram matching the top time in the log view.
• The 'Q' hotkey now pops the current view off of the stack while
  maintaining the top time between views.

Fixes:

• Issues with tailing JSON logs have been fixed.
• The jget() SQL function should now work for objects nested in arrays.

lnav v0.8.0

Features:

• Integration with "papertrailapp.com" for querying and tailing
  server log and syslog messages. See the Papertrail section in
  the online help for more details.
• Remote files can be opened when lnav is built with libcurl v7.23.0+
• SQL queries can now be done on lines that match a regular expression
  using the "log_search" table or by creating custom tables with the
  ":create-search-table" command.
• Log formats that are "containers" for other log formats, like
  syslog, are now supported. See the online help for more
  information.
• Formats can be installed from git repositories using the '-i' option.
  A standard set of extra formats can be installed by doing
  'lnav -i extra'. (You must have git installed for this to work.)
• Added support for 'VMware vSphere Auto Deploy' log format.
• Added a 'sudo' log format.
• Added hotkeys to move left/right by a smaller increment (H/L or
  Shift+Left/Shift+Right).
• A color-coded bar has been added to the left side to show where
  messages from one file stop and messages from another file start.
• The '-C' option will now try to check any specified log files to
  make sure the format(s) match all of the lines.
• Added an "all_logs" SQLite table that contains the message format
  extracted from each log line. Also added a ';.msgformat' SQL command
  that executes a query that returns the counts for each format and the
  first line where the format was seen.
• Added an "lnav_views" SQLite table that can be used to query and
  change the lnav view state.
• When typing in a command, the status bar will display a short
  summary of the currently entered command.
• Added a "delete-filter" command.
• Added a "log_msg_instance" column to the logline and log_search
  tables to make it easier to join tables that are matching log
  messages that are ordered.
• Added a "timeslice()" function to SQLite so that it is easier to
  group log messages by time buckets.
• The 'goto' command now supports relative time values like
  'a minute ago', 'an hour later', and many more.

Interface Changes:

• The 'r/R' hotkeys have been reassigned to navigate through the log
  messages by the relative time value that was last used with the
  'goto' command.

Fixes:

• The pretty-print view should now work for text files.
• Nested fields in JSON logs are now supported for levels, bodies, etc...
• Tab-completion should work for quoted SQL identifiers.
• 'lo-fi' mode key shortcut changed to CTRL+L.
• 'redraw' shortcut removed. Relegated to just a command.
• Fixed lnav hang in pretty-print mode while doing a dns lookup.
• The generic log message parser used to extract data has been
  optimized and should be a bit faster.

lnav v0.7.3

Features:

• Add 'pipe-to' and 'pipe-line-to' commands that pipe the currently
  marked lines or the current log message to a shell command,
  respectively.
• Added a "pretty-print" view (P hotkey) that tries to reformat log
  messages so that they are easier to read.
• Added a 'redraw' command (CTRL+L hotkey) to redraw the window in
  case it has been corrupted.
• Added a 'relative-goto' command to move the current view relative
  to its current position.
• Experimental support for linking with jemalloc.
• The plain text view now supports filtering.
• Added 'next-mark' and 'prev-mark' commands to jump to the next or
  previous bookmarked line (e.g. error, warning, ...)
• Added a 'zoom-to' command to change the zoom level of the histogram
  view.
• Log formats can now define their own timestamp formats with the
  'timestamp-format' field.

Fixes:

• Autotools scripts overhaul.
• Added a configure option to disable linking with libtinfo. The newer
  versions of ncurses don't require it, however the build silently pulls
  it in as a dependency, if it is available on the system. This can be
  explicitly disabled using the '--disable-tinfo' option during configure.
• Fixed the configure script behavior to ignore the values specified using
  the CFLAGS and LDFLAGS environment variables while searching for sqlite3
  when '--with-sqlite3' switch was specified without the prefix.
• The configure script now recognizes libeditline symlink'ed to masquerade
  as libreadline. This previously used to cause problems at compile time,
  specially on OS X. If you come across this error, use the
  '--with-readline=prefix' switch to specify the path to the correct
  location of libreadline.
• The order that log formats are tried against a log file is now
  automatically determined so that more specific formats are tested
  before more general ones. The order is determined on startup based on
  how each format matches each other formats sample lines.
• Command files (i.e. those executed via the '-f' flag) now support
  commands/queries that span more than one line.
• Added more log levels: stats, debug2 - debug5.

lnav v0.7.2

Features:

• Added log formats for vdsm, openstack, and the vmkernel.
• Added a "lo-fi" mode (L hotkey) that dumps the displayed log lines
  to the terminal without any decorations. The write-to, write-json-to,
  and write-csv-to commands will also write their output to the terminal
  when passed '-' as the file name. This mode can be useful for copying
  plain text lines to the clipboard.
• (OS X) Text search strings are copied to the system's "find" clipboard.
  Also, when starting a new search, the current value in the "find"
  clipboard can be tab-completed.