Skip to content

Improved logging output with timestamps, level settings and more#1326

Merged
lonvia merged 1 commit intoosm2pgsql-dev:masterfrom
joto:logging
Nov 18, 2020
Merged

Improved logging output with timestamps, level settings and more#1326
lonvia merged 1 commit intoosm2pgsql-dev:masterfrom
joto:logging

Conversation

@joto
Copy link
Copy Markdown
Collaborator

@joto joto commented Nov 17, 2020

This improves the logging output in several ways:

  • New logging functions log_debug/info/warn/error(). Almost all
    output now uses one of these functions.
  • Log messages start with timestamp.
  • Log messages of level warn/error are printed in red when
    logging to console.
  • New command line option: --log-level to set log level. Default
    is "info". The option -v, --verbose does the same as
    --log-level=debug.
  • New option --log-progress=true/false/auto. The default is auto
    which enables progress logging on the console and disables it
    when stderr has been redirected to a file.
  • Extra functions to log SQL commands/SQL COPY data. These can
    be activated by command line options --log-sql and --log-sql-data,
    respectively, instead of the compile time setting. Output is
    printed in blue when logging to console.

Most log messages are in level "info" at the moment, so they are shown
as before with the default settings. Many of these probably don't need
to be shown in normal operation and can be downgraded to "debug". I'll
leave this for a later commit.

There are several places in the code where the output is updated on the
screen while some longer process is running:

  • The code displaying regular updates of a counter in the expire code
    has been removed. Writing out the expire data doesn't take that long,
    so it isn't important to have this. Instead a log message is now
    generated.
  • The output of the objects being processed and the pending ways and
    relations output is now en/disabled by the --log-progress option.
    Otherwise it has not been touched. I'll leave cleaning that up
    for a later time.

Some error reporting code has also been changed. Usually errors should
be reported by throwing an exception that is then caught in the main()
function and written out. The exception should contain the whole
error message, sometimes additional logging lines are created before
throwing the error. There is probably some more cleanup needed here.

This introduces a new global object "the_logger". Otherwise we'd have to
add the logger as parameter to a lot of functions!

Fixes #202
Fixes #1246
Fixes #591

This improves the logging output in several ways:

* New logging functions log_debug/info/warn/error(). Almost all
  output now uses one of these functions.
* Log messages start with timestamp.
* Log messages of level warn/error are printed in red when
  logging to console.
* New command line option: --log-level to set log level. Default
  is "info". The option -v, --verbose does the same as
  --log-level=debug.
* New option --log-progress=true/false/auto. The default is `auto`
  which enables progress logging on the console and disables it
  when stderr has been redirected to a file.
* Extra functions to log SQL commands/SQL COPY data. These can
  be activated by command line options --log-sql and --log-sql-data,
  respectively, instead of the compile time setting. Output is
  printed in blue when logging to console.

Most log messages are in level "info" at the moment, so they are shown
as before with the default settings. Many of these probably don't need
to be shown in normal operation and can be downgraded to "debug". I'll
leave this for a later commit.

There are several places in the code where the output is updated on the
screen while some longer process is running:

* The code displaying regular updates of a counter in the expire code
  has been removed. Writing out the expire data doesn't take that long,
  so it isn't important to have this. Instead a log message is now
  generated.
* The output of the objects being processed and the pending ways and
  relations output is now en/disabled by the `--log-progress` option.
  Otherwise it has not been touched. I'll leave cleaning that up
  for a later time.

Some error reporting code has also been changed. Usually errors should
be reported by throwing an exception that is then caught in the main()
function and written out. The exception should contain the whole
error message, sometimes additional logging lines are created before
throwing the error. There is probably some more cleanup needed here.

This introduces a new global object "the_logger". Otherwise we'd have to
add the logger as parameter to a lot of functions!

Fixes osm2pgsql-dev#202
Fixes osm2pgsql-dev#1246
Fixes osm2pgsql-dev#591
@lonvia lonvia merged commit 440ee07 into osm2pgsql-dev:master Nov 18, 2020
@joto joto deleted the logging branch November 19, 2020 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add start and end times to the log add option to suppress progress log (-q --quiet) Use a logging library

2 participants