Logging

Logging

Logging setup

• Each module needs to have logger initiated

import logging

logger = logging.getLogger(__name__)

• Modules themselves shall not configure logger in the main code, for testing it is recommended to configure logger in the end of the module

if __name__ == "__main__":
    import sys
    logging.basicConfig(
        format='%(levelname)-10s %(asctime)s.%(msecs)03d %(name)-30s %(funcName)-35s %(lineno)-5d: %(message)s',
        datefmt='%Y-%m-%dT%H:%M:%S',
        level=logging.INFO,
        handlers=[logging.StreamHandler(sys.stdout)]
    )

• Main services and processes should use custom logger class and initiate logging as follows

TODO - update to more relevant example

if __name__ == '__main__':

    time_horizon = settings.cgm_time_horizon
    scenario_date = settings.cgm_scenario_date
    version = settings.cgm_version

    import custom_logger

    custom_logger.customConfig(elk_server=settings.elk_server,
                               index=settings.elk_logging_index,
                               extra={"time_horizon": time_horizon,
                                      "scenario_date": scenario_date,
                                      "version": version,
                                      "merge_type": settings.cgm_area,
                                      "task_id": str(uuid4())})

It is possible to add additional metadata to custom_logger:

1. On initialization, pass dictionary to extra [example: Above]
2. Once initialised [example: log_handler.extra["scenario_date"] = scenario_date.strftime("%Y-%m-%dT%H:%M")]
3. On each log entry, passing dictionary to extra

Business process monitoring

Business metadata

All metadata from task object should be added to the logging as extra

Lifecycle monitoring

• process_id:

  • persistent in time and fixed in settings
  • Definition: Process is the root structure and common denominator for configuration activity
  • Example: CGM Creation

• run_id:

  • persistent in time and fixed in settings
  • Example: D-1
  • Purpose: Aggregate all logs pertaining to a run within a process

• job_id:

  • generated on the go, UUID4.
  • Definition: Job is a concrete instance of a run.
  • Example: D-1 run for a given day is a job.
  • Purpose: Aggregate all logs pertaining to an instance of a run of a specific period

• task_id:

  • generated on the go, UUID4.
  • Example: All activity for a single Scenario Time within a given job is a task
  • Purpose: Get all logs for a specific task
  • TODO: Should we have parent_task_id(s) to link tasks with each other (replicate OpenTelemetry span and parent span logic)