VDK Logging And Error Handling: Technical Analysis

Goals

Logging

Logging configuration is implemented as a plugin. The plugin hooks into the vdk_configure and initialize_job methods. The vdk_configure hook configures the host, port, enabled flag and socket type for SYSLOG. The initialize_job hook fetches all the relevant job data and passes it to the configure_loggers method.

https://github.com/vmware/versatile-data-kit/blob/main/projects/vdk-core/src/vdk/internal/builtin_plugins/config/log_config.py#L207

configure_logs deals with more SYSLOG configuration and configures logging for different environments, e.g. CLOUD vs. LOCAL.

There is only one logging formatter and it's used in all environments.

    DETAILED_FORMAT = (
        f"%(asctime)s [VDK] {job_name} [%(levelname)-5.5s] %(name)-30.30s %(filename)20.20s:%("
        f"lineno)-4.4s %(funcName)-16.16s[id:{attempt_id}]- %(message)s"
    )

https://github.com/vmware/versatile-data-kit/blob/main/projects/vdk-core/src/vdk/internal/builtin_plugins/config/log_config.py#L110 https://github.com/vmware/versatile-data-kit/blob/main/projects/vdk-core/src/vdk/internal/builtin_plugins/config/log_config.py#L148 https://github.com/vmware/versatile-data-kit/blob/main/projects/vdk-core/src/vdk/internal/builtin_plugins/config/log_config.py#L160

Error Handling

Problem definition

Next Steps