VDK Logging And Error Handling: Technical Analysis

Goals

• Create a high-level overview of existing code
• Identify potential areas of improvement for VDK logging and error handling

Logging

Logging configuration is implemented as a plugin. The plugin hooks into the vdk_configure and initialize_job methods. The vdk_configure hook configures the host, port, enabled flag and socket type for SYSLOG. The initialize_job hook fetches all the relevant job data and passes it to the configure_loggers method.

Line 207

configure_logs deals with more SYSLOG configuration and configures logging for different environments, e.g. CLOUD vs. LOCAL.

There is only one logging formatter which is hardcoded. It's used in all environments.

    DETAILED_FORMAT = (
        f"%(asctime)s [VDK] {job_name} [%(levelname)-5.5s] %(name)-30.30s %(filename)20.20s:%("
        f"lineno)-4.4s %(funcName)-16.16s[id:{attempt_id}]- %(message)s"
    )

Line 110

Line 148

Line 160

Error Handling

Error handling is built into VDK core.

https://github.com/vmware/versatile-data-kit/blob/main/projects/vdk-core/src/vdk/internal/core/errors.py

There are three types of errors thrown by VDK

• PLATFORM_ERROR - infrastructure errors
• USER_ERROR - errors in user code/configuration
• CONFIG_ERROR - errors in the configuration provided to VDK

Each error has a corresponding accountable

• PLATFORM_ERROR - should be fixed by the PLATFORM (SRE Team, Platform team, operating the infrastructure and services).
• USER_ERROR - should be fixed by the end USER (or data job owner), for example: supplied bad arguments, bug in user code.
• CONFIG_ERROR that occurred during:
  • platform run (in case the data job runs on platfrom infrastructure), is handled by the PLATFORM;
  • local run (in case the data job runs on local end user infrastructure), is handled by the USER.

Errors that are caught are wrapped in the Resolvable class. It describes who is responsible for handling the error and how it should be handled.

https://github.com/vmware/versatile-data-kit/blob/main/projects/vdk-core/src/vdk/internal/core/errors.py#L80

• resolvable_by: Indicates the resolvable type.
• resolvable_by_actual: Who is actually responsible for resolving it
• error_message: the error message
• exception: the exception related to the error
• resolved: indicate if the error is resolved (for example error may be handled in user code and they are considred resolved). It should be use for informative purposes. It may be None/empty (for example if error originates from a new thread spawned by a job step)

Note: A lot of the logic in errors.py that is not mentioned in this document has to do with determining who is accountable for resolving errors and ways to resolve them. This is helpful when logging and propagating errors to other components of the system, e.g. plugins, templates, user code.

Errors are reported by the following functions

log_exception

log_and_throw

log_and_rethrow

They share similar behaviour when it comes to logging.

They build the error message, which has the following format.

    error_message = __build_message_for_end_user(
        to_be_fixed_by,
        resolvable_by_actual,
        what_happened,
        why_it_happened,
        consequences,
        countermeasures,
    )

Then the error is pushed to the resolvable context and the method decides what to do with the underlying exception, e.g. throw a new one, re-throw or do nothing. The resolvable context is used to determine who is accountable for fixing the error.

TODO: Expand resolvable context description, sync with @tozka

Potential Areas of Improvement

Next Steps

1. Analyze existing customer data
2. Check for other areas of improvement based on data
3. Prioritize identified areas of improvement based on data.