Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

compute: tokenize error logging #18772

Merged
merged 6 commits into from
Apr 17, 2023
Merged

compute: tokenize error logging #18772

merged 6 commits into from
Apr 17, 2023

Commits on Apr 17, 2023

  1. compute: move MaybeValidatingRow into errors module

    @teskje
    teskje committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    16aa56c View commit details
    Browse the repository at this point in the history

  2. compute: introduce ErrorLogger for rendering 

    This commit introduces a new `ErrorLogger` type that will be used for
logging errors inside dataflows. This type wraps a dataflow shutdown
token and encapsules both shutdown-checked logging and our
`warn!`/`error!` approach to logging dataflow errors.
    @teskje
    teskje committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    8dd6e20 View commit details
    Browse the repository at this point in the history

  3. compute: add shutdown tokens to rendering context 

    This commit extends the rendering `Context` by a `shutdown_token` field
that can be used by rendered operators to check whether the dataflow is
in the progess of shutting down. The shutdown token is replaced for each
entry in `objects_to_build`, to support finer-grained dataflow shutdown
in a future where dataflows can have more than one export.

This commit also adjusts variable naming to make it clear that the
various `Context::export_*` methods receive a list of all dependency
IDs, which include `objects_to_build` IDs, rather than just IDs of
dataflow imports. As a consequence, there is no need to adjust the
export methods to ensure that they also pick up the new object tokens.
    @teskje
    teskje committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    2a9097f View commit details
    Browse the repository at this point in the history

  4. compute: don't log errors during dataflow shutdown 

    This commit modifies the error logging behavior of compute operators to
only log messages when the dataflow is not in the process of shutting
down. This avoids producing noise caused by dataflow operators emitting
partial update sets when they are cancelled.

The shutdown checking is performed through a shutdown token and wrapped
inside the new `ErrorLogger` type that gets passed around to the various
operator renderers.
    @teskje
    teskje committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    e469c82 View commit details
    Browse the repository at this point in the history

  5. compute: make more rendering functions Context methods 

    This commit converts all freestanding reduce and top-k rendering
functions that want to do error logging into methods of the `Context`
type. The benefit is that we can rid of two parameters, `debug_name` and
`error_logger`, in exchange for a new `&self` parameter, and changed
indentation.

Note that the top-k functions where previously definied inside the scope
of `Context::render_topk` (for no apparent reason). Making them methods
of `Context` actually reduces indentation.
    @teskje
    teskje committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    bcc7f8d View commit details
    Browse the repository at this point in the history

  6. compute: build dataflow name logging into ErrorLogger 

    Everywhere we log something in compute rendering, we include the debug
name of the dataflow in the logged details. It makes sense to build the
dataflow name logging directly into the `ErrorLogger`, so we don't have
to repeat ourselves at the callsites.
    @teskje
    teskje committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    05cd9ae View commit details
    Browse the repository at this point in the history