Skip to content

[Enhancement] Add MDC log traces #62394

@zhiqiang-dev

Description

@zhiqiang-dev

Search before asking

  • I had searched in the issues and found no similar issues.

Description

Image

How to use Flink MDC(jobid)
https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/logging/

Flink adds the following fields to [MDC](https://www.slf4j.org/api/org/slf4j/MDC.html) of most of the relevant log messages (experimental feature):

Job ID
key: flink-job-id
format: string
length 32
This is most useful in environments with structured logging and allows you to quickly filter the relevant logs.

The MDC is propagated by slf4j to the logging backend which usually adds it to the log records automatically (e.g. in [log4j2 json layout](https://logging.apache.org/log4j/2.x/manual/json-template-layout.html#event-template-resolver-mdc).

Log4j 2 JsonTemplateLayout [#](https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/logging/#log4j-2-jsontemplatelayout)
JsonTemplateLayout is a customizable, efficient, and garbage-free JSON generating layout. It encodes LogEvents according to the structure described by the JSON template provided.

The required jar log4j-layout-template-json is bundled in the flink-dist for convenience.

For example templates, see the [Event Templates](https://logging.apache.org/log4j/2.x/manual/json-template-layout.html#event-templates).

Log4j 2 PatternLayout [#](https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/logging/#log4j-2-patternlayout)
Alternatively, it can be configured explicitly - [log4j pattern layout](https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html) might look like this:

[%-32X{flink-job-id}] %c{0} %m%n.

Solution

MdcUtils.wrapRunnable( obid, () -> {} );
        scheduledExecutorService.scheduleWithFixedDelay(
                MdcUtils.wrapRunnable(MdcUtils.asContextData(jobId), this::intervalFlush),
                flushIntervalMs,
                flushIntervalMs,
                TimeUnit.MILLISECONDS);

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions