[FLINK-31234][kubernetes] Add an option to redirect stdout to log dir. #22138

huwh · 2023-03-09T13:07:33Z

What is the purpose of the change

Add an option for redirecting stdout/stderr to files. When this is enabled.

flink-console.sh will redirect stdout/err to file.
flink-console.sh use log4j.properties as log4j configuration to avoid logs both in log file and stdout file.
Of course, this option is false by default.

Brief change log

introduce new option kubernetes.flink.std.redirect.enabled, default is false
when enabled, use log4j.properties/logback.xml as log configuration to avoid output log duplicated
when enabled, redirect stdout/stderr to log directory so that user can view these in Flink's web ui

Verifying this change

Added unit tests for FlinkConfMountDecoratorTest.java
Manually verified the change with kubernetes.flink.std.redirect.enabled is true or false.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)

flinkbot · 2023-03-09T13:12:42Z

CI report:

b20a894 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

wangyang0918

This PR looks pretty good to me. And I have verified with standalone mode. It works well. The stdout and stderr could be redirected to individual files. After then we could access the stdout in the Flink webUI.

echo "env.std.redirect: true" >>conf/flink-conf.yaml
./bin/jobmanager.sh start-foreground
./bin/taskmanager.sh start-foreground

wangyang0918 · 2023-06-28T07:03:25Z

flink-core/src/main/java/org/apache/flink/configuration/CoreOptions.java

+     */
+    @SuppressWarnings("unused")
+    public static final ConfigOption<Boolean> FLINK_STD_REDIRECT =
+            ConfigOptions.key("env.std.redirect")


Maybe the key name could be env.stdout-err.redirect-to-file. It is more explicit.

wangyang0918 · 2023-06-28T07:04:33Z

flink-dist/src/main/flink-bin/bin/flink-console.sh

 FLINK_ENV_JAVA_OPTS=$(eval echo ${FLINK_ENV_JAVA_OPTS})

+if [ "${STD_REDIRECT}" == "true" ]; then
+  # disable console appender to avoid redundant logs in log file and out file


We never have redundant logs in log file. Right?
Here is only to avoid the redundant logs in out file.

…files.

…err to files.

huwh · 2023-06-28T09:50:33Z

@wangyang0918 , Thanks for your review. comments addressed. And verified by running on Kubernetes.

wangyang0918

+1 for merging.

xiaows08 · 2024-09-29T10:55:34Z

flink-dist/src/main/flink-bin/bin/flink-console.sh


+if [ "${STD_REDIRECT_TO_FILE}" == "true" ]; then
+  # disable console appender to avoid redundant logs in out file
+  log_setting=("-Dconsole.log.level=OFF" "${log_setting[@]}")


进过验证-Dconsole.log.level=OFF 对于log4j来说似乎不能把appender.console的日志禁用🤔
那.out不就包含了log输出和sout吗，这样的话感觉 .log 和 .out两个文件没有同时存在的必要了。
或者三个文件同时存在的话，是不是可以把.out 移除Console的输出

Thanks for reporting this.

This option works fine for disabling console log output in my case.

Could you provide your log4j.properties file and the steps to reproduce this issue?

xiaows08 · 2024-10-09T08:10:36Z

@huwh 感谢回复，环境如下：
flink-kubernetes-operator 1.9.0
flink: flink:1.18.1-scala_2.12
log4j-console.properties 是镜像默认的

apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: basic
spec:
  image: flink:1.18.1-scala_2.12
  flinkVersion: v1_18
  ingress:
    className: "nginx"
    template: "{{name}}.fo.xyuqing.com" # 引号必须有
  podTemplate:
    spec:
      containers:
        - name: flink-main-container
          ports:
            - name: metrics
              containerPort: 9249
              protocol: TCP
          env:
            - name: TZ  # 设置容器运行的时区
              value: Asia/Shanghai
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
    env.stdout-err.redirect-to-file: "true"
    metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
  serviceAccount: flink
  jobManager:
    resource:
      cpu: 1
      memory: 2G
  taskManager:
    resource:
      cpu: 1
      memory: 2G
  job:
    jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
    parallelism: 2
    upgradeMode: stateless

我的意思是虽然 kubectl logs -f pods/xxx 不会打印log4j2的日志内容，但是对应的 xxx.out 文件还是包含有 log4j2 xxx.log的日志内容，最终导致代码中的log4j的输出会存在两份，也就会占两份存储，而且xxx.out 文件不能做到按大小或时间自动切分

huwh · 2024-10-12T08:02:51Z

@huwh 感谢回复，环境如下： flink-kubernetes-operator 1.9.0 flink: flink:1.18.1-scala_2.12 log4j-console.properties 是镜像默认的

apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: basic
spec:
  image: flink:1.18.1-scala_2.12
  flinkVersion: v1_18
  ingress:
    className: "nginx"
    template: "{{name}}.fo.xyuqing.com" # 引号必须有
  podTemplate:
    spec:
      containers:
        - name: flink-main-container
          ports:
            - name: metrics
              containerPort: 9249
              protocol: TCP
          env:
            - name: TZ  # 设置容器运行的时区
              value: Asia/Shanghai
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
    env.stdout-err.redirect-to-file: "true"
    metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
  serviceAccount: flink
  jobManager:
    resource:
      cpu: 1
      memory: 2G
  taskManager:
    resource:
      cpu: 1
      memory: 2G
  job:
    jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
    parallelism: 2
    upgradeMode: stateless

我的意思是虽然 kubectl logs -f pods/xxx 不会打印log4j2的日志内容，但是对应的 xxx.out 文件还是包含有 log4j2 xxx.log的日志内容，最终导致代码中的log4j的输出会存在两份，也就会占两份存储，而且xxx.out 文件不能做到按大小或时间自动切分

@xiaows08 This is because flink-kubernetes-operator uses its log4j-console.properties, which does not support the console appender filtering function.

Currently, flink-kubernetes-operator does not recommend enable env.stdout-err.redirect-to-file, which does cause log duplication.

flinkbot added the component=Deployment/Kubernetes label Mar 9, 2023

huwh force-pushed the huwh_redirects branch 3 times, most recently from 9c89077 to 60f39ee Compare June 20, 2023 03:57

huwh force-pushed the huwh_redirects branch from 60f39ee to fc2254a Compare June 26, 2023 10:03

wangyang0918 reviewed Jun 28, 2023

View reviewed changes

huwh added 2 commits June 28, 2023 15:59

[FLINK-31234][deployment] Add an option to redirect stdout/stderr to …

8187482

…files.

fixup! [FLINK-31234][deployment] Add an option to redirect stdout/std…

b20a894

…err to files.

huwh force-pushed the huwh_redirects branch from fc2254a to b20a894 Compare June 28, 2023 08:21

wangyang0918 approved these changes Jun 29, 2023

View reviewed changes

wangyang0918 merged commit dfe6bda into apache:master Jun 29, 2023

huwh deleted the huwh_redirects branch June 29, 2023 14:21

xiaows08 reviewed Sep 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-31234][kubernetes] Add an option to redirect stdout to log dir. #22138

[FLINK-31234][kubernetes] Add an option to redirect stdout to log dir. #22138

huwh commented Mar 9, 2023

Uh oh!

flinkbot commented Mar 9, 2023 •

edited

Loading

Uh oh!

wangyang0918 left a comment

Uh oh!

wangyang0918 Jun 28, 2023

Uh oh!

wangyang0918 Jun 28, 2023

Uh oh!

huwh commented Jun 28, 2023

Uh oh!

wangyang0918 left a comment

Uh oh!

xiaows08 Sep 29, 2024 •

edited

Loading

Uh oh!

huwh Oct 8, 2024

Uh oh!

xiaows08 commented Oct 9, 2024

Uh oh!

huwh commented Oct 12, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[FLINK-31234][kubernetes] Add an option to redirect stdout to log dir. #22138

[FLINK-31234][kubernetes] Add an option to redirect stdout to log dir. #22138

Conversation

huwh commented Mar 9, 2023

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

flinkbot commented Mar 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

wangyang0918 left a comment

Choose a reason for hiding this comment

Uh oh!

wangyang0918 Jun 28, 2023

Choose a reason for hiding this comment

Uh oh!

wangyang0918 Jun 28, 2023

Choose a reason for hiding this comment

Uh oh!

huwh commented Jun 28, 2023

Uh oh!

wangyang0918 left a comment

Choose a reason for hiding this comment

Uh oh!

xiaows08 Sep 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huwh Oct 8, 2024

Choose a reason for hiding this comment

Uh oh!

xiaows08 commented Oct 9, 2024

Uh oh!

huwh commented Oct 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

flinkbot commented Mar 9, 2023 •

edited

Loading

xiaows08 Sep 29, 2024 •

edited

Loading

huwh commented Oct 12, 2024 •

edited

Loading