[bug] flink args build problem #3457

island-lonely · 2020-08-10T12:56:16Z

Describe the bug
There was a BUG in the Dolphin Scheduler-1.3.1 that Dolphin set Flink launch parameters

To Reproduce

If the slot number and taskManager number are set, Jar package cannot be found when flink task is executed
Among other parameters, specifying -yqu (the queue name of Flink on Yarn) or -ynm (Flink appName) is invalid

！！！！ Setting other boot parameters, such as -yqu and -ynm, can cause confusion 2020-08-10 21:03:31.400 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend 2020-08-10 21:03:31.401 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend 2020-08-10 21:03:31.401 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.401 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.401 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.401 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend 2020-08-10 21:03:31.401 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.401 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.401 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.401 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.401 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.401 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.401 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.402 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.402 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.402 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.402 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.402 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.402 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.402 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.402 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.402 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - 2020-08-10 21:03:31.402 10.15.1.19 [main] INFO org.apache.flink.client.cli.CliFrontend - in the order in which the flink boot parameters are set
- Hadoop version: 3.0.0-cdh6.3.2
- JVM Options:
-Dlog.file=/opt/flink-1.10.1/log/flink-dscheduler-client-cdh-05.log
-Dlog4j.configuration=file:/opt/flink-1.10.1/conf/log4j-cli.properties
-Dlogback.configurationFile=file:/opt/flink-1.10.1/conf/logback.xml
- Program Arguments:
run
-m
yarn-cluster
-yjm
1G
-ytm
6G
-d
-c
cn.~~.analysis.DurationAndMileage
analysis-assembly-2.3.jar
--qu
default
-yqu
test
-ynm
DurationAndMileage

yangyichao-mango · 2020-08-11T02:31:38Z

Hi,
1.Can you successfully run in your env without using DolphinScheduler?
2.It will be better to add a screenshot to help us to locate this problem.

1763734039 · 2020-08-11T03:14:46Z

I also encountered the same problem. If you want to specify other parameters, they will not take effect. I added them to other parameters. Then look at the worker log information, the Flink command will be spliced as follows: Flink run - M yarn Cluster - ys 2 - YJM 2 - YTM 8 - D - C com.xxx.common . task.xxx flink/xx-xxx-xx-1.0- SNAPSHOT.jar --Qu default - P 10, so the parameter will not take effect, Thank you for your reply to solve this bug

island-lonely · 2020-08-11T06:30:54Z

Hi,
1.Can you successfully run in your env without using DolphinScheduler?
2.It will be better to add a screenshot to help us to locate this problem.

Flink can run normally in my working environment

1、when you set the first parameter in the image，Jar package cannot be found when flink task is executed

2、When setting the second parameter in the picture，parameter will not take effect

yangyichao-mango · 2020-08-12T07:51:40Z

From flink params source processing code, 其他参数 will be add after the jar, so it will not take effect, @nauu please confirm the usage of 其他参数. Can we change the order of 其他参数 in front of jar?

island-lonely · 2020-08-12T08:25:16Z

From flink params source processing code, 其他参数 will be add after the jar, so it will not take effect, @nauu please confirm the usage of 其他参数. Can we change the order of 其他参数 in front of jar?

In the first parameter of the picture
Flink 1.10
Remove the --yn/-- yarnContainer command parameter
Remove the -- yST /--yarnstreaming command parameter

qiaozhanwei · 2020-10-14T10:10:10Z

flink run [OPTIONS] jar-file arguments . flink params sequence
1 . DS can set queue instead of other params set
2.
for example
flink run -m yarn-cluster -ys 1 -yn 2 -yjm 1G -ytm 2G -d -c com.journey.flink.wc.StreamWordCount flink-demo-1.0-SNAPSHOT-jar-with-dependencies.jar --qu default -yqu test -ynm testappl

other prama can follow run . but Different versions of flink version parameters are different . so add a version control or other ?

chengshiwen · 2020-12-08T11:17:09Z

@islandgit In pr #4166, I have fixed this issue.

Lucaszlei · 2021-01-12T12:31:24Z

The problem was listed in the 1.3.4-release notes, and the problem was closed via 4166, but 4166 did not merge into the 1.3.4-release node.

这个问题被列在了1.3.4-release的notes中，并且这个问题通过4166被关闭，但是4166并没有合入1.3.4-release的分支呀，这个怎么解释呢？

Lucaszlei · 2021-01-12T12:39:47Z

The problem was listed in the 1.3.4-release notes, and the problem was closed via 4166, but 4166 did not merge into the 1.3.4-release notes.

这个问题被列在了1.3.4-release的notes中，并且这个问题通过4166被关闭，但是4166并没有合入1.3.4-release的分支呀，这个怎么解释呢？

@chengshiwen 您好，是否可以回复下此问题，有点疑惑

chengshiwen · 2021-01-12T13:24:30Z

@chuyang-zl Thank you for your feedback, the pr #4166 fixed the issue #3457 has merged into dev, but not cherry-picked into the 1.3.4-release due to negligence

Lucaszlei · 2021-01-13T02:03:43Z

Is there a definite release date for 1.3.5?

1.3.5-release有确定发布时间计划吗？

dcy652701 · 2021-01-13T07:47:16Z

I am a little confused about the '--qu'. what does it means? My flink program can't deal with '--qa'. How do I get rid of this parameter？

chengshiwen · 2021-01-13T12:16:22Z

@dcy652701 The option --qu is wrong, while the correct one should be -yqu which means the flink queue on yarn. Please see details in #4166
The option --qa should be invalid. If you want use custom options in flink, please set the options in other parameters based on next release 1.3.5.

chengshiwen · 2021-01-13T12:20:49Z

@chuyang-zl It shouldn't be too long.

island-lonely added the bug Something isn't working label Aug 10, 2020

xingchun-chen added this to the 1.3.3-release milestone Sep 3, 2020

xingchun-chen assigned davidzollo Sep 3, 2020

davidzollo assigned qiaozhanwei and unassigned davidzollo Oct 14, 2020

lgcareer modified the milestones: 1.3.3-release, 1.4.0 Oct 23, 2020

chengshiwen mentioned this issue Dec 8, 2020

[Fix-3457][flink] fix flink args build problem #4166

Merged

CalvinKirs linked a pull request Dec 8, 2020 that will close this issue

[Fix-3457][flink] fix flink args build problem #4166

Merged

davidzollo closed this as completed in #4166 Dec 10, 2020

CalvinKirs changed the title ~~Dolphin Scheduler 执行Flink 出现问题~~ [bug] flink args build problem Jan 12, 2021

xingchun-chen modified the milestones: 1.4.0, 1.3.5-release Jan 12, 2021

CalvinKirs mentioned this issue Jan 28, 2021

[1.3.5-prepare][cherry-pick]#4166 #4290(backend) flink args build problem bug #4600

Closed

chengshiwen mentioned this issue Jan 28, 2021

[1.3.5-prepare][cherry-pick]#4166 flink args build problem bug and support flink 1.10 or newer #4611

Merged

chengshiwen mentioned this issue Apr 29, 2021

[Question] fail to submit flink Job #5404

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bug] flink args build problem #3457

[bug] flink args build problem #3457

island-lonely commented Aug 10, 2020 •

edited

yangyichao-mango commented Aug 11, 2020

1763734039 commented Aug 11, 2020

island-lonely commented Aug 11, 2020

yangyichao-mango commented Aug 12, 2020

island-lonely commented Aug 12, 2020

qiaozhanwei commented Oct 14, 2020 •

edited

chengshiwen commented Dec 8, 2020 •

edited

Lucaszlei commented Jan 12, 2021 •

edited

Lucaszlei commented Jan 12, 2021 •

edited

chengshiwen commented Jan 12, 2021 •

edited

Lucaszlei commented Jan 13, 2021

dcy652701 commented Jan 13, 2021

chengshiwen commented Jan 13, 2021 •

edited

chengshiwen commented Jan 13, 2021

[bug] flink args build problem #3457

[bug] flink args build problem #3457

Comments

island-lonely commented Aug 10, 2020 • edited

yangyichao-mango commented Aug 11, 2020

1763734039 commented Aug 11, 2020

island-lonely commented Aug 11, 2020

yangyichao-mango commented Aug 12, 2020

island-lonely commented Aug 12, 2020

qiaozhanwei commented Oct 14, 2020 • edited

chengshiwen commented Dec 8, 2020 • edited

Lucaszlei commented Jan 12, 2021 • edited

Lucaszlei commented Jan 12, 2021 • edited

chengshiwen commented Jan 12, 2021 • edited

Lucaszlei commented Jan 13, 2021

dcy652701 commented Jan 13, 2021

chengshiwen commented Jan 13, 2021 • edited

chengshiwen commented Jan 13, 2021

island-lonely commented Aug 10, 2020 •

edited

qiaozhanwei commented Oct 14, 2020 •

edited

chengshiwen commented Dec 8, 2020 •

edited

Lucaszlei commented Jan 12, 2021 •

edited

Lucaszlei commented Jan 12, 2021 •

edited

chengshiwen commented Jan 12, 2021 •

edited

chengshiwen commented Jan 13, 2021 •

edited