Skip to content

[Improvement-5452][Task] ds flink task support submit a PyFlink job via the CLI#5453

Merged
CalvinKirs merged 4 commits intoapache:devfrom
zhuangchong:dev-fix-flink-python
Jun 17, 2021
Merged

[Improvement-5452][Task] ds flink task support submit a PyFlink job via the CLI#5453
CalvinKirs merged 4 commits intoapache:devfrom
zhuangchong:dev-fix-flink-python

Conversation

@zhuangchong
Copy link
Contributor

Purpose of the pull request

this close #5452

ds flink task support submit a PyFlink job via the CLI

$ ./bin/flink run --python examples/python/table/batch/word_count.py

@codecov-commenter
Copy link

codecov-commenter commented May 12, 2021

Codecov Report

❌ Patch coverage is 40.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.31%. Comparing base (9d70c7e) to head (80f62f0).
⚠️ Report is 3774 commits behind head on dev.

Files with missing lines Patch % Lines
.../dolphinscheduler/server/utils/FlinkArgsUtils.java 40.00% 1 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##                dev    #5453      +/-   ##
============================================
- Coverage     45.33%   45.31%   -0.02%     
+ Complexity     3674     3670       -4     
============================================
  Files           607      607              
  Lines         24780    24780              
  Branches       2801     2801              
============================================
- Hits          11234    11230       -4     
  Misses        12479    12479              
- Partials       1067     1071       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

50.0% 50.0% Coverage
0.0% 0.0% Duplication

public static final String FLINK_MAIN_CLASS = "-c";
public static final String FLINK_PARALLELISM = "-p";
public static final String FLINK_SHUTDOWN_ON_ATTACHED_EXIT = "-sae";
public static final String FLINK_CLI_OPTION_PYTHON = "-py";
Copy link
Member

@chengshiwen chengshiwen May 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think maybe FLINK_PYTHON is shorter and better

if (ProgramType.PYTHON == programType) {
// -py
args.add(Constants.FLINK_CLI_OPTION_PYTHON);
}
Copy link
Member

@chengshiwen chengshiwen May 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines should not place here. It's better to modify them to:

        if (programType != null) {
            if (programType == ProgramType.PYTHON) {
                args.add(Constants.FLINK_PYTHON);        //-py
            } else if (StringUtils.isNotEmpty(mainClass)) {
                args.add(Constants.FLINK_MAIN_CLASS);    //-c
                args.add(param.getMainClass());          //main class
            }
        }

@CalvinKirs CalvinKirs added bug Something isn't working enhancement New feature or request and removed bug Something isn't working labels May 18, 2021
@CalvinKirs
Copy link
Member

@zhuangchong Hi, how is it getting?

@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

72.7% 72.7% Coverage
0.0% 0.0% Duplication

@zhuangchong
Copy link
Contributor Author

@zhuangchong Hi, how is it getting?

done.

@CalvinKirs CalvinKirs requested a review from chengshiwen June 17, 2021 05:24
Copy link
Member

@chengshiwen chengshiwen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@CalvinKirs CalvinKirs merged commit 3026f04 into apache:dev Jun 17, 2021
@CalvinKirs CalvinKirs added this to the 1.3.7-release milestone Jun 17, 2021
@zhuangchong zhuangchong deleted the dev-fix-flink-python branch June 17, 2021 08:08
CalvinKirs added a commit that referenced this pull request Jul 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement][Task] ds flink task support submit a PyFlink job via the CLI

4 participants