[Improvement][Task] Support end-to-end transfer file between tasks#13343
[Improvement][Task] Support end-to-end transfer file between tasks#13343Radeity wants to merge 9 commits intoapache:devfrom
Conversation
Radeity
requested review from
EricGao888,
SbloodyS,
Tianqi-Dotes,
caishunfeng,
ruanwenjun and
zhongjiajie
as code owners
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
...er/src/main/java/org/apache/dolphinscheduler/server/worker/utils/TaskFilesTransferUtils.java
Fixed
Show fixed
Hide fixed
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/FileUtils.java
Fixed
Show fixed
Hide fixed
|
|
||
| boolean isZip = downloadPath.endsWith(PACK_SUFFIX + TEMPLATE_SUFFIX); | ||
|
|
||
| String execCommand = String.format(commandString, downloadPath); |
Check warning
Code scanning / CodeQL
Unused format argument
Codecov Report
@@ Coverage Diff @@
## dev #13343 +/- ##
============================================
- Coverage 39.44% 39.42% -0.02%
- Complexity 4307 4313 +6
============================================
Files 1083 1085 +2
Lines 40738 40866 +128
Branches 4669 4681 +12
============================================
+ Hits 16069 16112 +43
- Misses 22884 22962 +78
- Partials 1785 1792 +7
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
SonarCloud Quality Gate failed. |
| try { | ||
| File delDir = new File(tmpDir); | ||
| String parentPath = delDir.getParent(); | ||
| org.apache.commons.io.FileUtils.deleteDirectory(delDir); | ||
| FileUtils.deleteEmptyParentDir(parentPath); | ||
| logger.info("Success clear the tmp dir: {}.", tmpDir); | ||
| } catch (IOException e) { | ||
| logger.error("Tmp dir clear failed!"); | ||
| } |
There was a problem hiding this comment.
Have to consider about the scenario of using task cache mechanism, upload to resource center, rather simply delete.
| if (isScpCommandTemplate) { | ||
| isPack = scpFetchFile(downloadPath, targetPath); | ||
| } |
There was a problem hiding this comment.
If upstreaming task uses cache mechanism, scp fetch will fail, however, transferred files have been uploaded to resource center. Try to remove template suffix of resource path and download them.
|
This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs. |
|
This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request. |
3 participants
Purpose of the pull request
Brief change log
Add configuration
tmp.transfer.file.size, default size 100MB.Add
TmpDirClearProcessorto clean temporary directory after process is finished.If usage of temporary storage doesn't reach the limit, transfer files will be stored in local temporary path:
DATA_BASEDIR/tmp/{tenandCode}/{projectCode}/{processDefineCode}/{processDefineVersion}/{processInstanceId}If downloaded file is scp command template file(with
.templatesuffix), complete and execute scp command to fetch files from upstream worker.Verify this pull request
Add UT for
FileUtilsand exist UT covers changes inTaskFileTransferUtils.Manually test scp way can work in standalone mode. However, it has to be verified that deploy DS in cluster mode and k8s mode, can workers communicate with each other without password? Otherwise, scp command will not execute successfully.
Log in upstream task:
Log in downstream task: