Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improve][HDFS][#1336] Fix the path problem of writing to HDFS. #1337

Merged
merged 1 commit into from
Oct 24, 2022
Merged

[Improve][HDFS][#1336] Fix the path problem of writing to HDFS. #1337

merged 1 commit into from
Oct 24, 2022

Conversation

zhengchaoken
Copy link
Contributor

@zhengchaoken zhengchaoken commented Oct 23, 2022 •
edited

Purpose of this pull request

修复写入HDFS后的路径问题。使本地调试时能正常写hdfs。

Which issue you fix

Fixes # (1336).

Checklist:

  • I have executed the 'mvn spotless:apply' command to format my code.
  • I have a meaningful commit message (including the issue id, the template of commit message is '[label-type-#issue-id][fixed-module] a meaningful commit message.')
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have checked my code and corrected any misspellings.
  • My commit is only one. (If there are multiple commits, you can use 'git squash' to compress multiple commits into one.)

@zhengchaoken
Copy link
Contributor Author

本地调试，没有单元测试用例

FlechazoW
FlechazoW reviewed Oct 24, 2022
View reviewed changes
...nnector-hdfs/src/main/java/com/dtstack/chunjun/connector/hdfs/sink/BaseHdfsOutputFormat.java
@@ -119,6 +118,12 @@ protected void checkOutputDir() {
}
}

/** 文件分隔符(File.separatorChar)在windows为\，而在linux中为/，在hadoop中路径需要固定为/， */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you comment in English?

FlechazoW
FlechazoW reviewed Oct 24, 2022
View reviewed changes
...nnector-hdfs/src/main/java/com/dtstack/chunjun/connector/hdfs/sink/BaseHdfsOutputFormat.java
@@ -119,6 +118,12 @@ protected void checkOutputDir() {
}
}

/** 文件分隔符(File.separatorChar)在windows为\，而在linux中为/，在hadoop中路径需要固定为/， */
protected char getHdfsPathChar() {
// hadoop 文件系统固定为/，避免路径不对，文件写入错误及移动失败
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

@FlechazoW FlechazoW changed the title fix BUG #1336,修复写入HDFS后的路径问题。 [Improve][HDFS][#1336] Fix the path problem of writing to HDFS. Oct 24, 2022
@FlechazoW FlechazoW linked an issue Oct 24, 2022 that may be closed by this pull request
[Bug] [chunjun-connector-hdfs] hdfswriter路径写死，在windows debug时写入目录路径错误 #1336
Closed
3 tasks
@FlechazoW
Copy link
Member

+1

@FlechazoW FlechazoW merged commit a62b374 into DTStack:master Oct 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FlechazoW FlechazoW FlechazoW left review comments

Assignees

@zhengchaoken zhengchaoken

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] [chunjun-connector-hdfs] hdfswriter路径写死，在windows debug时写入目录路径错误
2 participants
@zhengchaoken @FlechazoW