Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add additional hdfs url check #31042

Merged
merged 3 commits into from
Nov 9, 2021
Merged

Add additional hdfs url check #31042

merged 3 commits into from
Nov 9, 2021

Conversation

Avogar
Copy link
Member

@Avogar Avogar commented Nov 3, 2021

Changelog category (leave one):

  • Bug Fix (user-visible misbehaviour in official stable or prestable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fix abort in debug server and DB::Exception: std::out_of_range: basic_string error in release server in case of bad hdfs url by adding additional check of hdfs url structure.

Detailed description / Documentation draft:
Such query let to abort in debug server and DB::Exception: std::out_of_range: basic_string error in release server:

SELECT * FROM hdfs('abcd', 'CSV', 'x UInt32')

@robot-clickhouse robot-clickhouse added the pr-bugfix Pull request with bugfix, not backported by default label Nov 3, 2021
@kssenii kssenii self-assigned this Nov 3, 2021
@Avogar Avogar merged commit 1708518 into ClickHouse:master Nov 9, 2021
Avogar added a commit that referenced this pull request Nov 10, 2021
Backport #31042 to 21.9: Add additional hdfs url check
Avogar added a commit that referenced this pull request Nov 10, 2021
Backport #31042 to 21.11: Add additional hdfs url check
Avogar added a commit that referenced this pull request Nov 10, 2021
Backport #31042 to 21.10: Add additional hdfs url check
Avogar added a commit that referenced this pull request Nov 18, 2021
Backport #31042 to 21.8: Add additional hdfs url check
@alpex29
Copy link

alpex29 commented Nov 29, 2021

@Avogar, @kssenii - by adding hdfs uri check a bug for valid hdfs uri's using HA namenode address is introduced. Currently used regex requires each uri to include ":portnumber" which in case of HA namenode address + hdfs-site.xml config is not required/doesn't work. Would you like me to open a bug report on the same? The issue is introduced in this commit + currently only present in 21.11.x release.

@Avogar
Copy link
Member Author

Avogar commented Nov 29, 2021

@Avogar, @kssenii - by adding hdfs uri check a bug for valid hdfs uri's using HA namenode address is introduced. Currently used regex requires each uri to include ":portnumber" which in case of HA namenode address + hdfs-site.xml config is not required/doesn't work. Would you like me to open a bug report on the same? The issue is introduced in this commit + currently only present in 21.11.x release.

Hi! Thanks for reporting it, I will fix it this week. No need to open an issue.

@itinycheng
Copy link

I have also encountered this issue, the exception:

Code: 36. DB::Exception: Received from localhost:9000. DB::Exception: Bad hdfs url: hdfs://hdfs@nameservice1/user/admin/name.csv. It should have structure 'hdfs://<host_name>:<port>/<path>'. (BAD_ARGUMENTS)

ClickHouse 21.11.2.2 didn't merge this commit, I temporarily use this version.

@Avogar Avogar mentioned this pull request Dec 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-bugfix Pull request with bugfix, not backported by default
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants