Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generate decent error messages for Hadoop, Hive misconfigurations #540

Closed
joohokim1 opened this issue Oct 24, 2018 · 2 comments
Closed

Generate decent error messages for Hadoop, Hive misconfigurations #540

joohokim1 opened this issue Oct 24, 2018 · 2 comments
Assignees
@paigechoi @sang-hee-lee @joohokim1
Labels
@dataprep Component Name : Data preparation enhancement Request Change and Feature Enhancement
Milestone
3.2.0

Comments

@joohokim1
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Currently, if Hadoop or Hive setting is wrong, it is rather hard to find the cause.
In the cases we need the Hadoop or Hive configurations, we have to do some checks before creating a full snapshot to help users get informed the cause of the error, accurately and as quickly as possible.

It would be nice to inform the followings before creating a snapshot.

  • Whether core-site.xml was found in hadoopConfDir
  • Can write to stagingBaseDir?
  • Whether we can connect to HiveServer2

현재 Hadoop, Hive 설정이 잘못되어있는 경우, 그 원인을 찾는 것이 녹록치 않습니다.
Hadoop, Hive에 대한 설정을 필요로 하는 경우, 본격적인 snapshot 생성 작업 전에 몇 가지 체크를 해서, 사용자가 오류의 원인을 정확하게, 또 가능한한 빨리 파악할 수 있도록 지원하는 것이 필요합니다.

Describe the solution you'd like
대략 아래와 같은 내용들을 스냅샷 생성 이전에 알려주면 도움이 될 것 같습니다.

  • hadoopConfDir에 core-site.xml을 찾을 수 있었는지 여부
  • stagingBaseDir에 write를 할 수 있는지 여부
  • HiveServer2에 접속할 수 있는지 여부
@joohokim1 joohokim1 added enhancement Request Change and Feature Enhancement @dataprep Component Name : Data preparation labels Oct 24, 2018
@joohokim1 joohokim1 added this to the 3.1.0 milestone Oct 24, 2018
@joohokim1 joohokim1 self-assigned this Oct 24, 2018
@joohokim1
Copy link
Contributor Author

This issued cannot be done in 3.0.6. Moving to 3.0.7.

@joohokim1 joohokim1 modified the milestones: 3.1.0, 3.2.0 Nov 21, 2018
joohokim1 added a commit that referenced this issue Mar 21, 2019
@joohokim1
#540 comprehensible error msg when cannot find core-site.xml or hdfs-…
7e40eaa 
…site.xml
joohokim1 added a commit that referenced this issue Mar 21, 2019
@joohokim1
#540 refactoring to separate the utility class. throw PrepException w…
9e66b14 
…hen hadoop-essential files not found. revived some test cases.
joohokim1 added a commit that referenced this issue Mar 21, 2019
@joohokim1
#540 throw PrepException when essential hdfs works fail
a53d262
joohokim1 added a commit that referenced this issue Mar 22, 2019
@joohokim1
#540 appropriate error message when failed to hdfs name node. removed…
8e9d630 
… unused code
joohokim1 added a commit that referenced this issue Mar 22, 2019
@joohokim1
#540 appropriate error message when failed to hdfs name node
c0d3b90
@joohokim1 joohokim1 mentioned this issue Mar 22, 2019
Merged
9 tasks
@joohokim1
Copy link
Contributor Author

@paigechoi

3 error codes below has been added onto the file upload page:
msg.dp.alert.hadoop.core.site.not.found
msg.dp.alert.hadoop.hdfs.site.not.found
msg.dp.alert.hadoop.hdfs.failed.to.connect

Please add error handlers appropriately.

paigechoi added a commit that referenced this issue Mar 25, 2019
@paigechoi
#540 Show error alert when hdfs is not configured
59dc65f
joohokim1 added a commit that referenced this issue Mar 25, 2019
@joohokim1
#540 remove build error in test cases
f718abc
joohokim1 added a commit that referenced this issue Mar 26, 2019
@joohokim1
#540 better hadoop conf errmsg and some refactoring
7b1d376 
* #540 comprehensible error msg when cannot find core-site.xml or hdfs-site.xml

* #540 refactoring to separate the utility class. throw PrepException when hadoop-essential files not found. revived some test cases.

* #540 throw PrepException when essential hdfs works fail

* #540 appropriate error message when failed to hdfs name node. removed unused code

* #540 appropriate error message when failed to hdfs name node

* some files were left out at last commit

* #540 Show error alert when hdfs is not configured

* #540 remove build error in test cases
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@paigechoi paigechoi

@sang-hee-lee sang-hee-lee

@joohokim1 joohokim1

Labels
@dataprep Component Name : Data preparation enhancement Request Change and Feature Enhancement
Projects
None yet
Milestone
3.2.0
Development

No branches or pull requests
3 participants
@paigechoi @sang-hee-lee @joohokim1