Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix configuration on hadoop mapreduce job #972

Merged
merged 2 commits into from
Oct 11, 2017

Conversation

gom
Copy link
Contributor

@gom gom commented Jan 19, 2017

I got following "Wrong FS" errors, when I run SegmentCreation job with pinot-hadoop.

It seems new Configuration() loads only empty core-site.xml file, and loads fs.defaultFS value as file:/// in the MR job.
It will be fixed using context.getProperty() in the job.
Mapper would be better to use JobConf from their context.

Similar issue as #63.


Error on the map job log:

2017-01-17 09:39:50,712 ERROR [main] com.linkedin.pinot.hadoop.job.mapper.HadoopSegmentCreationMapReduceJob$HadoopSegmentCreationMapper: Got exceptions during creating segments!
java.lang.IllegalArgumentException: Wrong FS: hdfs://<HOSTNAME>:8020/user/pinot/input/data/part-4.avro, expected: file:///
	at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:661)
	at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:87)
	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:619)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:850)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:614)
	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:422)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
	at org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem.java:88)
	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1996)
	at com.linkedin.pinot.hadoop.job.mapper.HadoopSegmentCreationMapReduceJob$HadoopSegmentCreationMapper.createSegment(HadoopSegmentCreationMapReduceJob.java:154)
	at com.linkedin.pinot.hadoop.job.mapper.HadoopSegmentCreationMapReduceJob$HadoopSegmentCreationMapper.map(HadoopSegmentCreationMapReduceJob.java:134)
	at com.linkedin.pinot.hadoop.job.mapper.HadoopSegmentCreationMapReduceJob$HadoopSegmentCreationMapper.map(HadoopSegmentCreationMapReduceJob.java:44)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

@codecov-io
Copy link

codecov-io commented Jan 19, 2017

Codecov Report

❗ No coverage uploaded for pull request base (master@1bcb7f1). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff            @@
##             master    #972   +/-   ##
========================================
  Coverage          ?     67%           
========================================
  Files             ?     748           
  Lines             ?   35631           
  Branches          ?    4530           
========================================
  Hits              ?   23873           
  Misses            ?   10194           
  Partials          ?    1564

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1bcb7f1...f57a805. Read the comment docs.

@lionelfeng
Copy link

I met the same problem and it took me one day to direct me here to save my ass. This PR did fix my issue and please merge this PR asap since the new Configuration() is really confusing... Thanks.

@kishoreg
Copy link
Member

@jenniferdai can you please take a look at this PR

@jenniferdai jenniferdai merged commit 0b8461e into apache:master Oct 11, 2017
@jenniferdai
Copy link
Member

@gom thank you for fixing this!

@gom
Copy link
Contributor Author

gom commented Oct 11, 2017

@jenniferdai thank you for merging :)

@gom gom deleted the fix_hadoop_configuration branch October 11, 2017 23:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants