Skip to content
This repository has been archived by the owner on May 3, 2022. It is now read-only.

Config not being read on driver and/or executor #76

Closed
srowen opened this issue Nov 16, 2014 · 1 comment
Closed

Config not being read on driver and/or executor #76

srowen opened this issue Nov 16, 2014 · 1 comment

Comments

@srowen
Copy link
Member

srowen commented Nov 16, 2014

The executor or driver doesn't seem to get getting config, even though the file is deployed to executors and -Dconfig.file is set:

java.lang.IllegalArgumentException: Wrong FS: file://xxxxx.cloudera.com:8020/tmp/Oryx/data, expected: hdfs://sssss.cloudera.com:8020
    at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:192)
    at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:651)
    at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:104)
    at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:716)
    at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:712)
    at org.apache.hadoop.fs.Globber.listStatus(Globber.java:69)
    at org.apache.hadoop.fs.Globber.glob(Globber.java:217)
    at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1625)
    at com.cloudera.oryx.lambda.BatchUpdateFunction.call(BatchUpdateFunction.java:90)
    at com.cloudera.oryx.lambda.BatchUpdateFunction.call(BatchUpdateFunction.java:45)
    at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$2.apply(JavaDStreamLike.scala:282)
    at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$2.apply(JavaDStreamLike.scala:282)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:41)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
...
@srowen srowen self-assigned this Nov 16, 2014
@srowen srowen added this to the 2.0.0 milestone Nov 16, 2014
@srowen
Copy link
Member Author

srowen commented Nov 16, 2014

Ultimately due to a typo, but, resolved by printing config in b021d00

@srowen srowen closed this as completed Nov 16, 2014
@srowen srowen modified the milestones: 2.0.0, 2.0.0-alpha-1 Nov 26, 2014
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant