IllegalArgumentException Wrong FS for vcf_head files on HDFS #1272

Closed
heuermh opened this Issue Nov 16, 2016 · 1 comment

Comments

Projects
None yet
2 participants
@heuermh
Member

heuermh commented Nov 16, 2016

From adam-developers@googlegroups.com:

$ adam-submit adam2vcf \
  "hdfs://localhost:54310/vcf_pruebas/vcfpruebas.adam" \
  "hdfs://localhost:54310/vcf_pruebas/vcfpruebas2.vcf"

...
2016-11-10 16:46:49 ERROR Executor:91 - Exception in task 0.0 in stage 1.0 (TID 1)
java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost:54310/vcf_pruebas/vcfpruebas2.vcf_head, expected: file:///
  at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:647)
  at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:82)
  at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:604)
  at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:822)
  at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:599)
  at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
  at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:140)
  at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
  at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
  at org.seqdoop.hadoop_bam.util.WrapSeekable.openPath(WrapSeekable.java:60)
  at org.seqdoop.hadoop_bam.KeyIgnoringVCFOutputFormat.readHeaderFrom(KeyIgnoringVCFOutputFormat.java:82)
  at org.bdgenomics.adam.rdd.variation.ADAMVCFOutputFormat.getRecordWriter(ADAMVCFOutputFormat.scala:46)
...

When the process finishes, it leaves in the HDFS two new files:

  • An empty "vcfpruebas2.vcf" (the expected name for the final file)
  • A file "vcfpruebas2.vcf_head", that contains: a meta-information VCF structure, with just some of the FORMAT and INFO keys that were in the original VCF file (and many less than those that are in the INFO column of the original variants contents), and the header line, including all the sample columns.
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Mar 3, 2017

Member

@heuermh without more info, we can't debug this, as I suspect this is a Hadoop conf issue. We've added support for reading/writing with non-default configured FS. I am going to close this ticket, and we can reopen if this reoccurs.

Member

fnothaft commented Mar 3, 2017

@heuermh without more info, we can't debug this, as I suspect this is a Hadoop conf issue. We've added support for reading/writing with non-default configured FS. I am going to close this ticket, and we can reopen if this reoccurs.

@fnothaft fnothaft closed this Mar 3, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment