ISO8601 dates with timezone offset considered invalid

[bobrik]

I'm using 2.1.0Beta4 and Spark 1.3.1:

JavaEsSparkSQL.esDF().saveAsParquetFile()

Fails with error:

15/05/26 14:55:49 ERROR Executor: Exception in task 4.0 in stage 0.0 (TID 4)
java.lang.IllegalArgumentException: 2015-05-25T22:30:00+0300
    at com.sun.org.apache.xerces.internal.jaxp.datatype.XMLGregorianCalendarImpl$Parser.skip(XMLGregorianCalendarImpl.java:2844)
    at com.sun.org.apache.xerces.internal.jaxp.datatype.XMLGregorianCalendarImpl$Parser.parse(XMLGregorianCalendarImpl.java:2808)
    at com.sun.org.apache.xerces.internal.jaxp.datatype.XMLGregorianCalendarImpl.<init>(XMLGregorianCalendarImpl.java:435)
    at com.sun.org.apache.xerces.internal.jaxp.datatype.DatatypeFactoryImpl.newXMLGregorianCalendar(DatatypeFactoryImpl.java:536)
    at javax.xml.bind.DatatypeConverterImpl._parseDateTime(DatatypeConverterImpl.java:422)
    at javax.xml.bind.DatatypeConverterImpl.parseDateTime(DatatypeConverterImpl.java:417)
    at javax.xml.bind.DatatypeConverter.parseDateTime(DatatypeConverter.java:327)
    at org.elasticsearch.spark.serialization.ScalaValueReader.parseDate(ScalaValueReader.scala:113)
    at org.elasticsearch.spark.serialization.ScalaValueReader$$anonfun$date$1.apply(ScalaValueReader.scala:106)
    at org.elasticsearch.spark.serialization.ScalaValueReader$$anonfun$date$1.apply(ScalaValueReader.scala:106)
    at org.elasticsearch.spark.serialization.ScalaValueReader.checkNull(ScalaValueReader.scala:58)
    at org.elasticsearch.spark.serialization.ScalaValueReader.date(ScalaValueReader.scala:106)
    at org.elasticsearch.spark.serialization.ScalaValueReader.readValue(ScalaValueReader.scala:46)
    at org.elasticsearch.hadoop.serialization.ScrollReader.parseValue(ScrollReader.java:540)
    at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:528)
    at org.elasticsearch.hadoop.serialization.ScrollReader.map(ScrollReader.java:596)
    at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:519)
    at org.elasticsearch.hadoop.serialization.ScrollReader.readHitAsMap(ScrollReader.java:339)
    at org.elasticsearch.hadoop.serialization.ScrollReader.readHit(ScrollReader.java:290)
    at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:185)
    at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:164)
    at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:403)
    at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:76)
    at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:46)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)
    at org.apache.spark.sql.parquet.ParquetRelation2.org$apache$spark$sql$parquet$ParquetRelation2$$writeShard$1(newParquet.scala:669)
    at org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$insert$2.apply(newParquet.scala:689)
    at org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$insert$2.apply(newParquet.scala:689)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
    at org.apache.spark.scheduler.Task.run(Task.scala:64)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

[costin]

The issue seems to be caused by the missing colon : in the timezone offset. ISO8601 marks it as optional but several implementations expect it hence the error.
I'll try to provide a fix for it - out of curiosity, what generated the date in the first place?

[bobrik]

php does that:

~ λ php -r 'var_dump(date(DateTime::ISO8601));'
string(24) "2015-06-01T14:23:04+0300"

[costin]

Probably not the only one to do so. Ironically there's a comment I read on their official docs some minutes ago

Our solution was to create the following constant on our DateHelper object.

<?php
class DateHelper
{
    /**
     * An ISO8601 format string for PHP's date functions that's compatible with JavaScript's Date's constructor method
     * Example: 2013-04-12T16:40:00-04:00
     *
     * PHP's ISO8601 constant doesn't add the colon to the timezone offset which is required for iPhone
    **/
    const ISO8601 = 'Y-m-d\TH:i:sP';
}
?>

...

[costin]

Pushed a fix in master and snapshot in the OSS maven repo. Can you please try it out?

[bobrik]

I am probably doing it wrong, but:

[error] sbt.ResolveException: unresolved dependency: org.elasticsearch#elasticsearch-spark_2.11;2.1.0.BUILD-SNAPSHOT: not found

[costin]

Make sure you are using the dedicated snapshot repo (Maven central doesn't allow SNAPSHOTs to be published):
https://www.elastic.co/guide/en/elasticsearch/hadoop/master/install.html#download-dev

[bobrik]

Got it, but also got an error when writing to parquet:

15/06/02 15:30:10 ERROR Executor: Exception in task 4.0 in stage 0.0 (TID 4)
java.lang.ClassCastException: java.util.Date cannot be cast to java.sql.Timestamp
    at org.apache.spark.sql.parquet.RowWriteSupport.writePrimitive(ParquetTableSupport.scala:210)
    at org.apache.spark.sql.parquet.RowWriteSupport.writeValue(ParquetTableSupport.scala:192)
    at org.apache.spark.sql.parquet.RowWriteSupport.write(ParquetTableSupport.scala:171)
    at org.apache.spark.sql.parquet.RowWriteSupport.write(ParquetTableSupport.scala:134)
    at parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:120)
    at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:81)
    at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:37)
    at org.apache.spark.sql.parquet.ParquetRelation2.org$apache$spark$sql$parquet$ParquetRelation2$$writeShard$1(newParquet.scala:671)
    at org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$insert$2.apply(newParquet.scala:689)
    at org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$insert$2.apply(newParquet.scala:689)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
    at org.apache.spark.scheduler.Task.run(Task.scala:64)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

This is probably not your fault, though: SPARK-3034.

Feel free to close the issue then. Thanks!

[costin]

I think I can fix that. Unfortunately when it comes to Parquet I keep running into issues about the Hadoop version used
(1 vs 2).
What's your DF schema?

By the way, can you please jump on the IRC to speed up the fix? I'm costin on #elasticsearch

Thanks,

On 6/2/15 2:35 PM, Ian Babrou wrote:

Got it, but also got an error when writing to parquet:

|15/06/02 15:30:10 ERROR Executor: Exception in task 4.0 in stage 0.0 (TID 4)
java.lang.ClassCastException: java.util.Date cannot be cast to java.sql.Timestamp
at org.apache.spark.sql.parquet.RowWriteSupport.writePrimitive(ParquetTableSupport.scala:210)
at org.apache.spark.sql.parquet.RowWriteSupport.writeValue(ParquetTableSupport.scala:192)
at org.apache.spark.sql.parquet.RowWriteSupport.write(ParquetTableSupport.scala:171)
at org.apache.spark.sql.parquet.RowWriteSupport.write(ParquetTableSupport.scala:134)
at parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:120)
at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:81)
at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:37)
at org.apache.spark.sql.parquet.ParquetRelation2.org$apache$spark$sql$parquet$ParquetRelation2$$writeShard$1(newParquet.scala:671)
at org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$insert$2.apply(newParquet.scala:689)
at org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$insert$2.apply(newParquet.scala:689)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
|

This is probably not your fault, though: SPARK-3034 https://issues.apache.org/jira/browse/SPARK-3034.

Feel free to close the issue then. Thanks!

—
Reply to this email directly or view it on GitHub
#458 (comment).

Costin

[costin]

For the record, after talking on the IRC the initial problem was fixed. There is a separate one raised #464 but this one can be closed.

[bobrik]

Thanks a lot! 👍