pyspark.sql.types.DecimalType

[plandem]

Getting such error when trying to show dataframe. Its ok in case of FloatType.

: java.lang.NumberFormatException
	at java.math.BigDecimal.<init>(BigDecimal.java:494)
	at java.math.BigDecimal.<init>(BigDecimal.java:383)
	at java.math.BigDecimal.<init>(BigDecimal.java:806)
	at com.crealytics.spark.excel.ExcelRelation.bigDecimal$lzycompute$1(ExcelRelation.scala:114)
	at com.crealytics.spark.excel.ExcelRelation.bigDecimal$1(ExcelRelation.scala:114)
	at com.crealytics.spark.excel.ExcelRelation.com$crealytics$spark$excel$ExcelRelation$$castTo(ExcelRelation.scala:123)
	at com.crealytics.spark.excel.ExcelRelation$$anonfun$1$$anonfun$3.apply(ExcelRelation.scala:76)
	at com.crealytics.spark.excel.ExcelRelation$$anonfun$1$$anonfun$3.apply(ExcelRelation.scala:76)
	at com.crealytics.spark.excel.ExcelRelation$$anonfun$1$$anonfun$apply$2.apply(ExcelRelation.scala:89)
	at com.crealytics.spark.excel.ExcelRelation$$anonfun$1$$anonfun$apply$2.apply(ExcelRelation.scala:84)
	at com.crealytics.spark.excel.ExcelRelation$$anonfun$5$$anonfun$apply$3.apply(ExcelRelation.scala:94)
	at com.crealytics.spark.excel.ExcelRelation$$anonfun$5$$anonfun$apply$3.apply(ExcelRelation.scala:94)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)

[nightscape]

Can you post a small example file and the actual calls that you made?

[plandem]

Well, i provide schema where one field is like:

        {
           'name': 'amount',
           'type': DecimalType(15, 2),
        },

At excel it's custom type like # ##0 "EUR". Just trying to show() dataframe. with FloatType everything is ok

[nightscape]

Would you mind trying to debug this?
In order to do this, check the code out locally, put a breakpoint in ExcelRelation:114 and check what value actually makes it fail.

[plandem]

not sure....I'm using it from pyspark :(

[nightscape]

The parsing of BigDecimal has been changed to use the underlying Double value.
We potentially lose some precision by that, but parsing BigDecimal as Strings didn't work out well.
Can you try the latest version and comment here if it doesn't work?
I'll close this issue until then because I think it should be solved.