Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ST_Point doesn't accept null user data #321

Closed
pedromorfeu opened this issue Jan 15, 2019 · 1 comment

Comments

@pedromorfeu
Copy link

commented Jan 15, 2019

Expected behavior

ST_Point should allow null user data.

Actual behavior

A NullPointerException is thrown is a field has null data:

Dataframe:

|              _c0| _c1| _c2| _c3| _c4| _c5|_c6|               _c7|          _c8|           _c9|
| 214:03:7100:1020|null|null|null|null|null|  0|882869441975549952|28.1337406716|-15.4338995307|

Query without field _c1 works well:

SELECT ST_Point(cast(tiles._c8 as Decimal(24,20)),cast(tiles._c9 as Decimal(24,20)), _c0)

Query with field _c1 fails:

SELECT ST_Point(cast(tiles._c8 as Decimal(24,20)),cast(tiles._c9 as Decimal(24,20)), _c0, _c1)

Error:

java.lang.NullPointerException
	at org.apache.spark.sql.geosparksql.expressions.UserDataGeneratator$$anonfun$generateUserData$1.apply$mcVI$sp(Constructors.scala:323)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
	at org.apache.spark.sql.geosparksql.expressions.UserDataGeneratator$class.generateUserData(Constructors.scala:322)
	at org.apache.spark.sql.geosparksql.expressions.ST_Point.generateUserData(Constructors.scala:238)
	at org.apache.spark.sql.geosparksql.expressions.ST_Point.eval(Constructors.scala:252)

Steps to reproduce the problem

Settings

GeoSpark version = 1.1.3

Apache Spark version = 2.1.0

JRE version = 1.8? Yes

API type = Scala or Java? Scala

@jiayuasu jiayuasu self-assigned this Jan 16, 2019

@jiayuasu jiayuasu added this to the 1.2.0 milestone Jan 16, 2019

@jiayuasu

This comment has been minimized.

Copy link
Member

commented Jan 16, 2019

The way to bring non-spatial columns from DataFrame to RDD is changed in v1.2.0: https://github.com/DataSystemsLab/GeoSpark/blob/master/sql/src/main/scala/org/datasyslab/geosparksql/utils/Adapter.scala#L69

But the new way also doesn't handle the null value. Will do.

@jiayuasu jiayuasu added resolved and removed in progress labels Mar 4, 2019

@jiayuasu jiayuasu closed this Mar 4, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.