Need help with pyspark

[ananbas]

Expected behavior

GeoSparkRegistrator.registerAll(spark) return 0

Actual behavior

Got error

Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, bddevdn0001, executor 4): java.lang.IllegalStateException: unread block data

Steps to reproduce the problem

this is my code:

import findspark
findspark.init('/opt/cloudera/parcels/CDH/lib/spark')

import pyspark
import geo_pyspark
from pyspark.sql import SparkSession
from geo_pyspark.register import GeoSparkRegistrator

spark = SparkSession \
.builder \
.config("spark.driver.memory", "2g") \
.config("spark.jars", "local:/home/anung/geospark/geo_wrapper.jar,local:/home/anung/geospark/geospark-1.2.0.jar,local:/home/anung/geospark/geospark-sql_2.3-1.2.0.jar") \
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
.config("spark.kryo.registrator","org.datasyslab.geospark.serde.GeoSparkKryoRegistrator") \
.appName("anung-geospark-test") \
.enableHiveSupport() \
.getOrCreate()

GeoSparkRegistrator.registerAll(spark)

Settings

GeoSpark version = 1.2

Apache Spark version = 2.4.0-cdh6.2.1

JRE version = 1.8

API type = Python

[Imbruced]

Can you check few things ?

• GeoSparkSQL Scala/Java API produce the same error ?
• You have JAVA_HOME environment variable set ?
• Any action on Spark DataFrame works correctly (without GeoSpark) ?

[ananbas]

Hi Imbriced,

1. Using Scala API no error, everything worked, example project running.
2. Yes of course.
3. Yes, we are using pyspark for daily ETL.

[Imbruced]

Hi,
To me it looks like dependency issue, Library was tested with native spark (2.2, 2.3, 2.4) for hadoop distribution 2.7 and scala 2.11. I will try to reproduce the problem on cdh6.2.1 environment. I have two possible solutions, need to test them, (I need 2-3 days).

[Imbruced]

Hi,
I was not able to reproduce your issue. I added changes which can solve the issue on forked version,
https://github.com/Imbruced/GeoSpark/tree/master/python. Please let me know if it works for you.
geo_wrapper jar is no longer needed please remove it from spark.jars option.

[Butterflyer043]

Hi did you solve the problem? I have the same issue.

[jornfranke]

I recommend to put the jar geospark_2.11-1.3.1.jar somewhere on HDFS on the cluster. The following example assume that it is in the folder /jars on HDFS in the cluster (you can put it though anywhere on HDFS and adapt the URL below accordingly). Please replace "myhdfshost" with the hdfs url of your cluster in the following fragment:

from pyspark.sql import SparkSession
from geospark.register import GeoSparkRegistrator
from geospark.utils.adapter import Adapter
from geospark.core.spatialOperator import KNNQuery
from geospark.core.formatMapper import GeoJsonReader
from shapely.geometry import Point

from geospark.register import GeoSparkRegistrator
from geospark.utils.serde import GeoSparkKryoRegistrator,KryoSerializer
 
# Load geospark library
spark = SparkSession\
    .builder\
    .config("spark.jars", "hdfs://myhdfshost/jars/geospark_2.11-1.3.1.jar")\
    .config('spark.executor.memory', "2g")\
    .config("spark.driver.memory", "3g")\
    .config("spark.serializer", KryoSerializer.getName)\
    .config("spark.kryo.registrator", GeoSparkKryoRegistrator.getName)\
    .config("spark.kryoserializer.buffer.max", "1g")\
    .appName("Geospark-spatialjoin-PySpark")\
    .getOrCreate()