Unable to load sasdata set into Spark

[pathri-pk]

Using below code to load a sample sasdata set into spark and getting a timeout error, tried increasing the timeout using option 'metadataTimeout' but still timing out reading metadata, any help is appreciated

import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars /Spark/spark-2.4.1-bin-hadoop2.7/jars/spark-sas7bdat-2.1.0-s_2.11.jar pyspark-shell'
from pyspark.sql import SparkSession

spark = SparkSession.builder.master("local").appName("Spark App").getOrCreate()
df = spark.read.format("com.github.saurfang.sas.spark").load("airline.sas7bdat", forceLowercaseNames=True, inferLong=True)

Error:
Py4JJavaError: An error occurred while calling o226.load.
: java.util.concurrent.TimeoutException: Timed out after 60 sec while reading file metadata, file might be corrupt. (Change timeout with 'metadataTimeout' paramater)

[thesuperzapper]

@pathri-pk are you able to post that sas file? (You should be able to drag it into a reply to this issue, if its not huge)

[jene4ekjene4ek]

Hi all. I have the same problem with this file. The file is not corrupted, because I read it with pandas.
airline.zip

[mazenora]

I had the same problem, so i read the file with pandas then convert it to Spark DataFrame.

df = pd.read_sas('YOUR_FILE',format='sas7bdat')
mySchema = StructType([ ]) <-- create suitable schema structure for spark 
spark_df   = spark.createDataFrame(df,schema=mySchema)

[jene4ekjene4ek]

I've solved this issue by using jars parso and spark-sas7bdat

[mazenora]

could you please share the code?

[jene4ekjene4ek]

In my project folder I have folder jars with parso-2.0.10.jar and spark-sas7bdat-2.1.0-s_2.11.jar (I added printscreen) and run my script as spark-submit --jars jars/spark-sas7bdat-2.1.0-s_2.11.jar,jars/parso-2.0.10.jar script.py

spark = SparkSession.builder.config("spark.jars.packages", "saurfang:spark-sas7bdat:2.1.0-s_2.11") .getOrCreate()

df = spark.read.format('com.github.saurfang.sas.spark')
.load("airline.sas7bdat")