java.util.NoSuchElementException: key not found: path

[onsDridi]

I'm trying to test this code

from pyspark.sql import SQLContext
from pyspark import SparkContext
sc = SparkContext(appName="Connect Spark with Redshift")
sql_context = SQLContext(sc)
sc._jsc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId", "ACCESSID")
sc._jsc.hadoopConfiguration().set("fs.s3n.awsSecretAccessKey", "ACEESKEY")
df = sql_context.read \
    .option("url", "jdbc:redshift://example.coyf2i236wts.eu-central1.redshift.amazonaws.com:5439/agcdb?user=user&password=pwd") \
    .option("dbtable", "table_name") \
    .option("tempdir", "s3://bucket/path") \
    .load()

but i'm getting this error

Any ideas ?

[JoshRosen]

I think that you need to add .format("com.databricks.spark.redshift") to your sql_context.read call; my hunch is that Spark can't infer the format for this data source, so you need to explicitly specify that we should use the spark-redshift connector.

(This is an unhelpful error message in Spark; I'll see if there's a way to provide a more helpful one).

[onsDridi]

Thank you Josh , I tried to add it but now i'm getting this error : java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.redshift

[JoshRosen]

Is the Redshift connector JAR on your Spark driver's classpath?

[onsDridi]

Yes, i used this command to run the sript
bin/spark-submit --driver-class-path path/RedshiftJDBC41-1.1.17.1017.jar script.py
I tried also
bin/spark-submit redshiftTestCode/sparkRedshift.py --jars path/RedshiftJDBC41-1.1.17.1017.jar
but i still get the same error

[JoshRosen]

You also need to add the spark-redshift JAR; the Redshift JDBC driver is not sufficient by itself.

[onsDridi]

I did it too but i still have the same error

[JoshRosen]

Can you post the exact command that you tried most recently and which didn't work?

[onsDridi]

I run this commands :
bin/spark-submit redshiftTestCode/sparkRedshift.py --jars /Users/od/Documents/Work/spark-redshift_2.10-1.0.0.jar
bin/spark-submit redshiftTestCode/sparkRedshift.py --jars /Users/od/Documents/Work/RedsphiftJDBC41-1.1.17.1017.jar

Both commands give me the same error : java.util.NoSuchElementException: key not found: path

[JoshRosen]

Okay, and you also added .format("com.databricks.spark.redshift") to your code?

[tokland]

Note you write -- jars after the python script, but this is an option of spark-submit. For the record, this worked for me:

$ spark-submit --jars spark-redshift_2.10-1.1.0.jar,RedshiftJDBC.jar,minimal-json-0.9.4.jar test-redshift.py