Skip to content
This repository has been archived by the owner on Aug 31, 2021. It is now read-only.

How to import the module in pyspark? #91

Open
angeluzdomini opened this issue Feb 9, 2021 · 1 comment
Open

How to import the module in pyspark? #91

angeluzdomini opened this issue Feb 9, 2021 · 1 comment

Comments

@angeluzdomini
Copy link

I am trying to import in pyspark in EMR and not able to find the module. It is getting downloaded though:

Arguments :spark-submit --deploy-mode cluster --packages com.audienceproject:spark-dynamodb_2.12:1.1.1 s3://filepath

import com.audienceproject.spark.dynamodb_

if name == 'main':

sc = SparkContext(appName="reading csv")

# creating the context
sqlContext = SQLContext(sc)

dynamoDf = sparkSession.read.option("tableName", "event-poc").format("dynamodb").load() 
# Scan the table for the first 100 items (the order is arbitrary) and print them.
dynamoDf.show(10)

# write to some other table overwriting existing item with same keys
dynamoDf.write.option("tableName", "emr-test").format("dynamodb").save()
@angeluzdomini angeluzdomini changed the title Ho to import the module in pyspark? How to import the module in pyspark? Feb 9, 2021
@angeluzdomini
Copy link
Author

@jacobfi @johsbk

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Development

No branches or pull requests

1 participant