How to use with AWS Glue?

[kasunnirmala]

How to add the extension and the way to use it in AWS Glue? Either Notebook or shell script

[EnricoMi]

You should be able to use this as any other Spark package. Try to find out how this is done in general, it should apply to spark-extension as well.

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html#aws-glue-programming-python-libraries-job
https://stackoverflow.com/questions/53382166/how-to-import-spark-packages-in-aws-glue

[EnricoMi]

@leewalter78 do you have some suggestions, looks like you have used this with Glue.

[leewalter78]

#64 (comment)

[kasunnirmala]

I get an error something like this.

[EnricoMi]

You are using Python? Which spark-extension version did you use? Which Glue version?

Do you have step-by-step instructions for me to reconstruct this error on Glue?

[kasunnirmala]

You are using Python? Which spark-extension version did you use? Which Glue version?

Do you have step-by-step instructions for me to reconstruct this error on Glue?

I was using spark-extension_2.12:2.1.0-3.3 with Glue 3.0, and python 3

[EnricoMi]

Glue 3.0 uses Spark 3.1, so you should use spark-extension_2.12:2.1.0-3.1, not ...-3.3.

[kasunnirmala]

Glue 3.0 uses Spark 3.1, so you should use spark-extension_2.12:2.1.0-3.1, not ...-3.3.

@EnricoMi Thanks. And you have inserted the s3 path of jar file for Python library path or Dependent JARs path ?

[EnricoMi]

Glue 3.0 uses Spark 3.1, so you should use spark-extension_2.12:2.1.0-3.1, not ...-3.3.

@EnricoMi Thanks. And you have inserted the s3 path of jar file for Python library path or Dependent JARs path ?

As Python library path. I uploaded the jar file as is.