Support Spark 3.3 and EMR 6.10 #113
Conversation
| processors: ["cpu"] | ||
| python: ["py39"] | ||
| sm_version: "1.1" | ||
| sm_version: "1.0" |
There was a problem hiding this comment.
Why are we going down to 1.0?
There was a problem hiding this comment.
As discussed offline, this represents container's minor version. Hence, we downgrade it to 1.0 everytime major version gets upgraded because of spark or python version update.
| @@ -0,0 +1,132 @@ | |||
| FROM 137112412989.dkr.ecr.us-west-2.amazonaws.com/amazonlinux:2 | |||
There was a problem hiding this comment.
New file? I thought we would already have one dockerfile and would just need to update it. Why do we need a new docker file?
There was a problem hiding this comment.
Ah I see. So, for each major version upgrade we create new folder(missed the folder 3.3 earlier)
There was a problem hiding this comment.
yeah.. that's right!
| && yum install -y awscli bigtop-utils curl gcc gzip unzip zip gunzip tar wget liblapack* libblas* libopencv* libopenblas* | ||
|
|
||
| # Install python 3.9 | ||
| ARG PYTHON_BASE_VERSION=3.9 |
There was a problem hiding this comment.
Is this the latest version supported by emr release?
There was a problem hiding this comment.
As discussed offline, retaining the current python version (python 3.9) as per the plan.
3 participants
Description of changes:
Added support for Spark 3.3 and EMR 6.10
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.