Modify spark image to include azure and gcs dependency jars #198
Conversation
anusudarsan
force-pushed
the
anu/modify-spark-for-azure-gc
branch
from
9995ad4
to
94441e1
Compare
anusudarsan
force-pushed
the
anu/modify-spark-for-azure-gc
branch
from
94441e1
to
a25e72f
Compare
|
any ideas on the failures? I was able to build locally by adding a trusted host flag for pypi, not sure if this is what we want for CI or we have an expired ssl cert |
|
Looks like it started failing on the builds 2 weeks ago: https://github.com/trinodb/docker-images/actions/runs/9024908079 It's probably caused by using CentOS 7 as the base image, which might not getting any updates anymore. We should update |
|
I added a new commit to trinodb/trino#21958 for testing gcs/azure for spark compatibility |
ebyhr
force-pushed
the
anu/modify-spark-for-azure-gc
branch
from
a25e72f
to
ba3828c
Compare
|
Just rebased on master to run with #199 |
|
Started release job: https://github.com/trinodb/docker-images/actions/runs/9220333116 |
| RUN wget -nv https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-azure-datalake/3.3.6/hadoop-azure-datalake-3.3.6.jar | ||
| RUN wget -nv https://repo1.maven.org/maven2/com/microsoft/azure/azure-storage/8.6.6/azure-storage-8.6.6.jar | ||
| # install Google Hadoop connector so we can access gcs | ||
| RUN wget -nv https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop2-latest.jar |
There was a problem hiding this comment.
Can't we use a pinned version of this dependency?
Modify the spark testing image to include Azure and Google cloud storage hadoop dependency jars to enable querying their object stores