Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Support RAPIDS Spark plugin on ARM #6881

Closed
sameerz opened this issue Oct 21, 2022 · 5 comments · Fixed by #7070
Closed

[FEA] Support RAPIDS Spark plugin on ARM #6881

sameerz opened this issue Oct 21, 2022 · 5 comments · Fixed by #7070
Assignees
Labels
feature request New feature or request

Comments

@sameerz
Copy link
Collaborator

sameerz commented Oct 21, 2022

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I wish the RAPIDS Accelerator for Apache Spark would run on ARM

Describe the solution you'd like
Distribute a plugin jar that is functional on ARM-based architecture (including cudf binaries on ARM)

More details:
We need to distribute the spark-rapids-jni jar with 2 different classifers:

  1. cuda11-amd64
  2. cuda11-aarch64

And Spark-rapids may have 2 profiles,
one is for amd64 to depend on jni's cuda11-amd64 classifier,
another is for aarch64(arm) to depend on jni's cuda11-aarch64 classifier.

For amd64, we build on centos7.
For Arm, we'll build on ubuntu 18.04.
It would be nice if we can build jni jar with Dockerfile.
https://hub.docker.com/layers/nvidia/cuda/11.5.2-runtime-ubuntu18.04/images/sha256-05d99b36cccb63102dbb1f5d9e74daf07d49a62151175693a6146c6d0244dc10?context=explore

Describe alternatives you've considered

Additional context

@sameerz sameerz added feature request New feature or request ? - Needs Triage Need team to review and classify labels Oct 21, 2022
@sameerz sameerz added this to To do in Release 22.12 via automation Oct 21, 2022
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Oct 21, 2022
Release 22.12 automation moved this from To do to Done Nov 21, 2022
@pxLi pxLi removed this from Done in Release 22.12 Nov 24, 2022
@pxLi pxLi reopened this Nov 24, 2022
@pxLi
Copy link
Collaborator

pxLi commented Nov 24, 2022

Let's keep this open

We will reapply CICD update after developers refactor modules structures for different cpu archs in plugin and JNI repos.

@abellina
Copy link
Collaborator

UCX tracking issue: openucx/ucx#8762

@abellina
Copy link
Collaborator

Just to be clear @pxLi:

  1. Will our classifier be aarch64 or arm64?
  2. Is the docker image in the description still the right one?
  3. What is the latest/best method to produce arm artifacts that we could share with the UCX developers here Include arm64 in the UCX and JUCX published artifacts openucx/ucx#8762?

@pxLi
Copy link
Collaborator

pxLi commented Dec 12, 2022

Just to be clear @pxLi:

  1. Will our classifier be aarch64 or arm64?
  2. Is the docker image in the description still the right one?
  3. What is the latest/best method to produce arm artifacts that we could share with the UCX developers here Include arm64 in the UCX and JUCX published artifacts openucx/ucx#8762?
  1. I am not sure. As we decide to defer the arm support, the idea should come from spark team
  2. Our last try was to build on the arm-VM on aws as it provides us the minimal available versions of LIBC and LIBCXX libs, we were actually not trying to build in docker for our init arm arch artifact
  3. We have the PR to revert cpu_arch based build, but I assume that solution was not good enough https://github.com/NVIDIA/spark-rapids/pull/7135/files

@pxLi
Copy link
Collaborator

pxLi commented Nov 30, 2023

merged #9888, udpating internal pipelines. I will update this ticket no other issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants