[Feature][E2E] Provide ARM64 (aarch64) variants of E2E test Docker images for Apple Silicon Macs

[Dream95]

Search before asking

• I had searched in the feature and found no similar feature requirement.

Description

Many E2E tests depend on custom images under tyrantlucifer/* that are amd64-only. On Apple Silicon Macs , running them via Docker emulation is slow and often unstable.
Please provide ARM64 or multi-arch variants for other common E2E images , and use the same arch-based image selection pattern.

Usage Scenario

Develop and debug connectors / engine features on Apple Silicon Macs, and run Testcontainers E2E locally with native linux/arm64 images.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[DanielLeens]

Thanks for raising this. This reads like a reasonable E2E enhancement rather than a bug.

The underlying problem is clear: several of the custom E2E images used in SeaTunnel testing are effectively amd64-only today, which makes local E2E work on Apple Silicon slower and less reliable because it falls back to emulation. That is a real contributor-experience gap, especially for connector and engine debugging.

A good implementation path would be:

1. identify the commonly used tyrantlucifer/* E2E images that block local ARM64 runs most often;
2. publish ARM64 or multi-arch variants for those images first;
3. keep image selection explicit in the tests/build scripts, following the existing architecture-based selection pattern where available;
4. document which E2E suites become natively runnable on Apple Silicon and which still depend on amd64-only upstream images.

I've labeled this as feature and help wanted so contributors can pick it up. If someone works on it, the most useful first slice would be the highest-usage images plus one focused end-to-end verification showing native linux/arm64 execution on Apple Silicon.

[SEZ9]

Could you list the specific images and E2E modules affected (e.g., which tyrantlucifer/* images fail or run slowly under emulation)? That would help prioritize which images to rebuild first.

Note: the tyrantlucifer/* images are hosted under a personal Docker Hub account. A good long-term fix may be to publish the Dockerfiles in the repo (if not already) and build multi-arch images via docker buildx (linux/amd64 + linux/arm64), ideally under an Apache-managed namespace.

As a workaround pattern, some tests already select images by architecture; contributions that rebuild individual images as multi-arch and update the corresponding E2E test container config would be welcome — feel free to open incremental PRs per image.

[FenjuFu]

I audited current main for literal yrantlucifer/ references. The actual Docker image dependencies are concentrated in seatunnel-e2e-common (the other search hits in file connector tests/docs are test paths such as
ame=tyrantlucifer, not image names):

• Spark: yrantlucifer/spark:2.4.6 (Spark2Container) and yrantlucifer/spark:3.3.0 (Spark3Container).
• Flink: yrantlucifer/flink:1.13.6-scala_2.11_hadoop27, 1.14.6-scala_2.11_hadoop27, 1.15.3-scala_2.12_hadoop27, 1.16.0-scala_2.12_hadoop27, 1.17.2-scala_2.12_hadoop27, 1.18.0-scala_2.12_hadoop27, and 1.20.1-scala_2.12_hadoop27 in the corresponding Flink*Container classes.

That means the directly affected execution paths are the shared Spark 2/3 and Flink 1.13-1.18/1.20 E2E environments, rather than individual file connector modules.

There is an existing architecture-selection precedent in connector-cdc-oracle-e2e (AbstractOracleCDCIT#getImage, checking os.arch == aarch64) and a multi-arch image precedent noted in connector-cdc-postgres-e2e.

For an incremental first slice, I suggest publishing the Dockerfiles/build metadata in-repo and rebuilding the two Spark tags as linux/amd64,linux/arm64, since that covers both shared Spark environments with two images. If an Apache-managed registry/namespace is available, I can prepare the container-class updates and buildx workflow against the agreed image names; otherwise changing Java references before a durable image destination exists would leave the tests unusable. Please confirm the preferred registry path and where these Dockerfiles should live, and I'll take that slice.

[DanielLeens]

Thanks for doing the repo audit. I checked the current dev tree as well, and the scope you listed matches what is actually wired today:

1. the shared tyrantlucifer/* image references that affect Apple Silicon local E2E are concentrated in seatunnel-e2e-common, especially Spark2Container, Spark3Container, and the shared Flink 1.13/1.14/1.15/1.16/1.17/1.18/1.20 container classes;
2. I do not see repo-owned Dockerfiles for those shared E2E images under seatunnel-e2e today;
3. the repo already has project Docker publishing infrastructure in .github/workflows/publish-docker.yaml, including QEMU + Buildx for multi-arch publishing, so I would prefer the replacement path to stay project-owned rather than move to another personal namespace.

Given that, I agree with your proposed first slice:

• start with the two shared Spark images first, because that covers both shared Spark environments with the smallest image count;
• add the Dockerfiles / build metadata in-repo near the shared seatunnel-e2e-common container layer, since that is where the current image references live;
• keep the first PR focused on one native ARM64 proof point plus the Spark2Container / Spark3Container updates, instead of trying to rebuild every Flink tag at once.

I would not block you on the exact final publish destination before opening the PR. Please open a focused PR with:

1. the in-repo build context for the Spark images;
2. the Java container updates for Spark2Container and Spark3Container;
3. a short note in the PR description about the desired project-owned publish destination and which remaining Flink images are intentionally left for follow-up.

That should be a reviewable first step for this issue, and we can keep the remaining Flink image set as follow-up work once the Spark path is proven.