[SPARK-49243][INFRA][K8S] Add `OpenContainers` Annotations in docker images by dongjoon-hyun · Pull Request #47766 · apache/spark

dongjoon-hyun · 2024-08-15T04:10:12Z

What changes were proposed in this pull request?

This PR aims to add OpenContainers Annotations to docker image to add metadata like Apache License.

https://specs.opencontainers.org/image-spec/annotations/

Why are the changes needed?

BEFORE

$ docker inspect apache/spark:3.5.2 | jq '.[0].Config.Labels'
{
  "org.opencontainers.image.ref.name": "ubuntu",
  "org.opencontainers.image.version": "20.04"
}

AFTER

$ NO_MANUAL=1 ./dev/make-distribution.sh -Pkubernetes
$ bin/docker-image-tool.sh -t SPARK-49243 -n build
$ docker inspect spark:SPARK-49243 | jq '.[0].Config.Labels'
{
  "org.opencontainers.image.authors": "Apache Spark project <dev@spark.apache.org>",
  "org.opencontainers.image.licenses": "Apache-2.0",
  "org.opencontainers.image.ref.name": "Apache Spark Scala/Java Image",
  "org.opencontainers.image.version": ""
}

Does this PR introduce any user-facing change?

No, it's only a metadata change.

How was this patch tested?

Manual review.

Was this patch authored or co-authored using generative AI tooling?

No.

…images

dongjoon-hyun · 2024-08-15T04:33:36Z

Could you review this PR, @viirya ?

viirya

Looks good to me. Thanks @dongjoon-hyun

dongjoon-hyun · 2024-08-15T05:55:16Z

Thank you, @viirya !

yaooqinn · 2024-08-15T06:18:25Z

dev/create-release/spark-rm/Dockerfile


 # Image for building Spark releases. Based on Ubuntu 22.04.
 FROM ubuntu:jammy-20240227
+LABEL org.opencontainers.image.authors="Apache Spark project <dev@spark.apache.org>"


Apache Spark Community?

Apache Spark Community?

Isn't it too broad because it includes a read-only users?

I believe it's acceptable to include read-only users because non-code contributions, such as testing and sharing use cases, should also be acknowledged.
The User also plays a role in contributing according to 'The Apache Way'.

I believe it's acceptable to include read-only users because non-code contributions, such as testing and sharing use cases, should also be acknowledged. The User also plays a role in contributing according to 'The Apache Way'.

I got your point. It seems that there was a little misunderstanding between you and me.

When I say read-only users, I embraced read-only users (who didn't contribute at all in all ways) as Apache Spark Community.

The point is that I have a broader concept on the Community than yours.

The next question is that

Contributors are Authors?

One more thing. I used the term Project as the author in this way.

https://apache.org/projects/#what-is-an-open-source-project

An open source project includes all aspects of creating, maintaining, and distributing open source software including community building and mentoring, communication, the release process, and everything in between.

Shall we capitalize 'project' to make it consistent with others

Maybe, did you mean Apache Spark by with others? It's possible.

However, technically, Apache Spark™ is a trademark and the unique name of our project while project is a normal noun.

Here is a sample example sentence according to ASF guideline. Unlike Apache Spark, the following normal noun like software is used in a lower case. You can see that the phrase, the Apache Foo project, in the same document as a example.
https://www.apache.org/foundation/marks/#guidelines

"Free copies of Apache ProjectName software under the Apache License and support services for Apache ProjectName are available at my own company website."

Oh, I mean 'Apache Spark pProject' like 'S'cala, 'I'mage etc, but it's just a nit

yaooqinn · 2024-08-15T06:19:11Z

dev/create-release/spark-rm/Dockerfile

+LABEL org.opencontainers.image.authors="Apache Spark project <dev@spark.apache.org>"
+LABEL org.opencontainers.image.licenses="Apache-2.0"
+LABEL org.opencontainers.image.ref.name="Apache Spark Release Manager Image"
+LABEL org.opencontainers.image.version=""


Can it be removed if it's blank?

Ya, I tried, but we should have a blank here to remove the previous OS layer version as described in the PR description.

$ docker inspect apache/spark:3.5.2 | jq '.[0].Config.Labels' { "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "20.04" }

Without this blank, the final information could be interpreted as Apache Spark 20.04.

Got it. Thanks for the explanation.
Maybe we can add some comment here.

Sure. I will add comments, @yaooqinn .

Ya, I agree that the AS-IS is obscure.

So, I'm currently trying to improve this area including bin/docker-image-tool.sh via version detection from RELEASE. For the K8s images, I guess I can deliver a better way to fill this field. But, for this spark-rm/Dockerfile, I guess it's not worth to do.

via version detection from RELEASE

+1

yaooqinn · 2024-08-15T06:22:59Z

resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile

 FROM ${base_img:-spark}
+LABEL org.opencontainers.image.authors="Apache Spark project <dev@spark.apache.org>"
+LABEL org.opencontainers.image.licenses="Apache-2.0"
+LABEL org.opencontainers.image.ref.name="Apache Spark Python Image"


Ya, I also considered Apache Spark PySpark Image and Apache Spark SparkR Image.

But, I believe Python would be better because we have Apache Spark Scala/Java Image as a base image.

In addition, I'm not sure, but Apache Spark Go Image comes later. So, for consistency, language name could work extensively.

Also, please see our docker hub. The Docker convention in the name is languages like scala, r, python3 instead of SparkR or PySpark.

I was considering whether it will conflict with the https://pypi.org/project/pyspark-connect/ or not.

But it looks fine to me now

dongjoon-hyun · 2024-08-15T07:16:51Z

Thank you for thorough review, @yaooqinn .

I addressed your comments by adding a comment line and replied for the discussion part here.

[SPARK-49243][INFRA][K8S] Add OpenContainers Annotations in docker images #47766 (comment)
[SPARK-49243][INFRA][K8S] Add OpenContainers Annotations in docker images #47766 (comment)

Please let me know your opinion.

yaooqinn

LGTM

dongjoon-hyun · 2024-08-15T15:37:03Z

Thank you! Merged to master.

github-actions bot added KUBERNETES BUILD INFRA PYTHON R labels Aug 15, 2024

dongjoon-hyun marked this pull request as draft August 15, 2024 04:13

dongjoon-hyun force-pushed the SPARK-49243 branch 2 times, most recently from 35f93f2 to e34203c Compare August 15, 2024 04:19

dongjoon-hyun marked this pull request as ready for review August 15, 2024 04:26

[SPARK-49243][INFRA][K8S] Add OpenContainers Annotations in docker …

abbf7b8

…images

dongjoon-hyun force-pushed the SPARK-49243 branch from e34203c to abbf7b8 Compare August 15, 2024 04:27

viirya approved these changes Aug 15, 2024

View reviewed changes

yaooqinn reviewed Aug 15, 2024

View reviewed changes

Address comments by adding a comment

e04ce93

yaooqinn approved these changes Aug 15, 2024

View reviewed changes

dongjoon-hyun closed this in 470e3f9 Aug 15, 2024

dongjoon-hyun deleted the SPARK-49243 branch August 15, 2024 15:37

Conversation

dongjoon-hyun commented Aug 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

dongjoon-hyun commented Aug 15, 2024

Uh oh!

viirya left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Aug 15, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Aug 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Aug 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Aug 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Aug 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Aug 15, 2024

Uh oh!

yaooqinn left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Aug 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dongjoon-hyun commented Aug 15, 2024 •

edited

Loading

dongjoon-hyun Aug 15, 2024 •

edited

Loading

dongjoon-hyun Aug 15, 2024 •

edited

Loading

dongjoon-hyun Aug 15, 2024 •

edited

Loading

dongjoon-hyun Aug 15, 2024 •

edited

Loading