[BEAM-8106] Add version to java container image name #12505

emilymye · 2020-08-07T23:00:33Z

Adding version number to Java 8 image names (in anticipation of releasing java11 container)

This PR:

Changes the output tag/name for the docker container task to apache/beam_java8_sdk instead of apache/beam_java_sdk
Adjusts hardcoded/java8-only references to java_sdk to java8_sdk (leaving tackling the changes to allow for versioning to a different PR)
Removes separate Dockerfile for Java11 and add java_version argument to existing Dockerfile. Changes build command:

Old:

./gradlew :sdks:java:container:docker -Pdockerfile=Dockerfile-java11 ...

New:

./gradlew :sdks:java:container:docker -PimageJavaVersion=11 ...

The actual difference betweeen Dockerfile-java11 and Dockerfile are minimal - literally these COPY lines:

beam/sdks/java/container/Dockerfile

Lines 35 to 36 in 63f54fd

    
           COPY target/LICENSE /opt/apache/beam/ 
        
           COPY target/NOTICE /opt/apache/beam/

are the only difference, and I'm pretty sure they should be copied appropriately for the Java11 image as well.

Things that should be done at submission/after merge:

Add new repository to hub.docker.com/apache before next run of release script.
Send a notice out to dev@ and user@
Notify releaser that name of released image will have changed after version 3.XX.00

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

emilymye · 2020-08-11T19:10:11Z

Run Python2_PVR_Flink PreCommit

emilymye · 2020-08-11T19:13:48Z

retest this please

codecov · 2020-09-01T07:06:10Z

Codecov Report

Merging #12505 into master will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #12505      +/-   ##
==========================================
+ Coverage   82.31%   82.33%   +0.02%     
==========================================
  Files         455      452       -3     
  Lines       54596    53985     -611     
==========================================
- Hits        44941    44449     -492     
+ Misses       9655     9536     -119

Impacted Files	Coverage Δ
...eam/runners/interactive/options/capture_control.py	`92.00% <0.00%> (-8.00%)`	⬇️
sdks/python/apache_beam/__init__.py	`80.00% <0.00%> (-5.72%)`	⬇️
conftest.py	`77.77% <0.00%> (-5.56%)`	⬇️
.../python/apache_beam/io/gcp/bigquery_io_metadata.py	`86.95% <0.00%> (-3.67%)`	⬇️
...pache_beam/runners/interactive/interactive_beam.py	`76.02% <0.00%> (-2.83%)`	⬇️
sdks/python/apache_beam/dataframe/io.py	`89.62% <0.00%> (-2.38%)`	⬇️
...e_beam/io/gcp/big_query_query_to_table_pipeline.py	`29.03% <0.00%> (-2.01%)`	⬇️
sdks/python/apache_beam/io/fileio.py	`94.07% <0.00%> (-1.78%)`	⬇️
...beam/runners/interactive/background_caching_job.py	`94.78% <0.00%> (-1.74%)`	⬇️
...eam/runners/interactive/interactive_environment.py	`88.28% <0.00%> (-1.17%)`	⬇️
... and 39 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2bb58fa...d0ec608. Read the comment docs.

kennknowles · 2020-09-11T18:10:33Z

Just checking - is this blocked on anything that we can help with?

emilymye · 2020-09-11T23:52:38Z

@kennknowles sorry for the delay - I got distracted doing some other things after there was a Python precommit error that I think just ended up being a flake. Will ping when ready!

emilymye · 2020-09-14T18:20:34Z

run java postcommit

emilymye · 2020-09-14T22:58:12Z

R: @chamikaramj @TheNeuralBit
cc: @kennknowles
I think Kenneth mentioned y'all might be good reviewers for this PR. I previously ran the Java postcommit and it passed, but I'm uncertain if I correctly ran all the tests I need to actually verify this change.

TheNeuralBit · 2020-09-15T16:18:01Z

It looks like the spotless (Java code format) check is failing. FYI you can auto-format locally with ./gradlew spotlessApply. It's not critical until we're ready to merge though.

TheNeuralBit · 2020-09-15T16:21:00Z

Note the release manager that will likely be impacted is @robinyqiu for 2.25.0, since the 2.24.0 branch has already been cut.

TheNeuralBit · 2020-09-15T16:26:14Z

Run Dataflow PortabilityApi ValidatesRunner with Java 11

TheNeuralBit

I have a couple of minor comments and suggestions. Based on https://lists.apache.org/thread.html/rdf813078ee2728b6f26cb89c2857a844139f2c8ca78a68c69512f1ff%40%3Cdev.beam.apache.org%3E I agree that your set of post-merge tasks is sufficient.

TheNeuralBit · 2020-09-15T16:23:47Z

...-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java

+
+  private static String getDefaultJavaSdkHarnessContainerUrl() {
+    String javaVersionId =
+        Float.parseFloat(System.getProperty("java.specification.version")) >= 9 ? "java11" : "java8";


nit: could you make this an exact check for 8 and 11 and throw an exception for other (unsupported) versions

I went ahead and added an Enum to Environments, and removed similar checks in DataflowRunner

TheNeuralBit · 2020-09-15T16:33:38Z

website/www/site/content/en/documentation/runtime/environments.md

-REPOSITORY                          TAG                 IMAGE ID            CREATED           SIZE
-apache/beam_java_sdk               latest              16ca619d489e        2 weeks ago        550MB
+REPOSITORY                         TAG                 IMAGE ID            CREATED           SIZE
+apache/beam_java_8sdk              latest              16ca619d489e        2 weeks ago        550MB


typo here:

Suggested change

apache/beam_java_8sdk latest 16ca619d489e 2 weeks ago 550MB

apache/beam_java8_sdk latest 16ca619d489e 2 weeks ago 550MB

I'm assuming this change and the others like it are the result of a search to find and update all the references to beam_java_sdk, so we don't need to worry about there being other references?

Let's make sure that the correct container name is set in the environment send to the remote SDKs that use Java transforms as cross-language transforms. (We don't have integration tests setup for this by we can just try existing Kafka/SQL x-lang examples to confirm)

https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/kafkataxi
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/sql_taxi.py

We should make sure the references to beam_java_sdk are updated in those examples too

Weird! I don't know why they didn't show up in my initial ctrl-f

I bet it was a race condition, those examples were added in just the last couple of months

I went ahead and ran the Kafka example "from local Beam repo" without specifying the --sdk_harness_container_image_overrides to see if we should pass the correct URL, which it seems to do (based on the worker container manifest):

{ "args": [ ... ], "image": "apache/beam_java8_sdk:2.25.0.dev", "imagePullPolicy": "IfNotPresent", "name": "sdk-1-0", "volumeMounts": [ { "mountPath": "/var/opt/google", "name": "persist" },

Dataflow can't actually grab that container because it doesn't exist (on hub.docker.com), so the job itself doesn't work, but it seems to be grabbing the right URL.

That looks correct. Thanks.

sdks/java/container/Dockerfile-java11

emilymye · 2020-09-16T00:34:01Z

Will update once I've run the x-lang example

emilymye · 2020-09-17T02:22:10Z

retest this please

emilymye · 2020-09-18T17:23:03Z

Run Java PreCommit

emilymye · 2020-09-18T17:25:40Z

Run Python2_PVR_Flink PreCommit

TheNeuralBit

LGTM, thank you!

I think this is safe to merge but maybe we should hold off until INFRA-20866 is closed. If that takes a while I'd hate to have it hold up the release.

emilymye · 2020-09-28T23:47:10Z

@TheNeuralBit it looks like one of the python unit tests flake-failed? but i think this PR is ready to go

TheNeuralBit · 2020-09-29T23:10:36Z

Filed a jira for the flake: BEAM-10987

TheNeuralBit · 2020-09-29T23:12:23Z

FYI: @lostluck 2.26.0 will be the first release with a Java container published at beam_java8_sdk rather than beam_java_sdk

@emilymye you're planning on sending a message to user@ and dev@ about this right?

emilymye · 2020-09-30T00:13:33Z

@TheNeuralBit composing an email now

TheNeuralBit · 2020-09-30T00:38:28Z

Thank you!

* add java_version arg for building Java harness container * change default java container name to java8 * docs changes * delete Java11 Dockerfile * fix typo * add JavaVersion to Environments * fix more references in documentation to beam_java_sdk * fix names of functions * remove underscore

lostluck · 2020-09-30T22:42:30Z

Ack. Thanks!

…

On Tue, Sep 29, 2020, 5:38 PM Brian Hulette ***@***.***> wrote: Thank you! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12505 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADKDOFM2FUATUGA2Z7EPJBLSIJ4ZBANCNFSM4PYC3INA> .

…12505

probot-autolabeler bot added build core docker infra java runners labels Aug 7, 2020

emilymye force-pushed the version_java_images branch from f636828 to 35ac254 Compare August 10, 2020 20:48

emilymye force-pushed the version_java_images branch from 35ac254 to 2c28eff Compare August 11, 2020 22:41

emilymye force-pushed the version_java_images branch from 2c28eff to 1c46a53 Compare September 1, 2020 06:50

emilymye force-pushed the version_java_images branch 2 times, most recently from 6595512 to 46f5df4 Compare September 14, 2020 01:43

probot-autolabeler bot added dataflow examples python website labels Sep 14, 2020

emilymye force-pushed the version_java_images branch from 2d221be to 48c1242 Compare September 14, 2020 22:42

emilymye changed the title ~~[WIP][BEAM-8106] Add version to java container image name~~ [BEAM-8106] Add version to java container image name Sep 14, 2020

TheNeuralBit reviewed Sep 15, 2020

View reviewed changes

emilymye force-pushed the version_java_images branch from ba2a1fa to 00fb543 Compare September 16, 2020 00:32

emilymye force-pushed the version_java_images branch from 61e90df to e4ed684 Compare September 17, 2020 06:27

emilymye requested review from TheNeuralBit and chamikaramj September 18, 2020 17:23

emilymye force-pushed the version_java_images branch from e4ed684 to 023cb28 Compare September 18, 2020 23:52

TheNeuralBit approved these changes Sep 21, 2020

View reviewed changes

emilymye added 9 commits September 24, 2020 16:11

add java_version arg for building Java harness container

9fa6ca5

change default java container name to java8

59bae74

docs changes

205e56a

delete Java11 Dockerfile

49d4b6c

fix typo

de808a1

add JavaVersion to Environments

f2a0f08

fix more references in documentation to beam_java_sdk

4ec7a3c

fix names of functions

69221f6

remove underscore

d0ec608

emilymye force-pushed the version_java_images branch from 023cb28 to d0ec608 Compare September 24, 2020 23:14

TheNeuralBit merged commit 97a270f into apache:master Sep 29, 2020

emilymye added a commit to emilymye/beam that referenced this pull request Dec 7, 2020

Revert java container changes apache#12505

33f72dd

emilymye mentioned this pull request Dec 7, 2020

[BEAM-11409] Revert java-versioning of SDK container #12505 #13497

Merged

4 tasks

lostluck pushed a commit that referenced this pull request Dec 7, 2020

[release-2.26.0][BEAM-11409] Revert java-versioning of SDK container #…

12f0a9a

…12505

emilymye mentioned this pull request Dec 10, 2020

[BEAM-11433] Prevent defaulting to java11 container #13524

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-8106] Add version to java container image name #12505

[BEAM-8106] Add version to java container image name #12505

emilymye commented Aug 7, 2020 •

edited

emilymye commented Aug 11, 2020

emilymye commented Aug 11, 2020

codecov bot commented Sep 1, 2020 •

edited

kennknowles commented Sep 11, 2020

emilymye commented Sep 11, 2020

emilymye commented Sep 14, 2020

emilymye commented Sep 14, 2020

TheNeuralBit commented Sep 15, 2020

TheNeuralBit commented Sep 15, 2020

TheNeuralBit commented Sep 15, 2020

TheNeuralBit left a comment

TheNeuralBit Sep 15, 2020

emilymye Sep 16, 2020

TheNeuralBit Sep 15, 2020

chamikaramj Sep 15, 2020

TheNeuralBit Sep 16, 2020

emilymye Sep 16, 2020

TheNeuralBit Sep 16, 2020

emilymye Sep 17, 2020

chamikaramj Sep 17, 2020

emilymye commented Sep 16, 2020

emilymye commented Sep 17, 2020

emilymye commented Sep 18, 2020

emilymye commented Sep 18, 2020

TheNeuralBit left a comment

emilymye commented Sep 28, 2020

TheNeuralBit commented Sep 29, 2020

TheNeuralBit commented Sep 29, 2020

emilymye commented Sep 30, 2020

TheNeuralBit commented Sep 30, 2020

lostluck commented Sep 30, 2020 via email

	COPY target/LICENSE /opt/apache/beam/
	COPY target/NOTICE /opt/apache/beam/

	apache/beam_java_8sdk latest 16ca619d489e 2 weeks ago 550MB
	apache/beam_java8_sdk latest 16ca619d489e 2 weeks ago 550MB

[BEAM-8106] Add version to java container image name #12505

[BEAM-8106] Add version to java container image name #12505

Conversation

emilymye commented Aug 7, 2020 • edited

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

emilymye commented Aug 11, 2020

emilymye commented Aug 11, 2020

codecov bot commented Sep 1, 2020 • edited

Codecov Report

kennknowles commented Sep 11, 2020

emilymye commented Sep 11, 2020

emilymye commented Sep 14, 2020

emilymye commented Sep 14, 2020

TheNeuralBit commented Sep 15, 2020

TheNeuralBit commented Sep 15, 2020

TheNeuralBit commented Sep 15, 2020

TheNeuralBit left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emilymye commented Sep 16, 2020

emilymye commented Sep 17, 2020

emilymye commented Sep 18, 2020

emilymye commented Sep 18, 2020

TheNeuralBit left a comment

Choose a reason for hiding this comment

emilymye commented Sep 28, 2020

TheNeuralBit commented Sep 29, 2020

TheNeuralBit commented Sep 29, 2020

emilymye commented Sep 30, 2020

TheNeuralBit commented Sep 30, 2020

lostluck commented Sep 30, 2020 via email

emilymye commented Aug 7, 2020 •

edited

codecov bot commented Sep 1, 2020 •

edited