feat: generate selected libraries #2598

JoeWang1127 · 2024-03-21T19:23:03Z

In this PR:

Generate a selected list of libraries based on library_name

This PR is a preparation for improve nightly generation.

Note that we need to change the new library generation workflow in google-cloud-java to accommodate this change because library_generation/generate_repo.py has replaced --target-library-api-shortname with --target-library-names.

diegomarquezp

LGTM

blakeli0 · 2024-03-22T19:01:06Z

library_generation/cli/generate_repo.py

So this is supposed to be the endtrypoint in the future? If it is, can we rename this file to avoid confusion? Otherwise I think keeping the cli together with the logic is fine.

So this is supposed to be the entrypoint in the future?

Yes.

We can change the name, then before combining generate_pr_description.py, we will have some like this in google-cloud-java

entrypoint.py ... // generate libraries generate_pr_description.py ... // generate PR description

Do you think this is fine?

Yep, I think that's better than having two generate_repo.py. In general, if you have a design in your mind but will not implement it immediately, it's better to define the interfaces first so that others are easy to follow.

OK, I think this (having two generate_repo.py is confusing) makes sense.

I decided to remove cli/generate_repo.py and keep the cli part in generate_repo.py (as before) and modified description accordingly.

I'll introduce entrypoint.py in follow PR.

blakeli0 · 2024-03-22T19:34:59Z

library_generation/generate_repo.py

+    :param target_library_names: library_name of target libraries.
+    :return:
+    """
+    if target_library_names is None:


What if the list is empty? Does it mean we generate nothing or generate everything?

What if the list is empty

We generate nothing.

The rational is if we pass a list, then the libraries will be selected from the list. Therefore, an empty list means no generation.

If we don't pass a list, the default value is None and we generate everything.

SG. Can you mention the behavior in the comment above? I wonder how it is going to be implemented in generate_repo.py later, maybe just no-op if the list is empty?

Adde comments.

I wonder how it is going to be implemented in generate_repo.py later, maybe just no-op if the list is empty?

It will not call generate_composed_library.py but it will do a repo level post processing, which is no-op if no library is changed.

sonarcloud · 2024-03-25T23:29:25Z

Quality Gate passed for 'gapic-generator-java-root'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

sonarcloud · 2024-03-25T23:32:27Z

Quality Gate passed for 'java_showcase_integration_tests'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

In todays's nightly generation job ([log](https://github.com/googleapis/google-cloud-java/actions/runs/8457767424/job/23170489394)), no actual generation is done. I think this is because `main` method has been removed by #2598.

In this PR: - Generate a selected list of libraries based on `library_name` This PR is a preparation for improve nightly generation. Note that we need to change the [new library generation workflow](https://github.com/googleapis/google-cloud-java/blob/main/.github/workflows/generate_new_client_hermetic_build.yaml) in google-cloud-java to accommodate this change because `library_generation/generate_repo.py` has replaced `--target-library-api-shortname` with `--target-library-names`.

In todays's nightly generation job ([log](https://github.com/googleapis/google-cloud-java/actions/runs/8457767424/job/23170489394)), no actual generation is done. I think this is because `main` method has been removed by #2598.

In this PR: - In the integration test, run `generate_repo.py` and `generate_pr_description.py` in docker container. - Remove test image in cloud build. Context: In this [log](https://github.com/googleapis/google-cloud-java/actions/runs/8445620169/job/23133097820), the nightly generation job only took 3 mins. After investigation, we found out that removing `main` method in `generate_repo.py` (#2598) caused the CLI is not running despite that the job returns successfully. This non-explicit failure demonstrated that there's difference between calling CLI (`generate` method in `generate_repo.py`) and its implementation (`generate_from_yaml` method in `generate_repo.py`). We decided to change the integration test to run the CLI in docker container, rather than testing `generate_from_yaml` because it is the containerized CLI will be used in downstream libraries.

🤖 I have created a release *beep* *boop* --- <details><summary>2.39.0</summary> ## [2.39.0](v2.38.1...v2.39.0) (2024-04-18) ### Features * add `libraries_bom_version` to generation configuration ([#2639](#2639)) ([56c7ca5](56c7ca5)) * Add ChannelPoolSettings Getter for gRPC's ChannelProvider ([#2612](#2612)) ([d0c5191](d0c5191)) * add config change ([#2604](#2604)) ([8312706](8312706)) * add entry point ([#2616](#2616)) ([b19fa33](b19fa33)) * add generation config comparator ([#2587](#2587)) ([a94c2f0](a94c2f0)) * Add JavadocJar Task to build.gradle for self service libraries ([#2593](#2593)) ([993f5ac](993f5ac)) * Client/StubSettings' getEndpoint() returns the resolved endpoint ([#2440](#2440)) ([4942bc1](4942bc1)) * generate selected libraries ([#2598](#2598)) ([739ddbb](739ddbb)) * Validate the Universe Domain inside Java-Core ([#2592](#2592)) ([35d789f](35d789f)) ### Bug Fixes * add main to `generate_repo.py` ([#2607](#2607)) ([fedeb32](fedeb32)) * correct deep-remove and deep-preserve regexes ([#2572](#2572)) ([4c7fd88](4c7fd88)) * first attempt should use the min of RPC timeout and total timeout ([#2641](#2641)) ([0349232](0349232)) * remove duplicated calls to AutoValue builders ([#2636](#2636)) ([53a3727](53a3727)) * remove unnecessary slf4j and AbstractGoogleClientRequest native image configs ([0cb7d0e](0cb7d0e)) * remove unnecessary slf4j and AbstractGoogleClientRequest native image configs ([#2628](#2628)) ([0cb7d0e](0cb7d0e)) ### Dependencies * update arrow.version to v15.0.2 ([#2589](#2589)) ([777acf3](777acf3)) * update dependency com.google.cloud.opentelemetry:detector-resources-support to v0.28.0 ([#2649](#2649)) ([e4ed176](e4ed176)) * update dependency gitpython to v3.1.41 [security] ([#2625](#2625)) ([e41bd8f](e41bd8f)) * update dependency net.bytebuddy:byte-buddy to v1.14.13 ([#2646](#2646)) ([73ac5a4](73ac5a4)) * update dependency org.threeten:threeten-extra to v1.8.0 ([#2650](#2650)) ([226325a](226325a)) * update dependency org.threeten:threetenbp to v1.6.9 ([#2602](#2602)) ([371753e](371753e)) * update dependency org.threeten:threetenbp to v1.6.9 ([#2665](#2665)) ([8935bc8](8935bc8)) * update google api dependencies ([#2584](#2584)) ([cd20604](cd20604)) * update googleapis/java-cloud-bom digest to 7071341 ([#2608](#2608)) ([8d74140](8d74140)) * update netty dependencies to v4.1.109.final ([#2597](#2597)) ([8990693](8990693)) * update opentelemetry-java monorepo to v1.37.0 ([#2652](#2652)) ([f8fa2e9](f8fa2e9)) * update protobuf dependencies to v3.25.3 ([#2491](#2491)) ([b0e5041](b0e5041)) * update slf4j monorepo to v2.0.13 ([#2647](#2647)) ([f030e29](f030e29)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

feat: generate selected libraries

5d4c972

product-auto-label bot added the size: m Pull request size is medium. label Mar 21, 2024

JoeWang1127 added 3 commits March 21, 2024 19:40

add custom option

c0a0a19

use get_library_name

f32e32b

use string with comma

fc128f9

JoeWang1127 added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 21, 2024

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 21, 2024

change comment

81a3edc

JoeWang1127 marked this pull request as ready for review March 21, 2024 21:52

JoeWang1127 requested a review from a team as a code owner March 21, 2024 21:52

JoeWang1127 requested review from blakeli0 and diegomarquezp March 21, 2024 21:53

JoeWang1127 added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 21, 2024

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 21, 2024

diegomarquezp approved these changes Mar 22, 2024

View reviewed changes

blakeli0 reviewed Mar 22, 2024

View reviewed changes

JoeWang1127 added 3 commits March 24, 2024 00:39

add unit tests

4ceb816

add comment

40dd5bf

remove cli

2327a61

JoeWang1127 requested review from diegomarquezp and blakeli0 March 25, 2024 23:06

JoeWang1127 added 2 commits March 25, 2024 23:15

Merge branch 'main' into feat/generate-selected-libraries

a9ba111

add comment

d2f02c6

blakeli0 approved these changes Mar 26, 2024

View reviewed changes

JoeWang1127 merged commit 739ddbb into main Mar 26, 2024
31 checks passed

JoeWang1127 deleted the feat/generate-selected-libraries branch March 26, 2024 11:50

release-please bot mentioned this pull request Mar 26, 2024

chore(main): release 2.39.0 #2594

Merged

JoeWang1127 mentioned this pull request Mar 26, 2024

chore: update new client generation workflow googleapis/google-cloud-java#10621

Merged

JoeWang1127 mentioned this pull request Mar 27, 2024

fix: add main to generate_repo.py #2607

Merged

JoeWang1127 mentioned this pull request Apr 2, 2024

chore: add integration test running in docker container #2613

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: generate selected libraries #2598

feat: generate selected libraries #2598

JoeWang1127 commented Mar 21, 2024 •

edited

diegomarquezp left a comment

blakeli0 Mar 22, 2024

JoeWang1127 Mar 22, 2024

blakeli0 Mar 25, 2024

JoeWang1127 Mar 25, 2024 •

edited

blakeli0 Mar 22, 2024

JoeWang1127 Mar 22, 2024

blakeli0 Mar 25, 2024

JoeWang1127 Mar 25, 2024 •

edited

sonarcloud bot commented Mar 25, 2024

sonarcloud bot commented Mar 25, 2024

feat: generate selected libraries #2598

feat: generate selected libraries #2598

Conversation

JoeWang1127 commented Mar 21, 2024 • edited

diegomarquezp left a comment

Choose a reason for hiding this comment

blakeli0 Mar 22, 2024

Choose a reason for hiding this comment

JoeWang1127 Mar 22, 2024

Choose a reason for hiding this comment

blakeli0 Mar 25, 2024

Choose a reason for hiding this comment

JoeWang1127 Mar 25, 2024 • edited

Choose a reason for hiding this comment

blakeli0 Mar 22, 2024

Choose a reason for hiding this comment

JoeWang1127 Mar 22, 2024

Choose a reason for hiding this comment

blakeli0 Mar 25, 2024

Choose a reason for hiding this comment

JoeWang1127 Mar 25, 2024 • edited

Choose a reason for hiding this comment

sonarcloud bot commented Mar 25, 2024

Quality Gate passed for 'gapic-generator-java-root'

sonarcloud bot commented Mar 25, 2024

Quality Gate passed for 'java_showcase_integration_tests'

JoeWang1127 commented Mar 21, 2024 •

edited

JoeWang1127 Mar 25, 2024 •

edited

JoeWang1127 Mar 25, 2024 •

edited