feat: enable generation with postprocessing of multiple service versions #2342
Conversation
|
I disabled the https://github.com/googleapis/sdk-platform-java/actions/runs/7674168459/job/20918240804 There is some retry code problem, and I'm clueless. @JoeWang1127 could this be some missing additional proto problem?
|
I think the issue is that the grpc_service_config in BUILD has a leading ":". This case is considered when parsing gapic option using shell script (link). However, I didn't add the similar logic in the corresponding python script. |
@JoeWang1127 it was indeed a problem with a colon in the service config. It's now fixed, thanks! |
| #Optional. The root folder name of generated client libraries. If empty, modules will be created under current folder, useful for single module | ||
| destination-path: google-cloud-java | ||
| synthtool_commitish: fac8444edd5f5526e804c306b766a271772a3e2f | ||
| #Required. The root folder name of generated client libraries. The value "google-cloud-java" implies special treatment in the lower level scripts |
There was a problem hiding this comment.
Does "google-cloud-java" still implies special treatment in the lower level scripts?
There was a problem hiding this comment.
It's more precise to say "more than 1 library". I modified the comments of that yaml to explain this.
library_generation/utilities.sh
Outdated
| # inspects a $build_file for a certain $rule (e.g. java_gapic_library). If the | ||
| # first 15 lines after the declaration of the rule contain $pattern, then | ||
| # it will return $if_match if $pattern is found, otherwise $default | ||
| __get_config_from_BUILD() { |
There was a problem hiding this comment.
Thanks, I think all the parsing utils in this class can be removed now?
|
|
re-enables the changes from #2342 as part of [milestone 3.2](https://docs.google.com/document/d/1v-sJBmdNEVBRryL8n90XK8OIiqVphaJemXVhANw14Jk/edit?pli=1&resourcekey=0-QGUb2do-JBlDWKvptWBp_g&tab=t.0#bookmark=id.kl5fzqdav6u7) of Hermetic Build ### Changes: - add support for HW libs in `integration_tests.py` - adapt configuration of docker volumes - adapt usage of `fix-poms.py` to use monorepo and split_repo modes - fix temp_destination_path in `generate_composed_library.py` in order to avoid collisions - For example `google/bigtable/v2` and `google/bigtable/admin/v2` would both be sent to a temporal `java-bigtable-v2` folder, spoiling the files being transferred via `copy-code`. - The fix is to have a `proto_path`-based folder name to achieve uniqueness ### After this PR: - [ ] Reconfigure the nightly CI in google-cloud-java to use the updated volume mapping
context: enabling postprocessing for HW libraries
Changes in this PR
This PR provides a
generate_composed_library.pythat will callgenerate_library.shfor a series of GAPICs as defined in a configuration yaml. All of these calls are without postprocessing. The script will then organize a pre-processed-sources folder that will be used for a single final postprocessing run. The resulting post-processed combined library will then be transferred to the location specified in--repository_path(if--repository_pathis not specified, then the repo will be downloaded into theoutputfolder).What does this change allow?
This achieves generation for libraries that are composed of more than one service or service version (e.g. bigtable which is composed of two services
google/bigtable/v2andgoogle/bigtable/admin/v2, dialogflow which is composed of two service versionsgoogle/cloud/dialogflow/v2andgoogle/cloud/dialogflow/v2beta1), with its root problem being owlbot.py files that assume several service versions existing in theowl-bot-stagingfolder.Generating a set of libraries with their respective GAPICs from a single configuration yaml
This PR enhances the list of libraries in the integration test
generation_config.yamlsdk-platform-java/library_generation/test/resources/integration/google-cloud-java/generation_config.yaml
Lines 51 to 61 in f0b26ef
They take advantage of common configuration entries at the root of the configuration yaml
Trying it locally
For example, we will clone
google-cloud-javain/tmp......and remove a few source files from
java-assetThen we use the test configuration yaml (for convenience I only defined the
assetservice to speed up) to generate a few libraries, including all versions of assetThe resulting generation only modified the
README.mdfile (which is ignored in our integration tests)