Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: confirm owlbot-copy succeeeded to transfer java files #2235

Merged
merged 4 commits into from
Nov 10, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 21 additions & 16 deletions library_generation/test/generate_library_integration_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ fi
grep -v '^ *#' < "${proto_path_list}" | while IFS= read -r line; do
proto_path=$(echo "$line" | cut -d " " -f 1)
repository_path=$(echo "$line" | cut -d " " -f 2)
is_handwritten=$(echo "$line" | cut -d " " -f 3)
skip_postprocessing=$(echo "$line" | cut -d " " -f 3)
# parse destination_path
pushd "${output_folder}"
echo "Checking out googleapis-gen repository..."
Expand All @@ -110,8 +110,8 @@ grep -v '^ *#' < "${proto_path_list}" | while IFS= read -r line; do
service_yaml=${service_yaml},
include_samples=${include_samples}."
pushd "${output_folder}"
if [ "${is_handwritten}" == "true" ]; then
echo 'this is a handwritten library'
if [ "${skip_postprocessing}" == "true" ]; then
echo 'this library is not intended for postprocessing test'
popd # output folder
continue
else
Expand All @@ -126,7 +126,7 @@ grep -v '^ *#' < "${proto_path_list}" | while IFS= read -r line; do
# generate GAPIC client library
echo "Generating library from ${proto_path}, to ${destination_path}..."
generation_start=$(date "+%s")
if [ $enable_postprocessing == "true" ]; then
if [ "${enable_postprocessing}" == "true" ]; then
if [[ "${repository_path}" == "null" ]]; then
# we need a repository to compare the generated results with. Skip this
# library
Expand Down Expand Up @@ -173,9 +173,14 @@ grep -v '^ *#' < "${proto_path_list}" | while IFS= read -r line; do
echo "Generate library finished."
echo "Compare generation result..."
if [ $enable_postprocessing == "true" ]; then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "${enable_postprocessing}"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I added this and a couple other format fixes

if [ $(find "${output_folder}/workspace" -name '*.java' | wc -l) -eq 0 ];
then
echo 'no java files found in workspace. This probably means that owlbot copy failed'
exit 1
fi
echo "Checking out repository..."
pushd "${target_folder}"
SOURCE_DIFF_RESULT=0
source_diff_result=0
git diff \
--ignore-space-at-eol \
-r \
Expand All @@ -184,34 +189,34 @@ grep -v '^ *#' < "${proto_path_list}" | while IFS= read -r line; do
':!*pom.xml' \
':!*README.md' \
':!*package-info.java' \
|| SOURCE_DIFF_RESULT=$?
|| source_diff_result=$?

POM_DIFF_RESULT=$(compare_poms "${target_folder}")
pom_diff_result=$(compare_poms "${target_folder}")
popd # target_folder
if [[ ${SOURCE_DIFF_RESULT} == 0 ]] && [[ ${POM_DIFF_RESULT} == 0 ]] ; then
if [[ ${source_diff_result} == 0 ]] && [[ ${pom_diff_result} == 0 ]] ; then
echo "SUCCESS: Comparison finished, no difference is found."
# Delete google-cloud-java to allow a sparse clone of the next library
rm -rdf google-cloud-java
elif [ ${SOURCE_DIFF_RESULT} != 0 ]; then
elif [ ${source_diff_result} != 0 ]; then
echo "FAILURE: Differences found in proto path: ${proto_path}."
exit "${SOURCE_DIFF_RESULT}"
elif [ ${POM_DIFF_RESULT} != 0 ]; then
elif [ ${pom_diff_result} != 0 ]; then
echo "FAILURE: Differences found in generated poms"
exit "${POM_DIFF_RESULT}"
exit "${pom_diff_result}"
fi
elif [ $enable_postprocessing == "false" ]; then
elif [ "${enable_postprocessing}" == "false" ]; then
# include gapic_metadata.json and package-info.java after
# resolving https://github.com/googleapis/sdk-platform-java/issues/1986
SOURCE_DIFF_RESULT=0
source_diff_result=0
diff --strip-trailing-cr -r "googleapis-gen/${proto_path}/${destination_path}" "${output_folder}/${destination_path}" \
-x "*gradle*" \
-x "gapic_metadata.json" \
-x "package-info.java" || SOURCE_DIFF_RESULT=$?
if [ ${SOURCE_DIFF_RESULT} == 0 ] ; then
-x "package-info.java" || source_diff_result=$?
if [ ${source_diff_result} == 0 ] ; then
echo "SUCCESS: Comparison finished, no difference is found."
else
echo "FAILURE: Differences found in proto path: ${proto_path}."
exit "${SOURCE_DIFF_RESULT}"
exit "${source_diff_result}"
fi
fi

Expand Down
5 changes: 3 additions & 2 deletions library_generation/test/resources/proto_path_list.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# This file is used in integration test against `generate_library.sh`.
# Format:
# proto_path repository_path is_handwritten
# proto_path repository_path skip_postprocessing_test
# google/bigtable/admin/v2 java-bigtable true
# google/bigtable/v2 java-bigtable true
google/cloud/apigeeconnect/v1 java-apigee-connect false
Expand All @@ -17,7 +17,8 @@ google/cloud/redis/v1 java-redis false
google/cloud/redis/v1beta1 java-redis false
# google/example/library/v1 google-cloud-example-library-v1-java null false
google/devtools/containeranalysis/v1 java-containeranalysis false
google/iam/v1 java-iam false
google/iam/v1 java-iam true
google/iam/v2 java-iam false
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add another line for google/iam/v2 because I use google/iam/v1 to test proto-only library.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll disable v1 for the post-processing IT and add v2 separately. The iam owlbot yaml only supports v2 and v2beta1 so using v1 will fail to move any java files into the workspace

google/iam/credentials/v1 java-iamcredentials false
google/logging/v2 java-logging true
google/pubsub/v1 java-pubsub true
Expand Down
Loading