feat: add dependency versions script #741

salehsquared · 2020-07-24T19:22:41Z

Related to #735

Adds in two scripts to be used as presubmit checks:
(1) deps-finder.sh
If added as a presubmit check, this will first check if the associated PR was in the format of "deps: update...". If this is not the case, the script will return success and end. If a google-cloud dependency that is meant to use java-shared-dependencies was updated, then the script will check if the updated dependency has the most recent version of java-shared-dependencies. If it does not, the script will fail. In all other cases (e.g. something that is not meant to use java-shared-dependencies), the script will pass.

(2) release-versions.sh
If added as a presubmit check, this will check if the PR was made on a non-SNAPSHOT release branch. If it was not, the script will return success and end. Otherwise, it will check our dependencyManagement list for any client libraries that utilize java-shared-dependencies.
If at least one client library is meant to use java-shared-dependencies and does not have the latest version of java-shared-dependencies, the script will fail. Currently there are many client libraries in this category, and they are all summarized by the script, including which version of shared-dependencies they use.

Summary of the current state of the client libraries (master branch, checked 07/09):
Number of incorrect dependencies found: 50. Number of correct dependencies: 11.

Edit (07/10): java-storage-nio is no longer incorrectly tagged, and is thus no longer blocking the script.

Adding yaml for versions script.

Compares the latest version of google-cloud-shared-config to the parent of an added dependency

Permission denied error when running without sudo

Checking for functionality of versions script.

check

yaml

… to v0.118.1-alpha

…o v1.35.2

… specific client libraries and the other checks all client libraries

salehsquared · 2020-07-29T20:29:38Z

@chingor13 Completed updates on scripts per our conversation on Monday. Interested to hear your thoughts!

.kokoro/deps-finder.sh

google-cla · 2020-08-25T01:08:55Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

stephaniewang526 · 2020-08-25T01:17:27Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).

The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.

The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

@salehsquared -- please sigh cla since you are external contributor now. Thanks!

google-cla · 2020-08-25T01:18:16Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

salehsquared · 2020-08-26T19:15:17Z

@googlebot I signed it!

dependency-convergence/src/test/java/RecentCommitCheckTest.java

Fixing grammar

chingor13 · 2021-02-23T22:30:28Z

.kokoro/shared-dependencies-version.sh

+  pushd "${scriptDir}/../dependency-convergence"
+  mvn clean install -DskipTests
+  # Single quotes around double quotes so that the branch name is counted as only one argument
+  mvn exec:java -Dexec.args="'${runRelease}'"


I would suggest we break these into separate checks (rather than re-using the same check). We can run these from GitHub actions and skip if the commit does not match what we expect.

chingor13 · 2021-02-23T22:32:09Z

...ndency-convergence/src/main/java/com/google/cloud/cloudbomcommitcheck/RecentCommitCheck.java

+        .generateArtifactData(latestSharedDependencies);
+    String latestSharedDependenciesVersion = sharedDependenciesData.getLatestVersion();
+    if (latestSharedDependenciesVersion == null || latestSharedDependenciesVersion.isEmpty()) {
+      return 2;


These magic integers should either be enums or perhaps thrown exceptions

chingor13 · 2021-02-23T22:33:08Z

...ndency-convergence/src/main/java/com/google/cloud/cloudbomcommitcheck/RecentCommitCheck.java

+      return 1;
+    }
+    // A release PR was found
+    Arguments arguments = Arguments.readCommandLine("-f ../pom.xml");


This is very odd usage of command line parsing. Instead, you should be able to instantiate the Bom instance yourself. It's also odd to hard code this value.

chingor13 · 2021-02-23T22:34:58Z

...ndency-convergence/src/main/java/com/google/cloud/cloudbomcommitcheck/RecentCommitCheck.java

+    System.out.println("Total dependencies checked: " + clientLibraries.size());
+  }
+
+  @VisibleForTesting


This is marked visible for testing but is not tested directly.

chingor13 · 2021-02-23T22:36:44Z

dependency-convergence/src/main/java/com/google/cloud/cloudbomcommitcheck/ArtifactData.java

+   * google-cloud-shared-dependencies was not in the POM file, null if the POM file could not be
+   * found
+   */
+  private static String getSharedDependenciesVersion(String pomUrl) {


Much of this code seems to share functionality with the dashboard, should we combine the artifacts and share logic?

chingor13 · 2021-02-23T22:37:09Z

dependency-convergence/src/main/java/com/google/cloud/cloudbomcommitcheck/ArtifactData.java

+      }
+      for (Dependency dependency : model.getDependencyManagement()
+          .getDependencies()) {
+        if ("com.google.cloud".equals(dependency.getGroupId()) && "google-cloud-shared-dependencies"


It'd be nice if we tested this parsing.

stephaniewang526 · 2021-03-24T21:05:22Z

Closing as it is addressed in #1752

salehsquared and others added 17 commits July 24, 2020 14:13

Update ci.yaml

945f7d1

Adding yaml for versions script.

Check version of google-cloud-shared-config

d80399b

Compares the latest version of google-cloud-shared-config to the parent of an added dependency

Update pom.xml

afc4204

Update ci.yaml

98282ff

Permission denied error when running without sudo

Update pom.xml

1861d3b

Checking for functionality of versions script.

Update ci.yaml

438e5fa

check

Update ci.yaml

02e35fb

yaml

Updating permissions on versions script

77be68a

deps: update dependency com.google.cloud:google-cloud-logging-logback…

ed61410

… to v0.118.1-alpha

deps: update dependency com.google.cloud:google-cloud-firestore-bom t…

9f1fb9a

…o v1.35.2

Reverting pom.xml after testing changes

e224876

Merge branch 'master' of https://github.com/salehsquared/java-cloud-bom

0edb0e5

Reverting POM version

ed33def

Reverting POM version

b1fe397

Fixing formatting change on ci file

14c43ec

Fixing formatting change on ci file

1150208

Fix for formatting end of file

2616081

google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Jul 24, 2020

stephaniewang526 requested review from chingor13 and stephaniewang526 July 24, 2020 19:35

Saleh Mostafa added 2 commits July 29, 2020 16:09

Updated dependencies scripts. Separated out, so one script checks for…

b164f1a

… specific client libraries and the other checks all client libraries

Setting ci.yaml file back to original

1ad1ae1