test: add Retry Conformance Test JUnit framework (#939)

New JUnit test suite for running the Retry Conformance Test suite from
https://github.com/googleapis/conformance-tests.

Each way in which a specific api method can be invoked has a declared mapping in
RpcMethodMappings.

### Lifecycle
See google-cloud-storage/conformance-testing.md for a detailed explanation and
a sequence diagram of the lifecycle of running the retry conformance test suite.

### Components

#### TestBench
TestBench integrates the lifecycle of the storage-testbench into the JUnit tests.

The started docker container will use port forwarding as opposed to host networking
to allow ease of use on docker for macos.

#### RetryTestFixture
RetryTestFixture integrates and individual test with the TestBench. When a test
starts, a new `retry_test` resource will be registered with the test bench.
RetryTestFixture also takes on the responsibility of configuring the storage
client and any necessary headers needed to run an individual test.

#### GracefulConformanceEnforcement
When running in CI we don't want conformance tests which are not expected to
pass to result in a failed build. GracefulConformanceEnforcement allows for a
list of those test names which are expected to pass, and will fail in CI if a
regression is detected.

#### RpcMethodMapping
RpcMethodMapping provides the means of mapping an RpcMethod to a series of
method invocations. These method invocations are those public api methods from
com.google.cloud.storage.* which customers use.

RpcMethod defines a series of enums and mappings between strings and storage RPC
method names.

RpcMethodMappings provides the location for mappings between rpc method and
method invocations to be defined individually and then pulled together during
construction.

Functions, Ctx & State provide the "primitive" types which are used to build
mappings, and provide the means of running tests in a parallel friendly manner.

#### TestRetryConformance
TestRetryConformance represents the base configuration for a single test instance
after resolution against entries from com/google/cloud/conformance/storage/v1/retry_tests.json
along with a particular RpcMethodMapping.

Unique names will be generated and available for buckets, objects to allow
non-conflicting parallel test execution.

#### ParallelParameterized
A custom org.junit.runners.Parameterized test runner which provide a custom
scheduler to run tests in parallel.

The number of tests ran in parallel is derived based on the number of available
cores.

#### ITRetryConformanceTest
ITRetryConformanceTest loads up the test definition from
com/google/cloud/conformance/storage/v1/retry_tests.json permutes the definitions
against the mappings defined in RpcMethodMappings then runs each individual test.

Uses TestBench, GracefulConformanceEnforcement, RetryTestFixture and creates a
Ctx, runs setup, runs the test, runs teardown.

google-cloud-storage/assets/retry-conformance-tests-diagram.txt

@@ -0,0 +1,59 @@
+# This is a text representation of retry-conformance-tests.diagram.png generated
+# using https://www.websequencediagrams.com/
+
+participant ITRetryConformanceTest
+participant ITRetryConformanceTest.Static
+participant RetryTestCaseResolver
+participant GracefulConformanceEnforcement
+participant RetryTestFixture
+participant TestBench
+participant Docker
+participant RpcMethodMappings
+
+ITRetryConformanceTest->+ITRetryConformanceTest.Static: testCases
+    ITRetryConformanceTest.Static->RpcMethodMappings: <init>
+    ITRetryConformanceTest.Static->+RetryTestCaseResolver: getRetryTestCases
+        RetryTestCaseResolver->RetryTestCaseResolver: loadRetryTestDefinitions
+        RetryTestCaseResolver->RetryTestCaseResolver: generateTestCases
+        RetryTestCaseResolver->RetryTestCaseResolver: shuffle
+        RetryTestCaseResolver->RetryTestCaseResolver: validateGeneratedTestCases
+    RetryTestCaseResolver->-ITRetryConformanceTest.Static:
+ITRetryConformanceTest.Static->-ITRetryConformanceTest:
+
+ITRetryConformanceTest->+TestBench: apply
+    TestBench->TestBench: mktemp stdout
+    TestBench->TestBench: mktemp stderr
+    TestBench->+Docker: pull
+    Docker->-TestBench:
+    TestBench->+Docker: run
+      TestBench->+TestBench: await testbench up
+        TestBench->+Docker: GET /retry_tests
+        Docker->-TestBench:
+        deactivate TestBench
+        loop forEach test
+          ITRetryConformanceTest->+GracefulConformanceEnforcement: apply
+              ITRetryConformanceTest->+RetryTestFixture: apply
+                  RetryTestFixture->+TestBench: createRetryTest
+                    TestBench->+Docker: POST /retry_test
+                    Docker->-TestBench:
+                  TestBench->-RetryTestFixture:
+                  ITRetryConformanceTest->ITRetryConformanceTest: test
+                  RetryTestFixture->+TestBench: getRetryTest
+                    TestBench->+Docker: GET /retry_test/{id}
+                    Docker->-TestBench:
+                  TestBench->-RetryTestFixture:
+                  RetryTestFixture->RetryTestFixture: assert completion
+                  RetryTestFixture->+TestBench: deleteRetryTest
+                    TestBench->+Docker: DELETE /retry_test/{id}
+                    Docker->-TestBench:
+                  TestBench->-RetryTestFixture:
+              RetryTestFixture->-ITRetryConformanceTest:
+              opt if running in CI
+                  GracefulConformanceEnforcement->GracefulConformanceEnforcement: check allow list
+              end
+          GracefulConformanceEnforcement->-ITRetryConformanceTest:
+        end
+    Docker->-TestBench: docker stop
+    TestBench->TestBench: rmtemp stdout
+    TestBench->TestBench: rmtemp stderr
+TestBench->-ITRetryConformanceTest:

google-cloud-storage/conformance-testing.md

@@ -0,0 +1,45 @@
+# Conformance Testing
+
+This library leverages the conformance tests defined in [googleapis/conformance-tests](https://github.com/googleapis/conformance-tests)
+to ensure adherence to expected behaviors.
+
+Access to the conformance tests is achieved via dependencies on 
+[`com.google.cloud:google-cloud-conformance-tests`](https://github.com/googleapis/java-conformance-tests)
+which contains all generated protos and associated files necessary for loading
+and accessing the tests.
+
+## Running the Conformance Tests
+
+Conformance tests are written and run as part of the JUnit tests suite.
+
+## Suites
+
+### Automatic Retries
+
+The JUnit tests class is [`ITRetryConformanceTest.java`](./src/test/java/com/google/cloud/storage/conformance/retry/ITRetryConformanceTest.java)
+and is considered part of the integration test suite.
+
+This tests suite ensures that automatic retries for operations are properly defined
+and handled to ensure data integrity.
+
+#### Prerequisites
+1. Java 8+
+2. Maven
+3. Docker (Docker for MacOS has been tested and verified to work as well)
+
+#### Test Suite Overview
+
+The test suite uses the [storage-testbench](https://github.com/googleapis/storage-testbench)
+to configure and generate tests cases which use fault injection to ensure conformance.
+
+`ITRetryConformanceTest` encapsulates all the necessary lifecycle points needed
+to run the test suite, including:
+1. Running the testbench server via docker
+2. Setup, validation, cleanup of individual test cases with the testbench
+3. CI Graceful enforcement of test failures (enforce no regressions, but allow 
+   for some cases to not pass without failing the whole run)
+
+A sequence diagram of how the tests are loaded run, and interact with testbench
+can be seen below. Time moves from top to bottom, while component interactions
+are shown via arrows laterally.
+![](./assets/retry-conformance-tests-diagram.png)

google-cloud-storage/pom.xml

@@ -172,6 +172,11 @@
       <artifactId>httpcore</artifactId>
       <scope>test</scope>
     </dependency>
+    <dependency>
+      <groupId>com.google.errorprone</groupId>
+      <artifactId>error_prone_annotations</artifactId>
+      <scope>test</scope>
+    </dependency>
   </dependencies>
 
   <build>

google-cloud-storage/src/test/java/com/google/cloud/storage/PackagePrivateMethodWorkarounds.java

@@ -0,0 +1,41 @@
+/*
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.google.cloud.storage;
+
+import com.google.cloud.storage.BucketInfo.BuilderImpl;
+
+/**
+ * Several classes in the High Level Model for storage include package-local constructors and
+ * methods. For conformance testing we don't want to exist in the com.google.cloud.storage package
+ * to ensure we're interacting with the public api, however in a few select cases we need to change
+ * the instance of {@link Storage} which an object holds on to. The utilities in this class allow us
+ * to perform these operations.
+ */
+public final class PackagePrivateMethodWorkarounds {
+
+  private PackagePrivateMethodWorkarounds() {}
+
+  public static Bucket bucketCopyWithStorage(Bucket b, Storage s) {
+    BucketInfo.BuilderImpl builder = (BuilderImpl) BucketInfo.fromPb(b.toPb()).toBuilder();
+    return new Bucket(s, builder);
+  }
+
+  public static Blob blobCopyWithStorage(Blob b, Storage s) {
+    BlobInfo.BuilderImpl builder = (BlobInfo.BuilderImpl) BlobInfo.fromPb(b.toPb()).toBuilder();
+    return new Blob(s, builder);
+  }
+}

google-cloud-storage/src/test/java/com/google/cloud/storage/conformance/retry/CleanupStrategy.java

@@ -0,0 +1,23 @@
+/*
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.google.cloud.storage.conformance.retry;
+
+enum CleanupStrategy {
+  ALWAYS,
+  ONLY_ON_SUCCESS,
+  NEVER
+}

google-cloud-storage/src/test/java/com/google/cloud/storage/conformance/retry/Ctx.java

@@ -0,0 +1,78 @@
+/*
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.google.cloud.storage.conformance.retry;
+
+import com.google.cloud.storage.Storage;
+import com.google.cloud.storage.conformance.retry.Functions.EConsumer;
+import com.google.cloud.storage.conformance.retry.Functions.EFunction;
+import com.google.errorprone.annotations.Immutable;
+
+/**
+ * A simple context object used to track an instance of {@link Storage} along with {@link State} and
+ * provide some convenience methods for creating new instances.
+ */
+@Immutable
+final class Ctx {
+
+  private final Storage storage;
+  private final State state;
+
+  private Ctx(Storage s, State t) {
+    this.storage = s;
+    this.state = t;
+  }
+
+  /** Create a new instance of {@link Ctx} */
+  static Ctx ctx(Storage storage, State state) {
+    return new Ctx(storage, state);
+  }
+
+  public Storage getStorage() {
+    return storage;
+  }
+
+  public State getState() {
+    return state;
+  }
+
+  /**
+   * Create a new instance of {@link Ctx} by first applying {@code f} to {@code this.storage}.
+   * {@code this.state} is passed along unchanged.
+   */
+  public Ctx leftMap(EFunction<Storage, Storage> f) throws Throwable {
+    return new Ctx(f.apply(storage), state);
+  }
+
+  /**
+   * Create a new instance of {@link Ctx} by first applying {@code f} to {@code this.state}. {@code
+   * this.storage} is passed along unchanged.
+   */
+  public Ctx map(EFunction<State, State> f) throws Throwable {
+    return new Ctx(storage, f.apply(state));
+  }
+
+  /**
+   * Apply {@code f} by providing {@code this.state}.
+   *
+   * <p>This method is provided as convenience for those methods which have void return. In general
+   * {@link Ctx#map(EFunction)} should be used.
+   */
+  public Ctx peek(EConsumer<State> f) throws Throwable {
+    f.consume(state);
+    return this;
+  }
+}