Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up LocalGcdHelper and docs #821

Merged
merged 10 commits into from
Apr 8, 2016
59 changes: 34 additions & 25 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,55 +11,64 @@ This library provides tools to help write tests for code that uses the following

#### On your machine

You can test against a temporary local datastore by following these steps:
You can test against a temporary local Datastore by following these steps:

1. Start the local datastore emulator using `LocalGcdHelper`. This can be done in two ways:
- Run `LocalGcdHelper.java`'s `main` method with arguments `START` and (optionally) `--port=<port number>`. This will create a temporary folder on your computer and bind `localhost:<port number>` for communication with the local datastore. The port number is an optional argument. If no port number is specified, port 8080 will be used.
- Call `LocalGcdHelper.start(<project ID>, <port number>)` before running your tests. Save the `LocalGcdHelper` object returned so that you can stop the emulator later.
1. Start the local Datastore emulator before running your tests using `LocalDatastoreHelper`'s `create` and `start` methods. This will create a temporary folder on your computer and bind a port for communication with the local Datastore. There is an optional argument for `create`: consistency. The consistency setting controls the fraction of Datastore writes that are immediately visible in global queries.
```java
// Use the default consistency setting of 0.9
LocalDatastoreHelper helper = LocalDatastoreHelper.create();
// or explicitly set the consistency
helper = LocalDatastoreHelper.create(0.6);

helper.start(); // Starts the local Datastore emulator in a separate process
```

2. In your program, create and use a datastore whose host is set host to `localhost:<port number>`. For example,
2. Create and use a `Datastore` object with the options given by the `LocalDatastoreHelper` instance. For example:
```java
DatastoreOptions options = DatastoreOptions.builder()
.projectId(PROJECT_ID)
.host("http://localhost:8080")
.build();
Datastore localDatastore = options.service();
Datastore localDatastore = helper.options().service();
```

3. Run your tests.

4. Stop the local datastore emulator.
- If you ran `LocalGcdHelper.java`'s `main` function to start the emulator, run `LocalGcdHelper.java`'s `main` method with arguments `STOP` and (optionally) `--port=<port number>`. If the port is not supplied, the program will attempt to close the last port started.
- If you ran `LocalGcdHelper.start()` to start the emulator, call the `stop()` method on the `LocalGcdHelper` object returned by `LocalGcdHelper.start()`.
4. Stop the local datastore emulator by calling the `stop()` method, like so:
```java
helper.stop();
```

#### On a remote machine

You can test against a remote datastore emulator as well. To do this, set the `DatastoreOptions` project endpoint to the hostname of the remote machine, like the example below.
You can test against a remote Datastore emulator as well. To do this, set the `DatastoreOptions` project endpoint to the hostname of the remote machine, like the example below.

```java
DatastoreOptions options = DatastoreOptions.builder()
.projectId(PROJECT_ID)
.projectId("my-project-id") // must match project ID specified on remote machine
.host("http://<hostname of machine>:<port>")
.authCredentials(AuthCredentials.noAuth())
.build();
Datastore localDatastore = options.service();
```

Note that the remote datastore must be running before your tests are run.
We recommend that you start the emulator on the remote machine using the [Google Cloud SDK](https://cloud.google.com/sdk/gcloud/reference/beta/emulators/datastore/) from command line, as shown below:

```
gcloud beta emulators datastore start --host-port <hostname of machine>:<port>
```

### Testing code that uses Storage

Currently, there isn't an emulator for Google Cloud Storage, so an alternative is to create a test project. `RemoteGcsHelper` contains convenience methods to make setting up and cleaning up the test project easier. To use this class, follow the steps below:
Currently, there isn't an emulator for Google Cloud Storage, so an alternative is to create a test project. `RemoteStorageHelper` contains convenience methods to make setting up and cleaning up the test project easier. To use this class, follow the steps below:

1. Create a test Google Cloud project.

2. Download a JSON service account credentials file from the Google Developer's Console. See more about this on the [Google Cloud Platform Storage Authentication page][cloud-platform-storage-authentication].

3. Create a `RemoteGcsHelper` object using your project ID and JSON key.
Here is an example that uses the `RemoteGcsHelper` to create a bucket.
3. Create a `RemoteStorageHelper` object using your project ID and JSON key.
Here is an example that uses the `RemoteStorageHelper` to create a bucket.
```java
RemoteGcsHelper gcsHelper =
RemoteGcsHelper.create(PROJECT_ID, new FileInputStream("/path/to/my/JSON/key.json"));
Storage storage = gcsHelper.options().service();
String bucket = RemoteGcsHelper.generateBucketName();
RemoteStorageHelper helper =
RemoteStorageHelper.create(PROJECT_ID, new FileInputStream("/path/to/my/JSON/key.json"));
Storage storage = helper.options().service();
String bucket = RemoteStorageHelper.generateBucketName();
storage.create(BucketInfo.of(bucket));
```

Expand All @@ -68,7 +77,7 @@ Here is an example that uses the `RemoteGcsHelper` to create a bucket.
5. Clean up the test project by using `forceDelete` to clear any buckets used.
Here is an example that clears the bucket created in Step 3 with a timeout of 5 seconds.
```java
RemoteGcsHelper.forceDelete(storage, bucket, 5, TimeUnit.SECONDS);
RemoteStorageHelper.forceDelete(storage, bucket, 5, TimeUnit.SECONDS);
```

### Testing code that uses Resource Manager
Expand Down Expand Up @@ -134,4 +143,4 @@ Here is an example that clears the dataset created in Step 3.
```

[cloud-platform-storage-authentication]:https://cloud.google.com/storage/docs/authentication?hl=en#service_accounts
[create-service-account]:https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount
[create-service-account]:https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@
import com.google.gcloud.storage.BlobInfo;
import com.google.gcloud.storage.BucketInfo;
import com.google.gcloud.storage.Storage;
import com.google.gcloud.storage.testing.RemoteGcsHelper;
import com.google.gcloud.storage.testing.RemoteStorageHelper;

import org.junit.AfterClass;
import org.junit.BeforeClass;
Expand Down Expand Up @@ -135,7 +135,7 @@ public class ITBigQueryTest {
private static final String LOAD_FILE = "load.csv";
private static final String JSON_LOAD_FILE = "load.json";
private static final String EXTRACT_FILE = "extract.csv";
private static final String BUCKET = RemoteGcsHelper.generateBucketName();
private static final String BUCKET = RemoteStorageHelper.generateBucketName();
private static final TableId TABLE_ID = TableId.of(DATASET, "testing_table");
private static final String CSV_CONTENT = "StringValue1\nStringValue2\n";
private static final String JSON_CONTENT = "{"
Expand Down Expand Up @@ -172,9 +172,9 @@ public class ITBigQueryTest {
@BeforeClass
public static void beforeClass() throws InterruptedException {
RemoteBigQueryHelper bigqueryHelper = RemoteBigQueryHelper.create();
RemoteGcsHelper gcsHelper = RemoteGcsHelper.create();
RemoteStorageHelper storageHelper = RemoteStorageHelper.create();
bigquery = bigqueryHelper.options().service();
storage = gcsHelper.options().service();
storage = storageHelper.options().service();
storage.create(BucketInfo.of(BUCKET));
storage.create(BlobInfo.builder(BUCKET, LOAD_FILE).contentType("text/plain").build(),
CSV_CONTENT.getBytes(StandardCharsets.UTF_8));
Expand All @@ -200,7 +200,7 @@ public static void afterClass() throws ExecutionException, InterruptedException
RemoteBigQueryHelper.forceDelete(bigquery, DATASET);
}
if (storage != null) {
boolean wasDeleted = RemoteGcsHelper.forceDelete(storage, BUCKET, 10, TimeUnit.SECONDS);
boolean wasDeleted = RemoteStorageHelper.forceDelete(storage, BUCKET, 10, TimeUnit.SECONDS);
if (!wasDeleted && LOG.isLoggable(Level.WARNING)) {
LOG.log(Level.WARNING, "Deletion of bucket {0} timed out, bucket is not empty", BUCKET);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -339,7 +339,7 @@ public RestorableState<AuthCredentials> capture() {
/**
* A placeholder for credentials to signify that requests sent to the server should not be
* authenticated. This is typically useful when using the local service emulators, such as
* {@code LocalGcdHelper} and {@code LocalResourceManagerHelper}.
* {@code LocalDatastoreHelper} and {@code LocalResourceManagerHelper}.
*/
public static class NoAuthCredentials extends AuthCredentials {

Expand Down Expand Up @@ -425,7 +425,7 @@ public static ServiceAccountAuthCredentials createFor(String account, PrivateKey

/**
* Creates a placeholder denoting that no credentials should be used. This is typically useful
* when using the local service emulators, such as {@code LocalGcdHelper} and
* when using the local service emulators, such as {@code LocalDatastoreHelper} and
* {@code LocalResourceManagerHelper}.
*/
public static AuthCredentials noAuth() {
Expand Down