Skip to content

Add storage insights samples #8460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
minherz merged 8 commits into from
Aug 10, 2023
Merged

Add storage insights samples #8460

minherz merged 8 commits into from
Aug 10, 2023

Conversation

@JesseLovelace
Copy link
Member

@JesseLovelace JesseLovelace commented Jul 21, 2023

Adds samples for storage insights, intended for this page: https://cloud.google.com/storage/docs/insights/using-inventory-reports

@JesseLovelace JesseLovelace requested review from a team and yoshi-approver as code owners July 21, 2023 20:51
@snippet-bot
Copy link

snippet-bot bot commented Jul 21, 2023 •
edited
Loading

Here is the summary of changes.

You are about to add 10 region tags.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

  • Refresh this comment

@product-auto-label product-auto-label bot added samples Issues that are directly related to samples. api: storage Issues related to the Cloud Storage API. labels Jul 21, 2023
@frankyn frankyn requested review from danielbankhead and removed request for danielbankhead July 22, 2023 00:24
frankyn
frankyn suggested changes Jul 22, 2023
View reviewed changes
storageinsights/pom.xml Outdated
@@ -0,0 +1,78 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Copyright (c) 2015 Google Inc.
Copy link
Contributor

@frankyn frankyn Jul 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2023

.../main/java/com/google/cloud/storage/storageinsights/samples/CreateInventoryReportConfig.java Outdated
String sourceBucket = "your-source-bucket";

// The name of your Google Cloud Storage destination bucket
String destinationBucket = "your-destination=bucket";
Copy link
Contributor

@frankyn frankyn Jul 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

your-destination-bucket

...c/main/java/com/google/cloud/storage/storageinsights/samples/ListInventoryReportConfigs.java Outdated
String bucketLocation = "us-west-1";

listInventoryReportConfigs(projectId, bucketLocation);
}
Copy link
Contributor

@frankyn frankyn Jul 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newline

storageinsights/pom.xml Outdated

</dependencies>

</project> No newline at end of file
Copy link
Contributor

@frankyn frankyn Jul 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newline

.../main/java/com/google/cloud/storage/storageinsights/samples/CreateInventoryReportConfig.java Outdated

// [START storageinsights_create_inventory_report_config]

import com.google.cloud.storageinsights.v1.*;
Copy link
Contributor

@frankyn frankyn Jul 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you'll want to be explicit about which imports you're using?

.../src/main/java/com/google/cloud/storage/storageinsights/samples/GetInventoryReportNames.java Outdated
public static void getInventoryReportNames(String projectID, String location, String reportConfigUUID) throws IOException {
try (StorageInsightsClient storageInsightsClient = StorageInsightsClient.create()) {
ReportConfig config = storageInsightsClient.getReportConfig(ReportConfigName.of(projectID, location, reportConfigUUID));
String extension = config.hasCsvOptions() ? "csv" : "parquet";
Copy link
Contributor

@frankyn frankyn Jul 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible have both csv and parquet options?

Copy link
Member Author

@JesseLovelace JesseLovelace Jul 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope!

.../src/main/java/com/google/cloud/storage/storageinsights/samples/GetInventoryReportNames.java Outdated
String extension = config.hasCsvOptions() ? "csv" : "parquet";
System.out.println("You can use the Google Cloud Storage Client to download the following objects from Google Cloud Storage:");
for (ReportDetail reportDetail : storageInsightsClient.listReportDetails(config.getName()).iterateAll()) {
for(long index = reportDetail.getShardsCount() - 1; index >= 0; index--) {
Copy link
Contributor

@frankyn frankyn Jul 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the report spread across shards and not known by the API? I'm wondering if it's necessary to create the report location or if the API itself has this value already.

Copy link
Member Author

@JesseLovelace JesseLovelace Jul 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as far as i can tell it's necessary to construct it like this

.../main/java/com/google/cloud/storage/storageinsights/samples/CreateInventoryReportConfig.java Outdated
.setStartDate(Date.newBuilder().setDay(15).setMonth(8).setYear(3022).build())
.setEndDate(Date.newBuilder().setDay(15).setMonth(9).setYear(3022).build())
.build())
.setCsvOptions(CSVOptions.newBuilder()
Copy link
Contributor

@frankyn frankyn Jul 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a parquet option as well and you're only using Csv for this sample?

Copy link
Member Author

@JesseLovelace JesseLovelace Jul 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correct

@JesseLovelace JesseLovelace added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 25, 2023
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 25, 2023
@JesseLovelace JesseLovelace added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 26, 2023
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 26, 2023
This was referenced Jul 27, 2023
Closed
Closed
Closed
@JesseLovelace JesseLovelace added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 9, 2023
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 9, 2023
@JesseLovelace JesseLovelace added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 9, 2023
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 9, 2023
@JesseLovelace JesseLovelace added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 9, 2023
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 9, 2023
minherz
minherz approved these changes Aug 10, 2023
View reviewed changes
Copy link
Contributor

@minherz minherz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, follow up on means to elminiate 10 second sleep for each test run.

...java/com/google/cloud/storage/storageinsights/samples/test/ITStorageinsightsSamplesTest.java

grantBucketsInsightsPermissions(insightsServiceAccount, SOURCE_BUCKET);
grantBucketsInsightsPermissions(insightsServiceAccount, SINK_BUCKET);
Thread.sleep(10000); //gives time for service account permissions to propagate.
Copy link
Contributor

@minherz minherz Aug 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: i recommend to follow up on this. adding 10 seconds for each check "just in case" is not a good solution. did you consider using externally provisioned SA?

@minherz minherz merged commit ec40893 into main Aug 10, 2023
@minherz minherz deleted the newinsightssamples branch August 10, 2023 19:23
@minherz minherz mentioned this pull request Aug 10, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@minherz minherz minherz approved these changes

@yoshi-approver yoshi-approver Awaiting requested review from yoshi-approver yoshi-approver is a code owner

+1 more reviewer

@frankyn frankyn frankyn approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@minherz minherz

Labels

api: storage Issues related to the Cloud Storage API. samples Issues that are directly related to samples.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@JesseLovelace @frankyn @minherz @kokoro-team