Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation for uploading big files to GC Storage #40

Closed
olgakorichkovskaya opened this issue Oct 28, 2019 · 2 comments
Closed

Documentation for uploading big files to GC Storage #40

olgakorichkovskaya opened this issue Oct 28, 2019 · 2 comments
Assignees
Labels
api: storage Issues related to the googleapis/java-storage API. duplicate This issue or pull request already exists type: docs Improvement to the documentation for an API.

Comments

@olgakorichkovskaya
Copy link

Method 'create' of Storage interface which uses InputStream is marked as @deprecated with notice:

For large content, {@link #writer} is recommended as it uses resumable upload. By default any md5 and crc32c values in the given {@code blobInfo} are ignored unless requested via the {@code BlobWriteOption.md5Match} and {@code BlobWriteOption.crc32cMatch} options. The given input stream is closed upon success.

https://github.com/googleapis/google-cloud-java/blob/master/google-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/Storage.java

Recently in this project was an example of uploading big files into storage, for now, this code was removed:

    private void run(Storage storage, Path uploadFrom, BlobInfo blobInfo) throws IOException {
      if (Files.size(uploadFrom) > 1_000_000) {
        // When content is not available or large (1MB or more) it is recommended
        // to write it in chunks via the blob's channel writer.
        try (WriteChannel writer = storage.writer(blobInfo)) {
          byte[] buffer = new byte[1024];
          try (InputStream input = Files.newInputStream(uploadFrom)) {
            int limit;
            while ((limit = input.read(buffer)) >= 0) {
              try {
                writer.write(ByteBuffer.wrap(buffer, 0, limit));
              } catch (Exception ex) {
                ex.printStackTrace();
              }
            }
          }
        }
      } else {
        byte[] bytes = Files.readAllBytes(uploadFrom);
        // create the blob in one request.
        storage.create(blobInfo, bytes);
      }
      System.out.println("Blob was created");
    }

And there is another example of big file uploading with using crc32c in the ticket https://github.com/googleapis/google-cloud-java/issues/6416

Please add to code samples with a correct and efficient example of uploading big files (big means gigabytes).

@chingor13 chingor13 transferred this issue from googleapis/google-cloud-java Jan 7, 2020
@yoshi-automation yoshi-automation added triage me I really want to be triaged. 🚨 This issue needs some love. labels Jan 7, 2020
@crwilcox crwilcox added documentation Improvements or additions to documentation type: docs Improvement to the documentation for an API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. and removed documentation Improvements or additions to documentation labels Jan 15, 2020
@yoshi-automation yoshi-automation added 🚨 This issue needs some love. and removed 🚨 This issue needs some love. triage me I really want to be triaged. labels Jan 15, 2020
@dmitry-fa
Copy link
Contributor

Instead of writing examples on how to upload files of various size it's better to implement convenient methods as proposed in #62

@frankyn frankyn added duplicate This issue or pull request already exists and removed 🚨 This issue needs some love. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. labels Jan 28, 2020
@frankyn
Copy link
Member

frankyn commented Jan 28, 2020

Deduping against #62, it would be better to have a helper for InputStreams to reduce the replicating the same code in different projects.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/java-storage API. duplicate This issue or pull request already exists type: docs Improvement to the documentation for an API.
Projects
None yet
Development

No branches or pull requests

6 participants