Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: update cloudstorageconfiguration to improve copy accross cross-bucket performance #168

Merged
merged 5 commits into from Jul 27, 2020

Conversation

suraj-qlogic
Copy link
Contributor

@suraj-qlogic suraj-qlogic commented Jul 9, 2020

Fixes #60

@googlebot googlebot added the cla: yes label Jul 9, 2020
@suraj-qlogic suraj-qlogic changed the title Storage nio 60 feat: update cloudstorageconfiguration to improve copy accross cross-bucket performance Jul 9, 2020
@codecov
Copy link

@codecov codecov bot commented Jul 9, 2020

Codecov Report

Merging #168 into master will increase coverage by 0.66%.
The diff coverage is 88.23%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #168      +/-   ##
============================================
+ Coverage     71.13%   71.79%   +0.66%     
- Complexity      481      495      +14     
============================================
  Files            29       29              
  Lines          1642     1656      +14     
  Branches        265      268       +3     
============================================
+ Hits           1168     1189      +21     
+ Misses          348      342       -6     
+ Partials        126      125       -1     
Impacted Files Coverage Δ Complexity Δ
...ud/storage/contrib/nio/CloudStorageFileSystem.java 72.60% <88.23%> (+6.50%) 36.00 <9.00> (+9.00)
...ge/contrib/nio/CloudStorageFileSystemProvider.java 62.91% <0.00%> (+0.46%) 76.00% <0.00%> (+3.00%)
...age/contrib/nio/SeekableByteChannelPrefetcher.java 66.25% <0.00%> (+3.12%) 23.00% <0.00%> (+2.00%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 65d448e...8559a74. Read the comment docs.

@suraj-qlogic suraj-qlogic added the kokoro:force-run label Jul 9, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run label Jul 9, 2020
@athakor athakor added the kokoro:force-run label Jul 9, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run label Jul 9, 2020
@suraj-qlogic suraj-qlogic requested review from frankyn and elharo Jul 9, 2020
assertNotEquals(sourceFileSystem.config(), destFileSystem.config());
assertEquals("bucket", sourceFileSystem.bucket());
assertEquals("new-bucket", destFileSystem.bucket());
} catch (IOException e) {
Copy link
Contributor

@elharo elharo Jul 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IOException should fail test; i.e. don't catch it

Copy link
Contributor Author

@suraj-qlogic suraj-qlogic Jul 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@suraj-qlogic suraj-qlogic requested a review from elharo Jul 9, 2020
elharo
elharo approved these changes Jul 9, 2020
Copy link
Contributor

@dmitry-fa dmitry-fa left a comment

Your change does not fix the original issue which requests not creating a new CloudStorageFileSystem for a bucket and a provider, if such file system has been already created.

To achieve that you need to remember all created file systems and before creating a new one in the forBucket() method do search among previously created.

The code could look like:

private static final Map<String, CloudStorageFileSystem> fileSystemMap = new HashMap<>();
...

public static CloudStorageFileSystem forBucket(
      String bucket, CloudStorageConfiguration config, @Nullable StorageOptions storageOptions) {
   ...
   CloudStorageFileSystem  existing = getExisitngCloudStorageFileSystem(bucket, config, storageOptions);
   if (existing == null) {
      exiting = new CloudStorageFileSystem (bucket, config, storageOptions);
      fileSystemMap.put(bucket, existing);
   }
   return existing;
}

return new CloudStorageFileSystem(
cloudStorageFileSystem.provider(), bucketName, cloudStorageFileSystem.config());
}

Copy link
Contributor

@dmitry-fa dmitry-fa Jul 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Method name getExistingCloudStorageConfiguration implies returning existing configuration, you return FileSystem and create it each time.

Copy link
Contributor Author

@suraj-qlogic suraj-qlogic Jul 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@@ -195,6 +211,7 @@ public static CloudStorageFileSystem forBucket(
}
this.provider = provider;
this.config = config;
this.cloudStorageFileSystem = this;
}

Copy link
Contributor

@dmitry-fa dmitry-fa Jul 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this.cloudStorageFileSystem = this does not work as you probably expect. Each time a new CloudStorageFileSystem object is created the static field cloudStorageFileSystem will be overridden.

Copy link
Contributor Author

@suraj-qlogic suraj-qlogic Jul 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@suraj-qlogic
Copy link
Contributor Author

@suraj-qlogic suraj-qlogic commented Jul 21, 2020

@dmitry-fa dmitry-fa self-requested a review Jul 21, 2020
Copy link
Contributor

@dmitry-fa dmitry-fa left a comment

Looks good to me, thanks for addressing all my comments. Please wait for a review from a Googler

elharo
elharo approved these changes Jul 22, 2020
@suraj-qlogic
Copy link
Contributor Author

@suraj-qlogic suraj-qlogic commented Jul 27, 2020

@frankyn ,gentle ping

@frankyn frankyn merged commit db74524 into googleapis:master Jul 27, 2020
15 checks passed
@suraj-qlogic suraj-qlogic deleted the storage-nio-60 branch Jul 30, 2020
gcf-merge-on-green bot pushed a commit that referenced this issue Nov 6, 2020
🤖 I have created a release \*beep\* \*boop\* 
---
## [0.122.0](https://www.github.com/googleapis/java-storage-nio/compare/v0.121.2...v0.122.0) (2020-11-06)


### Features

* update cloudstorageconfiguration to improve copy accross cross-bucket performance ([#168](https://www.github.com/googleapis/java-storage-nio/issues/168)) ([db74524](https://www.github.com/googleapis/java-storage-nio/commit/db74524d68487df71c80b122d8c0ff384dc9ace3))
* **deps:** adopt flatten plugin and google-cloud-shared-dependencies ([#156](https://www.github.com/googleapis/java-storage-nio/issues/156)) ([510f6a5](https://www.github.com/googleapis/java-storage-nio/commit/510f6a5efc4a13b010020a8d67ec1511bbd46564))


### Bug Fixes

* FakeStorageRpc#list to filtering the files by bucket and prefix ([#208](https://www.github.com/googleapis/java-storage-nio/issues/208)) ([21f606e](https://www.github.com/googleapis/java-storage-nio/commit/21f606eca67b1c8a471ee28f7e2dd3851a0c493e))
* implement writeWithResponse in FakeStorageRpc ([#187](https://www.github.com/googleapis/java-storage-nio/issues/187)) ([10ddfae](https://www.github.com/googleapis/java-storage-nio/commit/10ddfae3923f1d5e64b151293db238f1af433d03))


### Dependencies

* update dependency com.google.apis:google-api-services-storage to v1-rev20200611-1.30.9 ([#166](https://www.github.com/googleapis/java-storage-nio/issues/166)) ([3cab5f2](https://www.github.com/googleapis/java-storage-nio/commit/3cab5f25e081c0198d06903a42c7de3dbd34bbad))
* update dependency com.google.apis:google-api-services-storage to v1-rev20200727-1.30.10 ([#171](https://www.github.com/googleapis/java-storage-nio/issues/171)) ([62998f0](https://www.github.com/googleapis/java-storage-nio/commit/62998f0adfb22b194e2dd633532e13bf27a79479))
* update dependency com.google.apis:google-api-services-storage to v1-rev20200814-1.30.10 ([#206](https://www.github.com/googleapis/java-storage-nio/issues/206)) ([0d7cd44](https://www.github.com/googleapis/java-storage-nio/commit/0d7cd44e73b0b9456a8ba96cc6e6d0e60a0b7d4e))
* update dependency com.google.apis:google-api-services-storage to v1-rev20200927-1.30.10 ([#233](https://www.github.com/googleapis/java-storage-nio/issues/233)) ([91b7918](https://www.github.com/googleapis/java-storage-nio/commit/91b7918539186763ecaa377cfb6f3908105df279))
* update dependency com.google.cloud:google-cloud-shared-dependencies to v0.10.0 ([#226](https://www.github.com/googleapis/java-storage-nio/issues/226)) ([83acbd7](https://www.github.com/googleapis/java-storage-nio/commit/83acbd766b06d66fc9a41385fdd49aa57d9a0291))
* update dependency com.google.cloud:google-cloud-shared-dependencies to v0.10.1 ([#239](https://www.github.com/googleapis/java-storage-nio/issues/239)) ([bd1928d](https://www.github.com/googleapis/java-storage-nio/commit/bd1928da234dae6fc2fb1ad6658d89a9ebc7af82))
* update dependency com.google.cloud:google-cloud-shared-dependencies to v0.10.2 ([#243](https://www.github.com/googleapis/java-storage-nio/issues/243)) ([4ca9869](https://www.github.com/googleapis/java-storage-nio/commit/4ca9869d99fc3d2e578fb86fbd9053f855f1e51c))
* update dependency com.google.cloud:google-cloud-shared-dependencies to v0.12.0 ([#246](https://www.github.com/googleapis/java-storage-nio/issues/246)) ([3e6171b](https://www.github.com/googleapis/java-storage-nio/commit/3e6171b2d4c4c7832b62288faf30f767c5278762))
* update dependency com.google.cloud:google-cloud-shared-dependencies to v0.12.1 ([#255](https://www.github.com/googleapis/java-storage-nio/issues/255)) ([73d7dd2](https://www.github.com/googleapis/java-storage-nio/commit/73d7dd2ec6ca371480955c844ffb5e6aed4f608f))
* update dependency com.google.cloud:google-cloud-shared-dependencies to v0.13.0 ([#260](https://www.github.com/googleapis/java-storage-nio/issues/260)) ([54895cf](https://www.github.com/googleapis/java-storage-nio/commit/54895cf441f1e1723cf941aa1f6cab249acae6b8))
* update dependency com.google.cloud:google-cloud-shared-dependencies to v0.14.1 ([#271](https://www.github.com/googleapis/java-storage-nio/issues/271)) ([058d7c9](https://www.github.com/googleapis/java-storage-nio/commit/058d7c9c1471846f3d897cac31cb5f6d42ce5a7d))
* update dependency com.google.cloud:google-cloud-shared-dependencies to v0.8.2 ([#167](https://www.github.com/googleapis/java-storage-nio/issues/167)) ([3b14bbc](https://www.github.com/googleapis/java-storage-nio/commit/3b14bbc05d5658c57d4f272fdfdf152d7d9ee18d))
* update dependency com.google.cloud:google-cloud-shared-dependencies to v0.8.3 ([#169](https://www.github.com/googleapis/java-storage-nio/issues/169)) ([4e7bac1](https://www.github.com/googleapis/java-storage-nio/commit/4e7bac10c2466d1407e8f1a6de838261e6c1b097))
* update dependency com.google.cloud:google-cloud-shared-dependencies to v0.8.4 ([#189](https://www.github.com/googleapis/java-storage-nio/issues/189)) ([af492b8](https://www.github.com/googleapis/java-storage-nio/commit/af492b8068101727793e64e89da2778205fa08b6))
* update dependency com.google.cloud:google-cloud-shared-dependencies to v0.8.6 ([#191](https://www.github.com/googleapis/java-storage-nio/issues/191)) ([9bbc1fc](https://www.github.com/googleapis/java-storage-nio/commit/9bbc1fc755271f2ac6b798d50f6c0ba8e022c589))
* update dependency com.google.cloud:google-cloud-shared-dependencies to v0.9.0 ([#202](https://www.github.com/googleapis/java-storage-nio/issues/202)) ([cf312a3](https://www.github.com/googleapis/java-storage-nio/commit/cf312a3fe46ec446d1b5ee418849b3ab9abee87e))
* update dependency com.google.cloud:google-cloud-storage to v1.110.0 ([#154](https://www.github.com/googleapis/java-storage-nio/issues/154)) ([fd6de38](https://www.github.com/googleapis/java-storage-nio/commit/fd6de38ee63376afff03c7e794dce1f4524a4375))
* update dependency com.google.cloud:google-cloud-storage to v1.112.0 ([#199](https://www.github.com/googleapis/java-storage-nio/issues/199)) ([8a38817](https://www.github.com/googleapis/java-storage-nio/commit/8a3881732551bcef1d76f76026a29071fc50dd43))
* update dependency com.google.cloud:google-cloud-storage to v1.113.0 ([#205](https://www.github.com/googleapis/java-storage-nio/issues/205)) ([068002e](https://www.github.com/googleapis/java-storage-nio/commit/068002e1145418ecb848ed066346d1fc6ec14807))
* update dependency com.google.cloud:google-cloud-storage to v1.113.1 ([#212](https://www.github.com/googleapis/java-storage-nio/issues/212)) ([acc1fe8](https://www.github.com/googleapis/java-storage-nio/commit/acc1fe8f23c5926000c7a869ca9a71372850c46a))
* update dependency com.google.cloud:google-cloud-storage to v1.113.2 ([#266](https://www.github.com/googleapis/java-storage-nio/issues/266)) ([9d2c11d](https://www.github.com/googleapis/java-storage-nio/commit/9d2c11d009394f3f3ffd1b115b97b350aa64d1b7))
* update dependency com.google.guava:guava to v30 ([#263](https://www.github.com/googleapis/java-storage-nio/issues/263)) ([4e81dab](https://www.github.com/googleapis/java-storage-nio/commit/4e81daba421e0e42695203c6fe74670b32a3f576))
* update dependency org.mockito:mockito-core to v3.4.4 ([#170](https://www.github.com/googleapis/java-storage-nio/issues/170)) ([3e06bd5](https://www.github.com/googleapis/java-storage-nio/commit/3e06bd5b07065acb1b41a03ebbb862487c046eeb))
* update dependency org.mockito:mockito-core to v3.5.10 ([#203](https://www.github.com/googleapis/java-storage-nio/issues/203)) ([33fdc31](https://www.github.com/googleapis/java-storage-nio/commit/33fdc315a5b4bccdebb13262cbc5868011abe444))
* update dependency org.mockito:mockito-core to v3.5.11 ([#214](https://www.github.com/googleapis/java-storage-nio/issues/214)) ([575c308](https://www.github.com/googleapis/java-storage-nio/commit/575c30882ab84e1b2b1ce34d5176b4f8688a8136))
* update dependency org.mockito:mockito-core to v3.5.7 ([#194](https://www.github.com/googleapis/java-storage-nio/issues/194)) ([8cc7616](https://www.github.com/googleapis/java-storage-nio/commit/8cc76166baf3f5f49223bb8eafc8704679c4d427))
---


This PR was generated with [Release Please](https://github.com/googleapis/release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants