feat: update cloudstorageconfiguration to improve copy accross cross-bucket performance#168
Conversation
Codecov Report
@@ Coverage Diff @@
## master #168 +/- ##
============================================
+ Coverage 71.13% 71.79% +0.66%
- Complexity 481 495 +14
============================================
Files 29 29
Lines 1642 1656 +14
Branches 265 268 +3
============================================
+ Hits 1168 1189 +21
+ Misses 348 342 -6
+ Partials 126 125 -1
Continue to review full report at Codecov.
|
| assertNotEquals(sourceFileSystem.config(), destFileSystem.config()); | ||
| assertEquals("bucket", sourceFileSystem.bucket()); | ||
| assertEquals("new-bucket", destFileSystem.bucket()); | ||
| } catch (IOException e) { |
There was a problem hiding this comment.
IOException should fail test; i.e. don't catch it
dmitry-fa
left a comment
There was a problem hiding this comment.
Your change does not fix the original issue which requests not creating a new CloudStorageFileSystem for a bucket and a provider, if such file system has been already created.
To achieve that you need to remember all created file systems and before creating a new one in the forBucket() method do search among previously created.
The code could look like:
private static final Map<String, CloudStorageFileSystem> fileSystemMap = new HashMap<>();
...
public static CloudStorageFileSystem forBucket(
String bucket, CloudStorageConfiguration config, @Nullable StorageOptions storageOptions) {
...
CloudStorageFileSystem existing = getExisitngCloudStorageFileSystem(bucket, config, storageOptions);
if (existing == null) {
exiting = new CloudStorageFileSystem (bucket, config, storageOptions);
fileSystemMap.put(bucket, existing);
}
return existing;
}
| return new CloudStorageFileSystem( | ||
| cloudStorageFileSystem.provider(), bucketName, cloudStorageFileSystem.config()); | ||
| } | ||
|
|
There was a problem hiding this comment.
Method name getExistingCloudStorageConfiguration implies returning existing configuration, you return FileSystem and create it each time.
| this.config = config; | ||
| this.cloudStorageFileSystem = this; | ||
| } | ||
|
|
There was a problem hiding this comment.
this.cloudStorageFileSystem = this does not work as you probably expect. Each time a new CloudStorageFileSystem object is created the static field cloudStorageFileSystem will be overridden.
f463c31 to
8811b6c
Compare
1f3643b to
d3c8b8b
Compare
|
@dmitry-fa @frankyn ,PTAL |
dmitry-fa
left a comment
There was a problem hiding this comment.
Looks good to me, thanks for addressing all my comments. Please wait for a review from a Googler
|
@frankyn ,gentle ping |
🤖 I have created a release \*beep\* \*boop\* --- ## [0.122.0](https://www.github.com/googleapis/java-storage-nio/compare/v0.121.2...v0.122.0) (2020-11-06) ### Features * update cloudstorageconfiguration to improve copy accross cross-bucket performance ([#168](https://www.github.com/googleapis/java-storage-nio/issues/168)) ([db74524](https://www.github.com/googleapis/java-storage-nio/commit/db74524d68487df71c80b122d8c0ff384dc9ace3)) * **deps:** adopt flatten plugin and google-cloud-shared-dependencies ([#156](https://www.github.com/googleapis/java-storage-nio/issues/156)) ([510f6a5](https://www.github.com/googleapis/java-storage-nio/commit/510f6a5efc4a13b010020a8d67ec1511bbd46564)) ### Bug Fixes * FakeStorageRpc#list to filtering the files by bucket and prefix ([#208](https://www.github.com/googleapis/java-storage-nio/issues/208)) ([21f606e](https://www.github.com/googleapis/java-storage-nio/commit/21f606eca67b1c8a471ee28f7e2dd3851a0c493e)) * implement writeWithResponse in FakeStorageRpc ([#187](https://www.github.com/googleapis/java-storage-nio/issues/187)) ([10ddfae](https://www.github.com/googleapis/java-storage-nio/commit/10ddfae3923f1d5e64b151293db238f1af433d03)) ### Dependencies * update dependency com.google.apis:google-api-services-storage to v1-rev20200611-1.30.9 ([#166](https://www.github.com/googleapis/java-storage-nio/issues/166)) ([3cab5f2](https://www.github.com/googleapis/java-storage-nio/commit/3cab5f25e081c0198d06903a42c7de3dbd34bbad)) * update dependency com.google.apis:google-api-services-storage to v1-rev20200727-1.30.10 ([#171](https://www.github.com/googleapis/java-storage-nio/issues/171)) ([62998f0](https://www.github.com/googleapis/java-storage-nio/commit/62998f0adfb22b194e2dd633532e13bf27a79479)) * update dependency com.google.apis:google-api-services-storage to v1-rev20200814-1.30.10 ([#206](https://www.github.com/googleapis/java-storage-nio/issues/206)) ([0d7cd44](https://www.github.com/googleapis/java-storage-nio/commit/0d7cd44e73b0b9456a8ba96cc6e6d0e60a0b7d4e)) * update dependency com.google.apis:google-api-services-storage to v1-rev20200927-1.30.10 ([#233](https://www.github.com/googleapis/java-storage-nio/issues/233)) ([91b7918](https://www.github.com/googleapis/java-storage-nio/commit/91b7918539186763ecaa377cfb6f3908105df279)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.10.0 ([#226](https://www.github.com/googleapis/java-storage-nio/issues/226)) ([83acbd7](https://www.github.com/googleapis/java-storage-nio/commit/83acbd766b06d66fc9a41385fdd49aa57d9a0291)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.10.1 ([#239](https://www.github.com/googleapis/java-storage-nio/issues/239)) ([bd1928d](https://www.github.com/googleapis/java-storage-nio/commit/bd1928da234dae6fc2fb1ad6658d89a9ebc7af82)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.10.2 ([#243](https://www.github.com/googleapis/java-storage-nio/issues/243)) ([4ca9869](https://www.github.com/googleapis/java-storage-nio/commit/4ca9869d99fc3d2e578fb86fbd9053f855f1e51c)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.12.0 ([#246](https://www.github.com/googleapis/java-storage-nio/issues/246)) ([3e6171b](https://www.github.com/googleapis/java-storage-nio/commit/3e6171b2d4c4c7832b62288faf30f767c5278762)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.12.1 ([#255](https://www.github.com/googleapis/java-storage-nio/issues/255)) ([73d7dd2](https://www.github.com/googleapis/java-storage-nio/commit/73d7dd2ec6ca371480955c844ffb5e6aed4f608f)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.13.0 ([#260](https://www.github.com/googleapis/java-storage-nio/issues/260)) ([54895cf](https://www.github.com/googleapis/java-storage-nio/commit/54895cf441f1e1723cf941aa1f6cab249acae6b8)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.14.1 ([#271](https://www.github.com/googleapis/java-storage-nio/issues/271)) ([058d7c9](https://www.github.com/googleapis/java-storage-nio/commit/058d7c9c1471846f3d897cac31cb5f6d42ce5a7d)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.8.2 ([#167](https://www.github.com/googleapis/java-storage-nio/issues/167)) ([3b14bbc](https://www.github.com/googleapis/java-storage-nio/commit/3b14bbc05d5658c57d4f272fdfdf152d7d9ee18d)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.8.3 ([#169](https://www.github.com/googleapis/java-storage-nio/issues/169)) ([4e7bac1](https://www.github.com/googleapis/java-storage-nio/commit/4e7bac10c2466d1407e8f1a6de838261e6c1b097)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.8.4 ([#189](https://www.github.com/googleapis/java-storage-nio/issues/189)) ([af492b8](https://www.github.com/googleapis/java-storage-nio/commit/af492b8068101727793e64e89da2778205fa08b6)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.8.6 ([#191](https://www.github.com/googleapis/java-storage-nio/issues/191)) ([9bbc1fc](https://www.github.com/googleapis/java-storage-nio/commit/9bbc1fc755271f2ac6b798d50f6c0ba8e022c589)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.9.0 ([#202](https://www.github.com/googleapis/java-storage-nio/issues/202)) ([cf312a3](https://www.github.com/googleapis/java-storage-nio/commit/cf312a3fe46ec446d1b5ee418849b3ab9abee87e)) * update dependency com.google.cloud:google-cloud-storage to v1.110.0 ([#154](https://www.github.com/googleapis/java-storage-nio/issues/154)) ([fd6de38](https://www.github.com/googleapis/java-storage-nio/commit/fd6de38ee63376afff03c7e794dce1f4524a4375)) * update dependency com.google.cloud:google-cloud-storage to v1.112.0 ([#199](https://www.github.com/googleapis/java-storage-nio/issues/199)) ([8a38817](https://www.github.com/googleapis/java-storage-nio/commit/8a3881732551bcef1d76f76026a29071fc50dd43)) * update dependency com.google.cloud:google-cloud-storage to v1.113.0 ([#205](https://www.github.com/googleapis/java-storage-nio/issues/205)) ([068002e](https://www.github.com/googleapis/java-storage-nio/commit/068002e1145418ecb848ed066346d1fc6ec14807)) * update dependency com.google.cloud:google-cloud-storage to v1.113.1 ([#212](https://www.github.com/googleapis/java-storage-nio/issues/212)) ([acc1fe8](https://www.github.com/googleapis/java-storage-nio/commit/acc1fe8f23c5926000c7a869ca9a71372850c46a)) * update dependency com.google.cloud:google-cloud-storage to v1.113.2 ([#266](https://www.github.com/googleapis/java-storage-nio/issues/266)) ([9d2c11d](https://www.github.com/googleapis/java-storage-nio/commit/9d2c11d009394f3f3ffd1b115b97b350aa64d1b7)) * update dependency com.google.guava:guava to v30 ([#263](https://www.github.com/googleapis/java-storage-nio/issues/263)) ([4e81dab](https://www.github.com/googleapis/java-storage-nio/commit/4e81daba421e0e42695203c6fe74670b32a3f576)) * update dependency org.mockito:mockito-core to v3.4.4 ([#170](https://www.github.com/googleapis/java-storage-nio/issues/170)) ([3e06bd5](https://www.github.com/googleapis/java-storage-nio/commit/3e06bd5b07065acb1b41a03ebbb862487c046eeb)) * update dependency org.mockito:mockito-core to v3.5.10 ([#203](https://www.github.com/googleapis/java-storage-nio/issues/203)) ([33fdc31](https://www.github.com/googleapis/java-storage-nio/commit/33fdc315a5b4bccdebb13262cbc5868011abe444)) * update dependency org.mockito:mockito-core to v3.5.11 ([#214](https://www.github.com/googleapis/java-storage-nio/issues/214)) ([575c308](https://www.github.com/googleapis/java-storage-nio/commit/575c30882ab84e1b2b1ce34d5176b4f8688a8136)) * update dependency org.mockito:mockito-core to v3.5.7 ([#194](https://www.github.com/googleapis/java-storage-nio/issues/194)) ([8cc7616](https://www.github.com/googleapis/java-storage-nio/commit/8cc76166baf3f5f49223bb8eafc8704679c4d427)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please).
Fixes #60