Ft/s3 c 1179/gcp s3 mpu #1123

alexanderchan-scality · 2018-02-13T16:41:12Z

Adds GCP MPU Functionality

ironman-machine · 2018-02-14T18:27:21Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-02-15T03:26:14Z

PR has been updated. Reviewers, please be cautious.

dora-korpar · 2018-02-16T21:26:43Z

lib/api/objectPutPart.js

                        return next(null, destinationBucket,
                            objectLocationConstraint, cipherBundle, splitter,
                            partInfo);
+                    } else if (partInfo && partInfo.dataStoreType === 'gcp') {


trivial, but these two else ifs could be combined with something like ['azure', 'gcp'].includes(partInfo.dataStoreType)

dora-korpar · 2018-02-16T21:27:48Z

lib/api/objectPutPart.js

-            if (partInfo && partInfo.dataStoreType === 'azure') {
+            if (partInfo &&
+               (partInfo.dataStoreType === 'azure' ||
+                partInfo.dataStoreType === 'gcp')) {


could use a similar structure as my previous comment, with the Array.includes() method

ironman-machine · 2018-02-20T01:42:18Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-02-20T05:26:37Z

PR has been updated. Reviewers, please be cautious.

alexanderchan-scality · 2018-02-20T05:29:31Z

@ironman-machine
DEFAULT_BRANCH=rel/7.4
SCALITY_INTEGRATION_BRANCH=test/zenkobb
SCALITY_FEDERATION_BRANCH=ft/S3C-1114/GCP-BACKEND
SCALITY_S3_BRANCH=ft/S3C-1179/GCP-S3-MPU
try

ironman-machine · 2018-02-20T05:29:34Z

Hello @alexanderchan-scality

"DEFAULT_BRANCH=rel/7.4": Success
"SCALITY_INTEGRATION_BRANCH=test/zenkobb": Success
"SCALITY_FEDERATION_BRANCH=ft/S3C-1114/GCP-BACKEND": Success
"SCALITY_S3_BRANCH=ft/S3C-1179/GCP-S3-MPU": Success
"try": Success: Try build successfully launched on 'http://ci.ironmann.io/gh/scality/Integration/19562' with the following env. args:

{
    "DEFAULT_BRANCH": "rel/7.4",
    "SCALITY_S3_BRANCH": "ft/S3C-1179/GCP-S3-MPU",
    "SCALITY_INTEGRATION_BRANCH": "test/zenkobb",
    "REPO_NAME": "S3",
    "SCALITY_FEDERATION_BRANCH": "ft/S3C-1114/GCP-BACKEND"
}

ironman-machine · 2018-02-20T07:18:38Z

☀️ 👍 circleCI test succeeded!

ironman-machine · 2018-02-21T15:55:05Z

PR has been updated. Reviewers, please be cautious.

alexanderchan-scality · 2018-02-21T15:55:52Z

@ironman-machine
DEFAULT_BRANCH=rel/7.4
SCALITY_INTEGRATION_BRANCH=test/zenkobb
SCALITY_FEDERATION_BRANCH=ft/S3C-1114/GCP-BACKEND
SCALITY_S3_BRANCH=ft/S3C-1179/GCP-S3-MPU
try

ironman-machine · 2018-02-21T15:55:56Z

Hello @alexanderchan-scality

"DEFAULT_BRANCH=rel/7.4": Success
"SCALITY_INTEGRATION_BRANCH=test/zenkobb": Success
"SCALITY_FEDERATION_BRANCH=ft/S3C-1114/GCP-BACKEND": Success
"SCALITY_S3_BRANCH=ft/S3C-1179/GCP-S3-MPU": Success
"try": Success: Try build successfully launched on 'http://ci.ironmann.io/gh/scality/Integration/19621' with the following env. args:

{
    "DEFAULT_BRANCH": "rel/7.4",
    "SCALITY_S3_BRANCH": "ft/S3C-1179/GCP-S3-MPU",
    "SCALITY_INTEGRATION_BRANCH": "test/zenkobb",
    "REPO_NAME": "S3",
    "SCALITY_FEDERATION_BRANCH": "ft/S3C-1114/GCP-BACKEND"
}

ironman-machine · 2018-02-21T17:17:21Z

☀️ 👍 circleCI test succeeded!

rahulreddy · 2018-02-21T18:22:37Z

constants.js

-    versioningNotImplBackends: { azure: true },
-    mpuMDStoredExternallyBackend: { aws_s3: true },
+    versioningNotImplBackends: { azure: true, gcp: true },
+    mpuMDStoredExternallyBackend: { aws_s3: true, gcp: true },


Are we actually storing MD on GCP like we do for AWS S3? Because for AWS we do initiate mpu but for GCP we emulate MPU.

Yes. On initiate MPU, a 0-byte object will be created to hold the metadata for the final object.

ironman-machine · 2018-02-21T19:51:30Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-02-22T01:17:20Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-02-22T01:17:20Z

CONFLICT (add/add): Merge conflict in tests/functional/raw-node/utils/gcpUtils.js
CONFLICT (add/add): Merge conflict in tests/functional/aws-node-sdk/test/multipleBackend/utils.js
CONFLICT (add/add): Merge conflict in lib/data/wrapper.js
CONFLICT (add/add): Merge conflict in lib/data/multipleBackendGateway.js
CONFLICT (add/add): Merge conflict in lib/data/external/GcpClient.js
CONFLICT (add/add): Merge conflict in lib/data/external/GCP/GcpUtils.js
CONFLICT (add/add): Merge conflict in lib/data/external/GCP/GcpApis/uploadPartCopy.js
CONFLICT (add/add): Merge conflict in lib/data/external/GCP/GcpApis/uploadPart.js
CONFLICT (add/add): Merge conflict in lib/data/external/GCP/GcpApis/listParts.js
CONFLICT (add/add): Merge conflict in lib/data/external/GCP/GcpApis/copyObject.js
CONFLICT (add/add): Merge conflict in lib/api/objectPutPart.js
CONFLICT (add/add): Merge conflict in constants.js

ironman-machine · 2018-02-22T03:28:41Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-02-22T03:28:41Z

CONFLICT (add/add): Merge conflict in tests/functional/raw-node/utils/gcpUtils.js
CONFLICT (add/add): Merge conflict in tests/functional/aws-node-sdk/test/multipleBackend/utils.js
CONFLICT (add/add): Merge conflict in lib/data/wrapper.js
CONFLICT (add/add): Merge conflict in lib/data/multipleBackendGateway.js
CONFLICT (add/add): Merge conflict in lib/data/external/GcpClient.js
CONFLICT (add/add): Merge conflict in lib/data/external/GCP/GcpUtils.js
CONFLICT (add/add): Merge conflict in lib/data/external/GCP/GcpApis/uploadPartCopy.js
CONFLICT (add/add): Merge conflict in lib/data/external/GCP/GcpApis/uploadPart.js
CONFLICT (add/add): Merge conflict in lib/data/external/GCP/GcpApis/listParts.js
CONFLICT (add/add): Merge conflict in lib/data/external/GCP/GcpApis/copyObject.js
CONFLICT (add/add): Merge conflict in lib/api/objectPutPart.js
CONFLICT (add/add): Merge conflict in constants.js

ironman-machine · 2018-02-22T04:07:18Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-02-22T11:27:38Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-03-09T18:09:45Z

PR has been updated. Reviewers, please be cautious.

alexanderchan-scality · 2018-03-09T18:10:25Z

@ironman-machine
DEFAULT_BRANCH=rel/7.4
SCALITY_INTEGRATION_BRANCH=ft/GCP
SCALITY_FEDERATION_BRANCH=ft/GCP
try

ironman-machine · 2018-03-09T18:10:27Z

Hello @alexanderchan-scality

"DEFAULT_BRANCH=rel/7.4": Success
"SCALITY_INTEGRATION_BRANCH=ft/GCP": Success
"SCALITY_FEDERATION_BRANCH=ft/GCP": Success
"try": Success: Try build successfully launched on 'http://ci.ironmann.io/gh/scality/Integration/20159' with the following env. args:

{
    "REPO_NAME": "S3",
    "DEFAULT_BRANCH": "rel/7.4",
    "SCALITY_INTEGRATION_BRANCH": "ft/GCP",
    "SCALITY_FEDERATION_BRANCH": "ft/GCP",
    "SCALITY_S3_BRANCH": "ft/S3C-1179/GCP-S3-MPU"
}

ironman-machine · 2018-03-09T19:29:04Z

💔 ☔ circleCI test failed.

alexanderchan-scality · 2018-03-09T22:58:28Z

Passing integration build: http://ci.ironmann.io/gh/scality/Integration/20163

Adds MPU functionalities to GCP Backend

Do it again human slave!:point_right: :runner: (Oh and the pull request has been updated, by the way.)

ironman-machine · 2018-03-09T23:06:33Z

PR has been updated. Reviewers, please be cautious.

alexanderchan-scality · 2018-03-09T23:20:17Z

@ironman-machine
DEFAULT_BRANCH=rel/7.4
SCALITY_INTEGRATION_BRANCH=ft/GCP
SCALITY_FEDERATION_BRANCH=ft/GCP
try

ironman-machine · 2018-03-09T23:20:19Z

Hello @alexanderchan-scality

"DEFAULT_BRANCH=rel/7.4": Success
"SCALITY_INTEGRATION_BRANCH=ft/GCP": Success
"SCALITY_FEDERATION_BRANCH=ft/GCP": Success
"try": Success: Try build successfully launched on 'http://ci.ironmann.io/gh/scality/Integration/20175' with the following env. args:

{
    "REPO_NAME": "S3",
    "DEFAULT_BRANCH": "rel/7.4",
    "SCALITY_INTEGRATION_BRANCH": "ft/GCP",
    "SCALITY_FEDERATION_BRANCH": "ft/GCP",
    "SCALITY_S3_BRANCH": "ft/S3C-1179/GCP-S3-MPU"
}

ironman-machine · 2018-03-10T00:39:13Z

☀️ 👍 circleCI test succeeded!

ironman-machine · 2018-03-10T01:58:28Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-03-10T01:58:28Z

CONFLICT (add/add): Merge conflict in tests/functional/aws-node-sdk/test/multipleBackend/utils.js
CONFLICT (add/add): Merge conflict in lib/data/wrapper.js
CONFLICT (add/add): Merge conflict in lib/data/multipleBackendGateway.js
CONFLICT (add/add): Merge conflict in lib/data/external/GcpClient.js
CONFLICT (add/add): Merge conflict in lib/data/external/GCP/GcpUtils.js
CONFLICT (add/add): Merge conflict in lib/data/external/GCP/GcpApis/uploadPartCopy.js
CONFLICT (add/add): Merge conflict in lib/data/external/GCP/GcpApis/uploadPart.js
CONFLICT (add/add): Merge conflict in lib/data/external/GCP/GcpApis/listParts.js
CONFLICT (add/add): Merge conflict in lib/api/objectPutPart.js
CONFLICT (add/add): Merge conflict in lib/Config.js
CONFLICT (add/add): Merge conflict in constants.js

ironman-machine · 2018-03-10T02:05:23Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-03-10T02:08:58Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-03-10T02:15:22Z

PR has been updated. Reviewers, please be cautious.

ironman-machine · 2018-03-10T02:19:03Z

PR has been updated. Reviewers, please be cautious.

alexanderchan-scality · 2018-03-10T05:16:01Z

@ironman-machine
DEFAULT_BRANCH=rel/7.4
SCALITY_INTEGRATION_BRANCH=ft/GCP
SCALITY_FEDERATION_BRANCH=ft/GCP
try

ironman-machine · 2018-03-10T05:16:04Z

Hello @alexanderchan-scality

"DEFAULT_BRANCH=rel/7.4": Success
"SCALITY_INTEGRATION_BRANCH=ft/GCP": Success
"SCALITY_FEDERATION_BRANCH=ft/GCP": Success
"try": Success: Try build successfully launched on 'http://ci.ironmann.io/gh/scality/Integration/20194' with the following env. args:

{
    "REPO_NAME": "S3",
    "DEFAULT_BRANCH": "rel/7.4",
    "SCALITY_INTEGRATION_BRANCH": "ft/GCP",
    "SCALITY_FEDERATION_BRANCH": "ft/GCP",
    "SCALITY_S3_BRANCH": "ft/S3C-1179/GCP-S3-MPU"
}

ironman-machine · 2018-03-10T06:38:52Z

☀️ 👍 circleCI test succeeded!

alexanderchan-scality force-pushed the ft/S3C-1179/GCP-S3-MPU branch from 84032fa to 47e9e14 Compare February 14, 2018 18:27

alexanderchan-scality force-pushed the ft/S3C-1179/GCP-S3-MPU branch from 47e9e14 to b6690e4 Compare February 15, 2018 03:26

dora-korpar reviewed Feb 16, 2018

View reviewed changes

alexanderchan-scality force-pushed the ft/S3C-1179/GCP-S3-MPU branch from b6690e4 to 9a09f2d Compare February 20, 2018 01:42

alexanderchan-scality force-pushed the ft/S3C-1179/GCP-S3-MPU branch from 9a09f2d to 634e33e Compare February 20, 2018 05:26

alexanderchan-scality force-pushed the ft/S3C-1179/GCP-S3-MPU branch from 634e33e to b38eaf2 Compare February 21, 2018 15:55

rahulreddy reviewed Feb 21, 2018

View reviewed changes

alexanderchan-scality force-pushed the ft/S3C-1179/GCP-S3-MPU branch from b38eaf2 to 6a5ffa7 Compare February 21, 2018 19:51

alexanderchan-scality force-pushed the ft/S3C-1179/GCP-S3-MPU branch from 6a5ffa7 to 6d43503 Compare February 22, 2018 01:17

alexanderchan-scality force-pushed the ft/S3C-1179/GCP-S3-MPU branch from 1586f97 to de6d448 Compare February 22, 2018 04:07

rahulreddy force-pushed the ft/S3C-1179/GCP-S3-MPU branch from de6d448 to 76d5cdf Compare February 22, 2018 11:27

alexanderchan-scality force-pushed the ft/S3C-1179/GCP-S3-MPU branch from 8bc6385 to f95e571 Compare March 9, 2018 18:09

JianqinWang previously approved these changes Mar 9, 2018

View reviewed changes

S3C-1179 FT: Adds GCP MPU Func

8b62925

Adds MPU functionalities to GCP Backend

alexanderchan-scality force-pushed the ft/S3C-1179/GCP-S3-MPU branch from f95e571 to 8b62925 Compare March 9, 2018 23:06

alexanderchan-scality force-pushed the ft/S3C-1179/GCP-S3-MPU branch from f2e9d5e to 9273dcf Compare March 10, 2018 02:05

alexanderchan-scality force-pushed the ft/S3C-1179/GCP-S3-MPU branch from 9273dcf to a257e76 Compare March 10, 2018 02:08

alexanderchan-scality force-pushed the ft/S3C-1179/GCP-S3-MPU branch from a257e76 to a32690b Compare March 10, 2018 02:15

Switch from npm to yarn

9c3a8f1

alexanderchan-scality force-pushed the ft/S3C-1179/GCP-S3-MPU branch from a32690b to 9c3a8f1 Compare March 10, 2018 02:18

alexanderchan-scality merged commit e5f966d into ft/GCP Mar 11, 2018

alexanderchan-scality deleted the ft/S3C-1179/GCP-S3-MPU branch April 18, 2018 16:50

Ft/s3 c 1179/gcp s3 mpu #1123

Ft/s3 c 1179/gcp s3 mpu #1123

Uh oh!

Conversation

alexanderchan-scality commented Feb 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ironman-machine commented Feb 14, 2018

Uh oh!

ironman-machine commented Feb 15, 2018

Uh oh!

dora-korpar Feb 16, 2018

Choose a reason for hiding this comment

Uh oh!

dora-korpar Feb 16, 2018

Choose a reason for hiding this comment

Uh oh!

ironman-machine commented Feb 20, 2018

Uh oh!

ironman-machine commented Feb 20, 2018

Uh oh!

alexanderchan-scality commented Feb 20, 2018

Uh oh!

ironman-machine commented Feb 20, 2018

Uh oh!

ironman-machine commented Feb 20, 2018

Uh oh!

ironman-machine commented Feb 21, 2018

Uh oh!

alexanderchan-scality commented Feb 21, 2018

Uh oh!

ironman-machine commented Feb 21, 2018

Uh oh!

ironman-machine commented Feb 21, 2018

Uh oh!

rahulreddy Feb 21, 2018

Choose a reason for hiding this comment

Uh oh!

alexanderchan-scality Feb 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ironman-machine commented Feb 21, 2018

Uh oh!

ironman-machine commented Feb 22, 2018

Uh oh!

ironman-machine commented Feb 22, 2018

Uh oh!

ironman-machine commented Feb 22, 2018

Uh oh!

ironman-machine commented Feb 22, 2018

Uh oh!

ironman-machine commented Feb 22, 2018

Uh oh!

ironman-machine commented Feb 22, 2018

Uh oh!

ironman-machine commented Mar 9, 2018

Uh oh!

alexanderchan-scality commented Mar 9, 2018

Uh oh!

ironman-machine commented Mar 9, 2018

Uh oh!

ironman-machine commented Mar 9, 2018

Uh oh!

alexanderchan-scality commented Mar 9, 2018

Uh oh!

ironman-machine commented Mar 9, 2018

Uh oh!

alexanderchan-scality commented Mar 9, 2018

Uh oh!

ironman-machine commented Mar 9, 2018

Uh oh!

ironman-machine commented Mar 10, 2018

Uh oh!

ironman-machine commented Mar 10, 2018

Uh oh!

ironman-machine commented Mar 10, 2018

Uh oh!

ironman-machine commented Mar 10, 2018

Uh oh!

alexanderchan-scality commented Feb 13, 2018 •

edited

Loading

alexanderchan-scality Feb 21, 2018 •

edited

Loading