BYO creds related issue #66

sandeep-paliwal · 2020-10-01T06:28:40Z

Relates to #52, #62 and #55

Description

Fixed issue with using BYO azure creds
Fixed issue with presign URL not working with BYO creds
Code refactoring to manage above changes
Fix test failure because of code refactoring

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

[ x] Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ x] I have signed the Adobe Open Source CLA.
[ x] My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
[x ] I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

Sync with Master

Fixed issue with BYO creds Fixed issue with presign URL feature not working with BYO creds

moritzraho

first of all, I see two issues in the current implementation:

there is one case which is not covered for presignURLs which is when the user brings his own storage via SAS credentials: i.e. when await files.init({ azure: { sasURLPrivate: 'xxx', sasURLPublic: 'xxx' } }), this behavior is defined in the API https://github.com/adobe/aio-lib-files/blob/master/doc/api.md#initconfig--promisefiles.
- files.generatePresignURL should throw an error with a meaningful error message explaining that the user should instead provide account credentials to use the presign feature
- An alternative is to remove support for init with SAS credentials, BUT it will be a breaking change and a lost in functionality
in AuthManager.init the TVM is initialized even if the user brought his own storage credentials, which we shouldn't do, as the TVM is not needed in that case.

now, in my opinion the added abstraction is premature and adds complexity to the code base. I prefer the following approach:

keep all the credential generation (tvm) logic in init, in pseudocode this is what I mean:

init (config) {
  if (config.azure) {
    // user brought his own credentials
    AzureBlobFiles.init(config.azure, null)
  } else {
    // do init tvm
    // do retrieve credentials via TVM
    AzureBlobFiles.init(credentials, tvm) // pass tvm instance to be able to call tvm.getAzureBlobPresignCredentials
  }
}

in AzureBlobFiles keep things mostly as before, just change the _getPresignUrl function as follows:

_getPresignUrl () {
  // case 1: init was done via OW credentials
  if (this.tvm) {
    // do generate presign via tvm
    return presignURL
  }

  // case 2: bring your own account credentials
  if (this.hasAccountCredentials) { // set this variable in init to true in init if credentials.storageAccount is set
    // copy logic of AzureTVMWrapper.getAzureBlobPresignCredentials in here
  }

  // case 3: bring you own SAS credentials, not supported for presign url
  logAndThrow(new codes.ERROR_XXX('<nice error message in here>'))
}

(Note: as an alternative AzureBlobFiles.init() could take a simple lambda function (tvm.getAzureBlobPresignCredentials.bind(tvm)) as argument instead of the whole tvm instance)

Why I prefer the direct approach:

With the proposed solution in this PR, contributors bringing a new Storage implementation will need to know about and implement 2 abstractions instead of one.
the proposed abstraction is very Azure specific, it requires every child class to implement getAzureBlobCredentials and getAzureBlobPresignCredentials which wouldn't make sense for another storage provider.
Also the generation of presign urls in other providers, e.g. AWS could maybe work without doing an extra request to the TVM. In those cases the abstraction would not be very useful.

Ultimately, I don't think it's a bad idea to wrap the credential handling in an abstraction, but I would wait for another cloud storage to be supported first, to give us a better idea of what needs to be included and what not.
Such an abstraction could eventually be even reused by state lib.

But maybe we need a third eye on this 🙂, @purplecabbage wdyt?

codecov · 2020-10-05T07:29:00Z

Codecov Report

Merging #66 into master will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##            master       #66   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            5         5           
  Lines          423       445   +22     
  Branches        81        85    +4     
=========================================
+ Hits           423       445   +22

Impacted Files	Coverage Δ
lib/FilesError.js	`100.00% <100.00%> (ø)`
lib/impl/AzureBlobFiles.js	`100.00% <100.00%> (ø)`
lib/init.js	`100.00% <100.00%> (ø)`
lib/utils.js	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d921fa2...12ebe86. Read the comment docs.

Added tests

Updated doc fixed tests for above change

lib/impl/AzureBlobFiles.js

moritzraho · 2020-10-05T09:30:25Z

lib/impl/AzureBlobFiles.js

@@ -383,6 +382,8 @@ class AzureBlobFiles extends Files {
   * @private
   */
  async _getAzureBlobPresignCredentials (params) {
+    if (this._azure.sasCreds) { logAndThrow(new codes.ERROR_UNSUPPORTED_OPERATION({ messageValues: ['generatePresignURL'], sdkDetails: params })) }


I think the condition should be this.hasOwnCredentials && this._azure.sasCreds because tvm credentials ARE SAS credentials

we already check for this.hasOwnCredentials within _getPresignUrl call, so we don't need to repeat that check.

moritzraho · 2020-10-05T09:31:59Z

lib/FilesError.js

@@ -45,6 +45,7 @@ E('ERROR_FILE_NOT_EXISTS', 'file `%s` does not exist')
 E('ERROR_BAD_FILE_TYPE', '%s')
 E('ERROR_OUT_OF_RANGE', '%s')
 E('ERROR_MISSING_OPTION', '%s')
+E('ERROR_UNSUPPORTED_OPERATION', 'Operation %s not supported for SAS credentails')


let's make this more generic: use simply %s

moritzraho · 2020-10-05T09:34:18Z

lib/impl/AzureBlobFiles.js

+   * @private
+   */
+  async _getAzureBlobPresignCredentials (params) {
+    if (this._azure.sasCreds) { logAndThrow(new codes.ERROR_UNSUPPORTED_OPERATION({ messageValues: ['generatePresignURL'], sdkDetails: params })) }


should we make the messageValues a bit more precise : ['generatePresignURL are not supported with Azure Container SAS credentials, please initialize the SDK with Azure storage account credentials instead']
?

moritzraho · 2020-10-05T09:41:48Z

test/impl/AzureBlobFiles.test.js

@@ -571,6 +606,41 @@ describe('_getPresignUrl', () => {
    expect(url).toEqual(expectedUrl)
    expect(tvm.getAzureBlobPresignCredentials).toHaveBeenCalledWith({ blobName: 'fakesub/afile', expiryInSeconds: 60, permissions: 'fake' })
  })
+
+  test('_getPresignUrl with correct options default permission own credentials', async () => {


the 4 tests should not simulate an init by setting private control variables. Instead those variable should be set by the constructor depending on the init input.
for example instead of setting files.hasOwnCredentials=true and files.credentials=fakeUserCredentials and files._azure.sasCreds=false you could do files = await AzureBlobFiles.init(fakeUserCredentials, null) => this will do the same but doesn't test internal control variable names which could change in the future

moritzraho · 2020-10-05T09:45:20Z

test/impl/AzureBlobFiles.test.js

+    files._azure.sasCreds = false
+    const ret = await files._getAzureBlobPresignCredentials('fakesub/afile', { expiryInSeconds: 60 })
+    expect(ret).toEqual({ signature: 'fakeSAS' })
+  })


missing test with tvm credentials ? you could pass a tvm mock to the init =>

mockFn = jest.fn() mockfiles = await AzureBlobFiles.init(fakeSASCredentials, { getAzureBlobPresignCredentials: mockFn })

then you can test

expect(mockFn.getAzureBlobPresignCredentials).toHaveBeenCalled()

This one is specific unit test for the private function _getAzureBlobPresignCredentials , there are other tests which mock tvm

Ok I see, I think we should test against _getPresignUrl' directly with different tvm, byo account and byo sas credential. I don't think we should test against _getAzureBlobPresignCredentials as it is not exposed by the Files interface

moritzraho

lgtm, thanks for doing the changes @sandeep-paliwal

sandeep-paliwal added 2 commits September 24, 2020 16:57

Merge pull request #4 from adobe/master

45d86b3

Sync with Master

Code refactoring to abstract TVM behaviour from Files impl

7fed333

Fixed issue with BYO creds Fixed issue with presign URL feature not working with BYO creds

sandeep-paliwal marked this pull request as draft October 1, 2020 06:29

meryllblanchet requested a review from moritzraho October 1, 2020 07:37

moritzraho reviewed Oct 2, 2020

View reviewed changes

sandeep-paliwal added 2 commits October 5, 2020 11:44

Made changes as per review comments

b2131a4

added tests

48128e4

sandeep-paliwal added 3 commits October 5, 2020 13:43

support hostName in BYO credentials

b406e00

Added tests

Throw error in case of sas byo creds

3b23711

Updated doc fixed tests for above change

minor fix for byo sas creds

1b7053d

moritzraho reviewed Oct 5, 2020

View reviewed changes

sandeep-paliwal added 2 commits October 5, 2020 16:50

closed review comments

17de909

removed unnecessary tests

ff2e6b8

moritzraho approved these changes Oct 5, 2020

View reviewed changes

added jsdoc comment

12ebe86

moritzraho marked this pull request as ready for review October 5, 2020 14:37

moritzraho merged commit 08e13f0 into adobe:master Oct 5, 2020

moritzraho mentioned this pull request Feb 26, 2021

Support for custom CDN host option #55

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BYO creds related issue #66

BYO creds related issue #66

sandeep-paliwal commented Oct 1, 2020 •

edited by moritzraho

moritzraho left a comment •

edited

codecov bot commented Oct 5, 2020 •

edited

moritzraho Oct 5, 2020

sandeep-paliwal Oct 5, 2020

moritzraho Oct 5, 2020

moritzraho Oct 5, 2020

moritzraho Oct 5, 2020

moritzraho Oct 5, 2020

moritzraho Oct 5, 2020

sandeep-paliwal Oct 5, 2020

moritzraho Oct 5, 2020

moritzraho left a comment

BYO creds related issue #66

BYO creds related issue #66

Conversation

sandeep-paliwal commented Oct 1, 2020 • edited by moritzraho

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

moritzraho left a comment • edited

Choose a reason for hiding this comment

codecov bot commented Oct 5, 2020 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moritzraho left a comment

Choose a reason for hiding this comment

sandeep-paliwal commented Oct 1, 2020 •

edited by moritzraho

moritzraho left a comment •

edited

codecov bot commented Oct 5, 2020 •

edited