TechDocs: Support AWS S3 as TechDocs external storage #3794

ayshiff · 2020-12-20T21:17:05Z

Hey, I just made a Pull Request!

Closes #3714

This PR adds the feature to use AWS S3 as TechDocs external storage.

Added a new awsS3.ts publisher Link
Added tests and mocks Link
Added steps about how to use AWS S3 Link
Updated config reference documentation Link

✔️ Checklist

A changeset describing the change and affected packages. (more info)
Added or updated documentation
Tests for new functionality and regression tests for bug fixes
Screenshots attached (for UI changes)

changeset-bot · 2020-12-20T21:17:08Z

🦋 Changeset detected

Latest commit: 1517009

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@backstage/techdocs-common	Patch
@backstage/plugin-techdocs-backend	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

OrkoHunter

Looking great so far @ayshiff ! Kudos for your work on this and supporting documentation as well.

Leaving some comments. Will take another look very soon. 🎉

.changeset/fast-colts-cross.md

app-config.yaml

docs/features/techdocs/configuration.md

docs/features/techdocs/using-cloud-storage.md

packages/techdocs-common/src/stages/publish/awsS3.ts

plugins/techdocs-backend/src/service/router.ts

.changeset/fast-colts-cross.md

andrewthauer

Excited this is being added.

andrewthauer · 2020-12-21T00:15:07Z

docs/features/techdocs/configuration.md

+    awsS3:
+      # An API key is required to write to a storage bucket.
+      credentials:
+        $file: '/path/to/aws_application_credentials.json',


Curious, can the credentials be set with just environment variables? Having a physical secrets file might be difficult to setup in some environments.

I agree with you.
At first I wanted to put the credentials as environment variables:

credentials: aws_access_key_id: ACCESS_KEY_ID aws_secret_access_key: SECRET_ACCESS_KEY

But to keep some consistency with the configuration for GCS I thought it might be interesting to keep the same credentials file logic.

But since it is not very relevant for aws we can perhaps replace file credentials by environment variables 👍

@OrkoHunter I would like to know your opinion on that.

Curious, can the credentials be set with just environment variables?

Yes! We can use the contents of the "**credentials.json" as the value for the environment variable (e.g. TECHDOCS_AWSS3_CREDENTIALS). And set

awsS3: credentials: $env: TECHDOCS_AWSS3_CREDENTIALS

I have used this with GCS on the demo site. But I see that it will be nice to write this ^ in the app-config.yaml comments and the Config reference docs.

Wouldn't the environment variable need to be a JSON string in this case? I'm personally not a big fan of managing them that way. It can make it more challenging to deal with some secret management processes. I'd rather be able to split out each field separately. If possible that would be preferable imo.

@andrewthauer I agree with you that JSON strings are not pretty in environment variables.

Here is my concern. With GCS, their credentials JSON looks like this

{ "type": "", "project_id": "", "private_key_id": "", "private_key": "-----BEGIN PRIVATE KEY-----<Insert 1600 characters>-----END PRIVATE KEY-----\n", "client_email": "", "client_id": "", "auth_uri": "", "token_uri": "", "auth_provider_x509_cert_url": "", "client_x509_cert_url": "" }

Even if we split these out into individual fields, the private_key will still be a huge JSON string. So, I do not see much of an improvement here over the existing ways i.e.

Copy over the credentials file as a huge $env variable.

(Kubernetes) Inject the $file using Kubernetes secrets management. https://kubernetes.io/docs/concepts/configuration/secret/#overview-of-secrets

...other typical ways to make a secret file available on a server.

That being said, if AWS allows smaller alphanumeric token-type secrets that can be generated and set as environment variables, I too would prefer that over the huge JSON string. It's just that, GCS is holding us back with how it authenticates service accounts.

If I run Backstage as a container in ECS will I be able to skip all of this config and use the role that is assigned to the task?

@mfrinnstrom Role and Region are not used as of this moment. Only bucket name and credentials are needed.

@OrkoHunter that was my concern. If I add credentials to my task in AWS ECS that would be picked up automatically by the AWS SDK. It seems like we bypass this here and require credentials to be provided from the app.

If I add credentials to my task in AWS ECS

Is this same as setting environment variables? We are open for suggestions. When the AWS sdk is initialized here, should it check some existing environment variables?
May I also ask what are the downsides of explicitly setting those environment variables in app-config.yaml?

@OrkoHunter not really, although the AWS SDK has some standard environment variables that are picked up automatically if available. Some more information can be found here.

We usually create roles and assign them to our different workloads, in this case an ECS task. This will then be picked up by the AWS SDK in the ECS task without having to create a user and generate credentials for that, and then having to securely manage and supply those credentials to the workload.

This works great when running in AWS but when running outside of AWS you will need something like what is implemented right now with the access and secret keys as environment variables. It should probably be possible to use the default environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) and have the SDK pick that up automatically, but that becomes a bit hidden and maybe not to easy to follow in the config.

docs/features/techdocs/using-cloud-storage.md

OrkoHunter

I have tested the setup with S3 and it works great! Thanks again @ayshiff for working on this.

ayshiff · 2020-12-21T15:10:38Z

I will fix the tests and fix the techdocs JSON schema by adding the region property 👍

OrkoHunter · 2020-12-21T15:12:31Z

Awesome!

ayshiff · 2020-12-21T15:42:58Z

I don't know if you have reviewed the example diagram for aws. Link

I used excalidraw to draw it but saw that the other graphs were made with draw.io.
Do you think it's better that I redo the graph with draw.io to keep the documentation consistent ?

OrkoHunter · 2020-12-21T15:54:39Z

@ayshiff Good that you noticed it! If you make it with draw.io and save as an svg, it will be easier to maintain/update! Try the draw.io VS Code extension and open a file with .drawio.svg extension to experience the magic. ✨

andrewthauer · 2020-12-21T18:14:40Z

@ayshiff Good that you noticed it! If you make it with draw.io and save as an svg, it will be easier to maintain/update! Try the draw.io VS Code extension and open a file with .drawio.svg extension to experience the magic. ✨

Or plantuml if you'd rather not think or fiddle with layout 😄

andrewthauer

Question - Will publishers support using nested directories? For instance, you want to share an s3 bucket for other purposes and the documentation sites are located in say /docs/sites as the root.

andrewthauer · 2020-12-21T18:23:50Z

packages/techdocs-common/package.json

@@ -42,6 +42,7 @@
    "@google-cloud/storage": "^5.6.0",
    "@types/dockerode": "^3.2.1",
    "@types/express": "^4.17.6",
+    "aws-sdk": "^2.813.0",


It looks like v3 was just released. Is it worth taking a look? Seems like it has better TS support - https://aws.amazon.com/about-aws/whats-new/2020/12/aws-sdk-javascript-version-3-generally-available/.

Nice! @ayshiff Would you this this a try? You can create an issue for the upgrade if you want to do it later or want someone else to do it - and that is totally fine.

Yes of course ! I'll look into it 👍

plugins/techdocs/package.json

OrkoHunter · 2020-12-21T18:36:15Z

Will publishers support using nested directories? For instance, you want to share an s3 bucket for other purposes and the documentation sites are located in say /docs/sites as the root.

Great question @andrewthauer and big +1 for adding that support. I'll create an issue for this. I imagine it can be defined as a rootPath inside techdocs.publisher.awsS3. Will do for both GCS and S3.

ayshiff · 2020-12-22T09:58:23Z

packages/techdocs-common/src/stages/publish/awsS3.ts

+          const techdocsMetadataJson = file?.Body?.toString();
+
+          if (!techdocsMetadataJson) {
+            throw new Error(
+              `Unable to parse the techdocs metadata file ${entityRootDir}/techdocs_metadata.json.`,
+            );
+          }


Do we have another way to get the Readable | ReadableStream | Blob in a better way ?

ayshiff · 2020-12-22T10:17:16Z

@OrkoHunter Do you know why the lint and tsc commands fail on the CI on a line that does not seem to exist ?

OrkoHunter · 2020-12-22T10:48:56Z

Hey @ayshiff! Could you try pulling the latest remote master branch into your feature branch? (Or do git rebase) That will cause some conflicts, maybe.

This #3764 was merged recently.

OrkoHunter · 2020-12-22T11:37:03Z

@ayshiff The failing test is gone (we had to rebase and fix merge conflict, that's all). Now there is a new tsc issue with the aws-sdk package it seems.

I also had to run yarn again after copying over yarn.lock from the master branch to sync.

ayshiff · 2020-12-22T11:45:57Z

Thanks @OrkoHunter 👍 I'll look into it.

ayshiff · 2020-12-22T11:51:39Z

I wonder why I didn't have conflicts while doing my rebase 🤔
I was probably not up to date with the remote. My bad.

ayshiff · 2020-12-22T12:01:50Z

It looks like an issue aws-sdk-js-v3 has: Related issue

andrewthauer · 2020-12-22T13:57:34Z

It looks like an issue aws-sdk-js-v3 has: Related issue

That's unfortunate. Maybe not worth the version bump until that is sorted out.

ayshiff · 2020-12-22T14:54:38Z

I'm going to move the v3 bump commits to another branch and I'm going to create a Draft PR while waiting for the issue to be fixed in #1812.

Does it sound good to you ?

also copy yarn.lock from master; and yarn again fix(techdocs-common): broken async test

lowjoel · 2020-12-31T01:22:04Z

packages/techdocs-common/src/stages/publish/awsS3.ts

+    let secretAccessKey = null;
+    let bucketName = '';
+    try {
+      accessKeyId = config.getString(


Possible improvement for a future iteration (happy to contribute if we reach there first): this should use either EC2 instance profiles/Kubernetes IRSA to remove the need to provide an access key. These credentials are automatically rotated and support is built into the AWS SDK.

Typically in AWS SDKs if you credentials are omitted it uses the default precedence rules, which (from memory) is env vars, ~/.aws/credentials, instance profile/IRSA

Maybe have the credentials as optional and if not present use the default credentials chain that @lowjoel mentions? Just new aws.S3() basically.

We would also be happy to contribute this since we have a policy to never create users and use access keys. I can understand that this might be necessary if you are not running in AWS but still want to use S3 though.

I'll be happy to add this to the existing PR !

@OrkoHunter Do we keep this PR in current state and iterate over it once merged to add optional credentials ? Or do we add that on this branch ?

@ayshiff IMO let us iterate in the future! We shouldn't delay this PR more. Hopefully we can merge this next week. :)

OrkoHunter · 2021-01-02T17:49:59Z

@backstage/maintainers Please review when possible!

benjdlambert · 2021-01-02T20:52:33Z

app-config.yaml

@@ -76,7 +76,7 @@ techdocs:
  generators:
    techdocs: 'docker' # Alternatives - 'local'
  publisher:
-    type: 'local' # Alternatives - 'googleGcs'. Read documentation for using alternatives.
+    type: 'local' # Alternatives - 'googleGcs' or 'awsS3'. Read documentation for using alternatives.


Not for now - But wondering if we can drop the provider prefix for these types? s3 and gcs reads much better than awsS3 and googleGcs respectively - thoughts @backstage/techdocs-core?

Thinking that maybe we would favour some form of configuration that looks like the below, where all the services below a provider live underneath a provider?

aws: s3: access_key_id: $S3_AWS_ACCESS_KEY_ID secret_access_key: $S3_AWS_SECRET_ACCESS_KEY redshift: access_key_id: $REDSHIFT_AWS_ACCESS_KEY_ID secret_access_key: $REDSHIFT_AWS_SECRET_ACCESS_KEY default: access_key_id: $AWS_ACCESS_KEY_ID secret_access_key: $AWS_SECRET_ACCESS_KEY

Hmm, interesting idea of unification of configs based on provider as you write above. I think we might do this just like how we did the integrations with GitHub/GitLab/etc. ➕
As of now for techdocs.publisher.type, IMHO googleGcs and awsS3 are slightly better than gcs and s3. :)

benjdlambert · 2021-01-02T20:54:35Z

docs/features/techdocs/configuration.md

+    # Required when techdocs.publisher.type is set to 'awsS3'. Skip otherwise.
+
+    awsS3:
+      # An API key is required to write to a storage bucket.


Does the sdk also read from the instance profile or ~/.aws/credentials too?

Yep! There are some additional ways to handle S3 configuration apart from creating an access user agent (discussed above in various threads). Thinking we can address it in a separate PR (to also improve the GCS auth on similar patterns).

packages/techdocs-common/src/stages/publish/awsS3.ts

OrkoHunter · 2021-01-05T10:27:25Z

@ayshiff Do you think this is good to go? I'll do a final Quality Assurance on this and merge. Could you also highlight some future fixes/improvements? We can create issues for them.

ayshiff · 2021-01-05T12:32:01Z

I think this is good to go, I’ll do a last check in the afternoon.

Here are the future fixes/improvements that I noted in the comments:

handle S3 configuration apart from creating an access user agent ( add the possibility to read from the instance profile or ~/.aws/credentials ) Link
Drop provider prefixes ? (awsS3 -> s3 and googleGcs -> gcs) Link
Unification of configs based on provider Link
aws-sdkversion bump (already implemented on a branch, waiting for some TypeScript issues to be fixed on aws-sdk-js-v3.)
enable publishers support using nested directories (gcs + s3)

OrkoHunter · 2021-01-05T13:15:08Z

@ayshiff Do let me know when it's ready according to you.
Thank you for summarising the improvements. I can go ahead and create separate issues for them once this is merged. But feel free and let me know if you want to create them yourself. :)

ayshiff · 2021-01-05T15:12:35Z

It looks good for me !
Do you want me to clean up the commit history ? (squash some commits ...)

I’ll be happy to create them. :)
For issues involving aws s3 and gcs do I create two issues or a single one that includes both ?

OrkoHunter · 2021-01-05T15:37:31Z

Do you want me to clean up the commit history ? (squash some commits ...)

No, the commit history looks good to me!

I’ll be happy to create them. :)
For issues involving aws s3 and gcs do I create two issues or a single one that includes both ?

Awesome and thank you! This is what I think -

(Only S3) Improved S3 credentials handling. Explicitly defining configs should not be required.
(Only GCS) Same ^. One should be able to do new GCS(), and it should not complain about missing configs. The logic should be - during initialization, if the publisher is able to read the metadata of the bucket -> Connected; unable to read metadata -> Throw error.
(Only S3) aws-sdk version bump (already implemented on a branch, waiting for some TypeScript issues to be fixed on aws-sdk-js-v3.)
(Joint) Enable publishers support using nested directories in a bucket

OrkoHunter · 2021-01-05T16:52:15Z

You did awesome on this PR @ayshiff! 👏 I appreciate your attention to detail with the changes you did, and the reviews you addressed (sparked many good conversations). :)

ayshiff · 2021-01-05T17:20:10Z

I think the problem with failed tests on master is due to bad path delimiters.
Do we revert the PR and add that or do I create another PR that fixes the tests ?

OrkoHunter · 2021-01-05T17:22:10Z

@ayshiff New one would be good! (Windows paths 😬)

OrkoHunter · 2021-01-05T17:24:05Z

You could use something like this in the tests. (Looking at existing examples)

const path = os.platform() === 'win32' ? 'C:\\rootDir' : '/rootDir';

ayshiff requested review from a team as code owners December 20, 2020 21:17

OrkoHunter reviewed Dec 20, 2020

View reviewed changes

OrkoHunter added this to Incoming in TechDocs project board via automation Dec 20, 2020

OrkoHunter moved this from Incoming to To Review/Review In Progress ⏳ in TechDocs project board Dec 20, 2020

OrkoHunter reviewed Dec 20, 2020

View reviewed changes

.changeset/fast-colts-cross.md Outdated Show resolved Hide resolved

andrewthauer reviewed Dec 21, 2020

View reviewed changes

OrkoHunter reviewed Dec 21, 2020

View reviewed changes

docs/features/techdocs/using-cloud-storage.md Outdated Show resolved Hide resolved

OrkoHunter approved these changes Dec 21, 2020

View reviewed changes

OrkoHunter mentioned this pull request Dec 21, 2020

Introduced config schema for techdocs plugins #3799

Merged

4 tasks

andrewthauer reviewed Dec 21, 2020

View reviewed changes

ayshiff commented Dec 22, 2020

View reviewed changes

ayshiff force-pushed the feature/techdocs-aws-s3 branch from a24f3d2 to 21d654d Compare December 22, 2020 10:54

OrkoHunter force-pushed the feature/techdocs-aws-s3 branch from a5a39d4 to fd0a9d8 Compare December 22, 2020 11:25

ayshiff force-pushed the feature/techdocs-aws-s3 branch from fd0a9d8 to 9b78b1a Compare December 22, 2020 16:45

OrkoHunter and others added 4 commits December 30, 2020 10:25

fix(techdocs-common) broken test

dc99b07

also copy yarn.lock from master; and yarn again fix(techdocs-common): broken async test

feat(techdocs): update aws-S3 diagram

3169c9b

feat(techdocs): update config schema

3418938

feat(techdocs): update dependency versions

3aa98a6

ayshiff force-pushed the feature/techdocs-aws-s3 branch from acdcf94 to 3aa98a6 Compare December 30, 2020 09:34

lowjoel reviewed Dec 31, 2020

View reviewed changes

benjdlambert reviewed Jan 2, 2021

View reviewed changes

refactor(techdocs-common): add try/catch statements + fs-extra

1517009

OrkoHunter mentioned this pull request Jan 3, 2021

TechDocs: Support AWS S3 as TechDocs external storage (to store and read generated documentation sites) #3714

Closed

gustavohalmvia approved these changes Jan 4, 2021

View reviewed changes

OrkoHunter requested a review from benjdlambert January 5, 2021 09:03

benjdlambert approved these changes Jan 5, 2021

View reviewed changes

OrkoHunter merged commit 0b9646e into backstage:master Jan 5, 2021

TechDocs project board automation moved this from To Review/Review In Progress ⏳ to Done ✅ Jan 5, 2021

ayshiff mentioned this pull request Jan 5, 2021

TechDocs: Fix tests path delimiters #3925

Merged

4 tasks

OrkoHunter mentioned this pull request Jan 6, 2021

TechDocs: Support Azure Storage as TechDocs external storage #3938

Closed

This was referenced Jan 6, 2021

TechDocs: AWS S3 SDK version bump #3944

Closed

TechDocs: Improve S3 credentials handling #3946

Closed

TechDocs: AWS S3 SDK version bump #3953

Merged

OrkoHunter mentioned this pull request Jan 14, 2021

[TechDocs] Add option to configure a sub-directory of the external Storage to be used #4070

Closed

TechDocs: Support AWS S3 as TechDocs external storage #3794

TechDocs: Support AWS S3 as TechDocs external storage #3794

Conversation

ayshiff commented Dec 20, 2020 • edited by OrkoHunter Loading

Hey, I just made a Pull Request!

✔️ Checklist

changeset-bot bot commented Dec 20, 2020 • edited Loading

🦋 Changeset detected

OrkoHunter left a comment

Choose a reason for hiding this comment

andrewthauer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OrkoHunter Dec 21, 2020 • edited Loading

Choose a reason for hiding this comment

andrewthauer Dec 21, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mfrinnstrom Dec 22, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OrkoHunter left a comment

Choose a reason for hiding this comment

ayshiff commented Dec 21, 2020

OrkoHunter commented Dec 21, 2020

ayshiff commented Dec 21, 2020

OrkoHunter commented Dec 21, 2020

andrewthauer commented Dec 21, 2020 • edited Loading

andrewthauer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OrkoHunter commented Dec 21, 2020

Choose a reason for hiding this comment

ayshiff commented Dec 22, 2020

OrkoHunter commented Dec 22, 2020

OrkoHunter commented Dec 22, 2020

ayshiff commented Dec 22, 2020

ayshiff commented Dec 22, 2020

ayshiff commented Dec 22, 2020 • edited Loading

andrewthauer commented Dec 22, 2020

ayshiff commented Dec 22, 2020

lowjoel Dec 31, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OrkoHunter commented Jan 2, 2021

Choose a reason for hiding this comment

OrkoHunter Jan 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OrkoHunter commented Jan 5, 2021

ayshiff commented Jan 5, 2021 • edited Loading

OrkoHunter commented Jan 5, 2021

ayshiff commented Jan 5, 2021

OrkoHunter commented Jan 5, 2021

OrkoHunter commented Jan 5, 2021

ayshiff commented Jan 5, 2021

OrkoHunter commented Jan 5, 2021 • edited Loading

OrkoHunter commented Jan 5, 2021 • edited Loading

ayshiff commented Dec 20, 2020 •

edited by OrkoHunter

Loading

changeset-bot bot commented Dec 20, 2020 •

edited

Loading

OrkoHunter Dec 21, 2020 •

edited

Loading

andrewthauer Dec 21, 2020 •

edited

Loading

mfrinnstrom Dec 22, 2020 •

edited

Loading

andrewthauer commented Dec 21, 2020 •

edited

Loading

ayshiff commented Dec 22, 2020 •

edited

Loading

lowjoel Dec 31, 2020 •

edited

Loading

OrkoHunter Jan 2, 2021 •

edited

Loading

ayshiff commented Jan 5, 2021 •

edited

Loading

OrkoHunter commented Jan 5, 2021 •

edited

Loading

OrkoHunter commented Jan 5, 2021 •

edited

Loading