Migrate from s3 sdk v2 to v3 #6731

lng2020 · 2024-03-28T17:07:32Z

close #5683. Nothing breaking, test with my local setup and everything works.
Noticeable change:
The s3 SDK v3 method PutObjectCommand doesn't support stream input and it requires a content-length field to be set, which is discussed in aws/aws-sdk-js-v3#2348.
So I didn't use PutObjectCommand because I think loading whole files into the buffer may cause a burden. As the issue comment above suggests, I use the Upload class from @aws-sdk/lib-storage which is compatible with v2.

Just in case anyone needs a clear answer: putObject can't write a stream to S3, you have to use the Upload class in @aws-sdk/lib-storage

CLAassistant · 2024-03-28T17:07:38Z

All committers have signed the CLA.

lng2020 · 2024-03-28T18:46:31Z

Need to polish about signature-v4-crt

tommoor · 2024-03-28T18:47:52Z

Appreciate the help here 🙏

lng2020 · 2024-03-29T04:14:20Z

package.json

@@ -47,6 +47,11 @@
    "> 0.25%, not dead"
  ],
  "dependencies": {
+    "@aws-sdk/client-s3": "3.535.0",
+    "@aws-sdk/lib-storage": "3.535.0",
+    "@aws-sdk/s3-presigned-post": "3.535.0",


My local env can work without importing @aws-sdk/s3-presigned-post as expected. But CI cannot find this module so I imported it here. Don't know the root cause and I found a similar issue with aws-sdk/middleware-endpoint before(aws/aws-sdk-js-v3#3983).

server/storage/files/S3Storage.ts

lng2020 · 2024-03-29T04:17:20Z

server/storage/files/S3Storage.ts

      region: env.AWS_REGION,
      endpoint: this.getEndpoint(),
-      signatureVersion: "v4",


This condition is deprecated and now is replaced by import "@aws-sdk/signature-v4-crt"; according to https://github.com/aws/aws-sdk-js-v3#functionality-requiring-aws-common-runtime-crt

lng2020 · 2024-03-29T04:20:36Z

ready for review

tommoor · 2024-04-03T03:01:44Z

So, there is a bug here – when downloading a document that includes an embedded image (Document -> ... -> Download -> Markdown), the image in the generated zip file is 0 bytes.

This would suggest that getFileStream is not working correctly. I checked out main to test it continues to work correctly there.

tommoor · 2024-04-03T03:05:35Z

server/storage/files/S3Storage.ts

-    try {
-      return this.client
-        .getObject({
+    this.client


The response is no longer returned, can we restore this to async/await style?

done in d520b03 (#6731) . I'm not so familiar with all promise, await, and async things. Please feel free to correct me if I'm doing wrong.

lng2020 · 2024-04-03T18:42:24Z

So, there is a bug here – when downloading a document that includes an embedded image (Document -> ... -> Download -> Markdown), the image in the generated zip file is 0 bytes.

From my local test, it's been fixed by d520b03 (#6731). There is a long discussion about this getObjectComand aws/aws-sdk-js-v3#1877 about this inflexibility command 😄 and It seems required to use the Readable type. Let me know if there are other malfunctions or other improvements.

tommoor · 2024-04-04T00:27:55Z

I really don't want to change the signature of the storage interface here, it's weird to have a method called getFileStream that returns a promise :/ It feels like it was very close before the last round of changes with the exception of that one issue

lng2020 · 2024-04-04T00:44:45Z

can we restore this to async/await style?

I assume this means using async/await in getFileStream and results in a promise returned so I change the signature. Could u kindly specify what the right approach is?

tommoor · 2024-04-04T00:53:17Z

I think you're right, this is a terrible limitation of the new version – how annoying.

tommoor · 2024-05-19T01:47:16Z

server/storage/files/S3Storage.ts

    invariant(
      env.AWS_S3_UPLOAD_BUCKET_NAME,
      "AWS_S3_UPLOAD_BUCKET_NAME is required"
    );

-    this.client
+    return this.client


This seemed to be a bug – nothing returned here, but with this change I think it's ready to merge

…97bcc by renovate (#23333) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [outlinewiki/outline](https://togithub.com/outline/outline) | minor | `0.76.1` -> `0.77.0` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>outline/outline (outlinewiki/outline)</summary> ### [`v0.77.0`](https://togithub.com/outline/outline/releases/tag/v0.77.0) [Compare Source](https://togithub.com/outline/outline/compare/v0.76.1...v0.77.0) #### What's Changed This release changes how documents are rendered, allowing the introduction of new editor functionality and attributes. Highlighted for this release are some big improvements to tables: ![image](https://togithub.com/outline/outline/assets/380914/6109e4a5-fc5a-4a25-ae33-eebbe9adc0db) - **Column resizing:** You can now resize columns by dragging the border between them - **Full-width tables:** Tables now expand to the full width of the editor - **More content:** Table cells now support embeds, code blocks, and all other content types - **Toggleable headers:** You can now toggle a header row and column - **Improved overflow:** Tables now scroll horizontally when they're too wide for the editor - **Improved row and column controls**: We've made it one click to add rows and columns ##### Other improvements - Added more highlighter color choices in [outline/outline#7012 - New collection permissions UI in [outline/outline#6917 - Big rendering performance improvements for documents with many images - Add `Manage` permission to documents allows other users to manage sharing in [outline/outline#7003 - Added an embedded Yandex Maps component by [@shashinma](https://togithub.com/shashinma) in [outline/outline#6881 - Added a permission selector in document/collection invite flow in [outline/outline#6948 - Added support for embedded youtube clips by [@Pranav2612000](https://togithub.com/Pranav2612000) in [outline/outline#6942 - Matomo integration in [outline/outline#7009 - `/` block insert shortcut now works anywhere on a line or table in [outline/outline#7026 - Added the ability to remove a previously uploaded team logo or profile picture - Added a menu option to create template directly in collection ##### Fixes - Various fixes to improve the fluidity of app loading - Image placeholders are now correctly sized when uploading retina images - Images and PDF's are now served with inline content disposition in [outline/outline#6924 - Added support for Figma's new link formats and Figjam boards by [@RenderCoder](https://togithub.com/RenderCoder) in [outline/outline#6946 - `files.get` now supports range header, fixes video loading in Safari in [outline/outline#6950 - Fixed an issue where zoomed images could get stuck in [outline/outline#6980 - Share button was removed on templates - Fixed race condition that could result in document emoji not saving - Removed trust of `state.host` in auth error redirect - Collapsed sidebar now works better in Arc/Edge redesign #### New Contributors - [@shashinma](https://togithub.com/shashinma) made their first contribution in [outline/outline#6881 - [@lng2020](https://togithub.com/lng2020) made their first contribution in [outline/outline#6731 - [@RenderCoder](https://togithub.com/RenderCoder) made their first contribution in [outline/outline#6946 - [@Frzk](https://togithub.com/Frzk) made their first contribution in [outline/outline#6971 - [@bkrausz](https://togithub.com/bkrausz) made their first contribution in [outline/outline#7022 **Full Changelog**: outline/outline@v0.76.0...v0.77.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

* chore: migrate from s3 sdk v2 to v3 * import signature-v4-crt * downgrade minor version * Add s3-presigned-post manually * Change s3 mock * Update server/storage/files/S3Storage.ts * docs * Upgrade aws-sdk --------- Co-authored-by: Tom Moor <tom.moor@gmail.com>

auto-assign bot requested a review from tommoor March 28, 2024 17:07

chore: migrate from s3 sdk v2 to v3

09eb61f

lng2020 force-pushed the migrate-s3-sdk branch from 0037685 to 09eb61f Compare March 28, 2024 18:29

lng2020 marked this pull request as draft March 28, 2024 18:45

lng2020 added 4 commits March 29, 2024 11:11

import signature-v4-crt

8dc8d83

downgrade minor version

cca160a

Add s3-presigned-post manually

c130f83

Change s3 mock

aa68b2e

lng2020 marked this pull request as ready for review March 29, 2024 04:14

lng2020 commented Mar 29, 2024

View reviewed changes

Update server/storage/files/S3Storage.ts

3c7adbd

Merge branch 'main' of github.com:outline/outline into migrate-s3-sdk

26f87a1

tommoor reviewed Apr 3, 2024

View reviewed changes

fix: getFileStream

0553e12

tommoor force-pushed the migrate-s3-sdk branch from f2de2f8 to 0553e12 Compare May 19, 2024 01:11

docs

05f27c0

tommoor reviewed May 19, 2024

View reviewed changes

tommoor approved these changes May 19, 2024

View reviewed changes

Upgrade aws-sdk

881c469

tommoor merged commit 3a7dd94 into outline:main May 19, 2024
11 checks passed

bagedevimo mentioned this pull request Jun 17, 2024

AWS S3 uploads broken when using instance roles on AWS #7060

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate from s3 sdk v2 to v3 #6731

Migrate from s3 sdk v2 to v3 #6731

lng2020 commented Mar 28, 2024 •

edited

Loading

CLAassistant commented Mar 28, 2024 •

edited

Loading

lng2020 commented Mar 28, 2024

tommoor commented Mar 28, 2024

lng2020 Mar 29, 2024 •

edited

Loading

lng2020 Mar 29, 2024

lng2020 commented Mar 29, 2024

tommoor commented Apr 3, 2024

tommoor Apr 3, 2024

lng2020 Apr 3, 2024

lng2020 commented Apr 3, 2024 •

edited

Loading

tommoor commented Apr 4, 2024

lng2020 commented Apr 4, 2024

tommoor commented Apr 4, 2024

tommoor May 19, 2024

Migrate from s3 sdk v2 to v3 #6731

Migrate from s3 sdk v2 to v3 #6731

Conversation

lng2020 commented Mar 28, 2024 • edited Loading

CLAassistant commented Mar 28, 2024 • edited Loading

lng2020 commented Mar 28, 2024

tommoor commented Mar 28, 2024

lng2020 Mar 29, 2024 • edited Loading

Choose a reason for hiding this comment

lng2020 Mar 29, 2024

Choose a reason for hiding this comment

lng2020 commented Mar 29, 2024

tommoor commented Apr 3, 2024

tommoor Apr 3, 2024

Choose a reason for hiding this comment

lng2020 Apr 3, 2024

Choose a reason for hiding this comment

lng2020 commented Apr 3, 2024 • edited Loading

tommoor commented Apr 4, 2024

lng2020 commented Apr 4, 2024

tommoor commented Apr 4, 2024

tommoor May 19, 2024

Choose a reason for hiding this comment

lng2020 commented Mar 28, 2024 •

edited

Loading

CLAassistant commented Mar 28, 2024 •

edited

Loading

lng2020 Mar 29, 2024 •

edited

Loading

lng2020 commented Apr 3, 2024 •

edited

Loading