Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perf Reduce s3 calls #295

Merged
merged 3 commits into from
Nov 2, 2023
Merged

Conversation

conico974
Copy link
Contributor

@conico974 conico974 commented Oct 23, 2023

The goal of this PR is to reduce the number of S3 call that we need to make for the S3 incremental cache

Before this PR there was 1 ListObjectsV2Command and 2 to 3 GetObjectCommand for every get of the incremental cache, this has been reduced to a single GetObjectCommand
For the set part of the incremental cache, we go from 2-3 PutObjectCommand to a single one.

This also reduce the number of objects in the S3 cache

TODO:

  • Update docs
  • Potentially filter the key not found error

@changeset-bot
Copy link

changeset-bot bot commented Oct 23, 2023

⚠️ No Changeset found

Latest commit: 8e5375c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@vercel
Copy link

vercel bot commented Oct 23, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
open-next ✅ Ready (Inspect) Visit Preview 💬 Add feedback Nov 2, 2023 4:59pm

packages/open-next/src/adapters/cache.ts Show resolved Hide resolved
// delete potential page data if we're redirecting
await this.deleteS3Objects(key);
await this.putS3Object(key, "redirect", JSON.stringify(data));
// // delete potential page data if we're redirecting
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment may be confusing, since the command is PUT and not DELETE. Perhaps you can provide more context?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad, i forgot to remove the comment

@@ -523,7 +508,7 @@ export default class S3Cache {

private async deleteS3Objects(key: string) {
try {
const regex = new RegExp(`\.(json|rsc|html|body|meta|fetch|redirect)$`);
const regex = new RegExp(`\.(fetch|cache)$`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is handled in Derek's PR but the \. should have been \\. to escape the RegExp constructor. Since there's no dynamic values here, it might be better just to use the inline one, eg: /\.(fetch|cache)$/

Copy link
Contributor

@khuezy khuezy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I deployed w/ this branch and everything looks good.
There are a few conflicts from Derek's PR though

@conico974 conico974 merged commit b7c8f47 into opennextjs:main Nov 2, 2023
3 checks passed
@github-actions github-actions bot mentioned this pull request Nov 2, 2023
conico974 added a commit to conico974/open-next that referenced this pull request Nov 17, 2023
@conico974 conico974 mentioned this pull request Nov 17, 2023
conico974 added a commit that referenced this pull request Nov 28, 2023
* fix: fetchCache for next 14+

* fix: regression from #295, tags were not generated

* test: add test for revalidateTag

* pin next version

* add changeset
@conico974 conico974 deleted the perf/reduce-s3-calls branch October 29, 2024 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants