Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor handling of max size for html/js/css #525

Merged
merged 5 commits into from
Apr 3, 2024

Conversation

ikreymer
Copy link
Member

@ikreymer ikreymer commented Apr 2, 2024

  • due to a typo (and lack of type-checking!) incorrectly passed in matchFetchSize instead of maxFetchSize, resulting in text/css/js for >5MB instead of >25MB not properly streamed back to the browser
  • add type checking to AsyncFetcherOptions to avoid this in the future.
  • refactor to avoid checking size altogether for 'essential resources', html(document), js and css, instead always fetch them fully and continue in the browser. Only apply rewriting if <25MB.
  • fixes Website fails to fully load properly since 1.0.0 #522

- due to a typo (and lack of type-checking!) incorrectly passed in matchFetchSize instead of maxFetchSize,
resulting in text/css/js for >5MB instead of >25MB not properly streamed back to the browser
- add type checking to AsyncFetcherOptions to avoid this
- raise the text size to 50MB instead of 25MB just in case
- fixes #522
@ikreymer ikreymer requested a review from tw4l April 2, 2024 18:28
Copy link
Contributor

@tw4l tw4l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch and thanks for the typing improvements!

src/util/recorder.ts Outdated Show resolved Hide resolved
@tw4l
Copy link
Contributor

tw4l commented Apr 2, 2024

Confirmed it resolves the issue with the example in #522. I'd recommend setting a --limit on the crawl when testing.

src/util/recorder.ts Outdated Show resolved Hide resolved
ikreymer and others added 4 commits April 2, 2024 14:06
- always fetch 'essential resources', html/js/css into memory, regardless of size/even if size is unknown
- only attempt rewriting if size <25MB
- warn if large non-essential resource is not being streamed back to browser (due to size)
…a document, script or stylesheet,

always load w/o streaming, for others, check size.
@ikreymer ikreymer changed the title fix check for max size for data streamed back to browser: refactor handling of max size for html/js/css Apr 3, 2024
@ikreymer ikreymer requested a review from tw4l April 3, 2024 14:52
@tw4l
Copy link
Contributor

tw4l commented Apr 3, 2024

Changes make sense and tested that it's still working well with example site now. Looks good to merge to me.

@ikreymer ikreymer merged commit a3f93ca into v1.0.4-release Apr 3, 2024
4 checks passed
@ikreymer ikreymer deleted the max-text-type-fix branch April 3, 2024 22:17
ikreymer added a commit that referenced this pull request Apr 4, 2024
refactor handling of max size for html/js/css (copy of #525)
- due to a typo (and lack of type-checking!) incorrectly passed in
matchFetchSize instead of maxFetchSize, resulting in text/css/js for
>5MB instead of >25MB not properly streamed back to the browser
- add type checking to AsyncFetcherOptions to avoid this in the future.
- refactor to avoid checking size altogether for 'essential resources',
html(document), js and css, instead always fetch them fully and
continue in the browser. Only apply rewriting if <25MB.
fixes #522
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants