Fix og:image requests when html in a web page is over 1.megabyte #30362
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When truncating we were adding the chunk and then checking if it was over 1MB. This meant that it was always over 1MB with that included chunk. You need to check before adding the chunk, otherwise it will always fail. Sites like the New Yorker and Wired often have 1MB + html in their articles now (not good but what can you do).
This seems to fix a bunch of issues dealing with preview images not loading which were originally thought to be webp or avif errors: #18762 #14983 #27370 (but not all of them ie. The Guardian images seem to be having issues with a certain CDN key or something. Anyway, this should fix bunch of stuff because it won't raise the error and stop the return.)
ps. I haven't written much Ruby before so feel free to make any changes etc.