Entirely disable streaming for htmlLimitedBots

[a463]

Goals

Content that is streamed will no longer be in <div hidden> for crawlers.

Non-Goals

No response

Background

Streaming metadata and htmlLimitedBots are a great addition, but I think it would be great to take this one step further. There are no real benefits to streaming HTML to bots at all. Sure most bots care only about the metadata, but still, there are many that find links, articles, images, etc... and there are concerns that search engines might ignore or diminish the importance of content within hidden elements.

Proposal

htmlLimitedBots is already there, so streaming can be entirely disabled if the useragent matches.

[Ubinquitous]

Even when the UserAgent matches htmlLimitedBots, is metadata currently being streamed?
Based on the logic, it seems like matching cases should show complete data rather than streaming, but I'm wondering if there's something I'm missing or overlooking.

next.js/packages/next/src/server/lib/streaming-metadata.ts

Line 9 in 64702a9

htmlLimitedBots: string | undefined

[icyJoseph]

If it matches, then it doesn't stream the metadata:

• Title tag position error. #79404 (comment)

If you are having some issue, could you try with canary?

[Ubinquitous]

I think I misspoke, I overlooked that this discussion was written around the time streaming metadata was released. Thanks for letting me know!