[🐞] 404 in Search Console reported for /~partytown/partytown-sandbox-sw.html?{timestamp}

[MTheProgrammer]

Describe the bug

Hello, other people mentioned this problem, but they couldn't reproduce the bug:

• [🐞] Firefox: 404 error on "/partytown/partytown-sandbox-sw.html?XXXXXX" #441
• [🐞]Gatsby 404 for /~partytown/partytown-sandbox-sw.html #444

In Google Search Console every crawl introduces new 404 pages for /~partytown/partytown-sandbox-sw.html?XXX url:

Reproduction

https://lucidmodules.com/~partytown/partytown-sandbox-sw.html?1706003192708

Steps to reproduce

This might vary depending on whether you've already been on this page and web worker has been installed. However, when you clean the browser cache it should be as follows.

• go to page https://lucidmodules.com/~partytown/partytown-sandbox-sw.html?1706003192708
• open dev console and hard reload (in Chrome on Linux this is Ctrl+R)
• page returns 404
• soft reload - this time page shows the worker content

404 page on hard reload/first time download:

The correct page returned after worker has been installed:

Browser Info

Chrome

Additional Information

Maybe adding <meta name=”robots” content=”noindex,nofollow“> to the head would solve the issue with google bot trying to index the partytown-sandbox-sw.html page.

[gioboa]

This solution makes sense to me. Is there any specific reason to index this html, I don't think so. Do you have the possibility to change your html file and verify if it works correctly?

[MTheProgrammer]

I've forked the repo and updated the code that generates the html:
5061997

In Gatsby, these files are copied from the partytown to the static directory:

function setupPartytown() {
  const path = require("path");
  const { copyLibFiles } = require("@builder.io/partytown/utils");

  exports.onPreBuild = async () => {
    await copyLibFiles(path.join(__dirname, "static", "~partytown"));
  };
}

I'll check GSC after few days to verify whether this page is still being indexed.

[MTheProgrammer]

It doesn't seem to help:

[gioboa]

@MTheProgrammer I see 🤔 maybe is the ~partytown folder. Can you try to remove the ~ from the folder name pls?

[MTheProgrammer]

That's the official documentation with ~patytown directory: https://partytown.builder.io/gatsby#copy-library-files
You mean to change the folder name in all places where it is used?

The page ~partytown/partytown-sandbox-sw.html is dynamic, the static folder contains only .js files:

My guess is that Google robot crawls the page without cache and without cache it returns 404 - because Partytown worker has not yet been installed.

Every page includes iframe with the link to the Partytown. However, attribute rel="nofollow" is not valid as in the anchor tag <a href=www.example.com rel="nofollow">

EDIT: I'm testing a hack with empty physical ~/partytown/partytown-sandbox-sw.html file containing noindex,nofollow directive. When worker is ready it returns the correct dynamic page.

[gioboa]

I see, great research. So I'm wondering how serve a different/valid html for the crawler but preserve the Partytown code in the html 🤔

[f33w]

did noindex, nofollow the script folder help? Facing the same issue in GSC
@MTheProgrammer

[Echo203]

Started noticing daily 404s in GSC since last week, any development / solution on this?

[asertym]

Same issue, I've added the folder to robots.txt, but it would be great if these won't appear in GSC at all, since now Google warns me that "some pages are blocked in robots.txt". 🙈

[f33w]

Same issue, I've added the folder to robots.txt, but it would be great if these won't appear in GSC at all, since now Google warns me that "some pages are blocked in robots.txt". 🙈

if you care seo wise, then do not block it in robots, add meta robots noindex or response header noindex. Cause disallow means "Google, you can index this page if you want, but you will not know what's there". You should allow google anywhere, but prohibit it from indexing specific pages. Google itself acknowledges meta robots tag and response header as ways to go

[asertym]

@f33w thank you, was not aware of this detail.