@astrojs/cloudflare - Robot.txt not valid

[RodrigoTomeES]

Astro Info

Astro                    v3.5.5
Node                     v20.9.0
System                   Windows (x64)
Package Manager          npm
Output                   hybrid
Adapter                  @astrojs/cloudflare
Integrations             @astrojs/tailwind
                         @astrojs/sitemap

Describe the Bug

PageSpeed said that Robot.txt is not valid

What's the expected result?

A valid Robot

Link to Minimal Reproducible Example

https://pagespeed.web.dev/analysis/https-rodrigotome-es/ee7rnvsq5l?form_factor=mobile

Participation

• I am willing to submit a pull request for this issue.

[alexanderniebuhr]

@RodrigoTomeES thanks for the issue report. However this is not a issue with Astro or the Cloudflare adapter. AFAICT you don't have a robots.txt in your public folder, therefore the page does not exists. However you have your site deployed as SPA to Cloudflare, which in fact does fallback to / for unfound routes.
You could try routes.strategy: exclude, but I'm not sure if that works. Otherwise you would need a static (SSG) 404 page, which will be available as 404.html for Cloudflare routing.

[RodrigoTomeES]

@alexanderniebuhr Hi, thanks. I thought that astro sitemap add a Robots.txt 🤔, I will investigate it and add a 404 page.

Sorry for the wrong issue

[alexanderniebuhr]

Oh I'm unsure about @astrojs/sitemap, if you can validate that it adds a file in the dist folder and that has wrong syntax, please let me know here and we can move the issue to the correct repo.

I also advice again to check the output of the _routes.json file. It should give clarity if Cloudflare runs in SPA mode, and why the routing might not work.

[RodrigoTomeES]

@alexanderniebuhr you are right, @astrojs/sitemap doesn't generate the sitemap, also not add the sitemap to the head. I think it should be added as a feature in that package like next-sitemap do.

This is my _routes.json

{
  "version": 1,
  "include": [
    "/_image",
    "/api/spotify"
  ],
  "exclude": []
}

How can I know if it is deployed as a SPA? I didn't touch any config of cloudflare plugin.

Thanks for all!

[alexanderniebuhr]

https://developers.cloudflare.com/pages/platform/serving-pages/#single-page-application-spa-rendering

[juanguillermo040488]

For those who have the same problem, I was able to solve it by adding a 404.astro component at the same level as index.astro.

In the particular case of Cloudflare, if it does not detect a 404.index error page, it will infer that the website is a SPA and will redirect everything to index.astro (source).

So even though /robots.txt existed, entering the path redirected me to /.

Just add an error page and you're done!

As of today, the official sitemap plugin does not add the robots.txt to /public, you must add it manually or follow these instructions: https://docs.astro.build/es/guides/integrations-guide/sitemap/#uso

Thanks @alexanderniebuhr for share that link.