Replies: 2 comments 2 replies
-
It could be due to caching? We cache this page for 24h by default, so you won't see the changes until the next caching cycle. Is this the case? export const loader = ({request}: LoaderArgs) => {
const url = new URL(request.url);
return new Response(robotsTxtData({url: url.origin}), {
status: 200,
headers: {
'content-type': 'text/plain',
// Cache for 24 hours. <--------------------------
'cache-control': `max-age=${60 * 60 * 24}`,
},
});
}; |
Beta Was this translation helpful? Give feedback.
2 replies
-
The solution was to simply was to switch the Hydrogen site. Oxygen by default prevents crawling of the site when in test mode. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello, I have this robots.txt that I haven't modified from the current version of hydrogen storefront codebase in my robots.txt.
When I run on localhost and go to localhost:3000/robots.txt I can see the file properly.
However, when I go to the site's Hydrogen storefront URL, I'm getting this.
I'm quite new to SEO and this is an important functionality of our site, so I wanted to check how we could fix this?
Beta Was this translation helpful? Give feedback.
All reactions