Replies: 1 comment 1 reply
-
I've been unable to reproduce this on https://hydrogen.shop, which is also hosted on Oxygen. Would you mind sharing the domain/url that is getting rejected? |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi,
We are having issues with Bing search engine not being able to crawl our Hydrogen store. Google seems to work just fine and I am unsure what it could be that we are missing.
I am assuming that it could be Oxygen since its an 403 error and within the error referring to Cloudflare.
This is the error response that we got back from Bing webmaster's URL inspection
This is our robots.txt file
`User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /carts
Disallow: /account
Sitemap: https://www.trueleafmarket.com/sitemap.xml
<remove the # for the sake of this post>Google adsbot ignores robots.txt unless specifically named!
User-agent: adsbot-google
Disallow: /checkouts/
Disallow: /checkout
Disallow: /carts
Disallow: /orders
User-agent: Pinterest
Crawl-delay: 1`
Thanks everyone
Beta Was this translation helpful? Give feedback.
All reactions