Add robots configuration to SEO meta tags

[cartogram]

WHY are these changes introduced?

This PR allows developers to set values in the SEO config to generate a robots meta tag in the Head of the document.

WHAT is this pull request doing?

This PR adds a robots option that allows users granular control over the robots meta tag. Like all SEO config values, this can be set on both a global and per-page basis using the handle.seo property.

export handle = {
  seo: {
    robots: {
      noIndex: false,
      noFollow: false,
    }
  }
}

HOW to test your changes?

Run the demo store and look for the robots meta tag. You can try different values and check the various results.

Post-merge steps

Checklist

• I've read the Contributing Guidelines
• I've considered possible cross-platform impacts (Mac, Linux, Windows)
• I've added a changeset if this PR contains user-facing or noteworthy changes
• I've added tests to cover my changes

[juanpprieto]

Is it worth adding a test for a page-level robots overwriting global robots?

[cartogram]

We'd need to add a lot of utilities for more e2e style testing like this... And wouldn't be a test for this abstraction, but actually the part that touches Remix/React (aka the <Seo /> component).

You are right we should have these, but its a bigger deal and doesn't make sense in this PR.

[juanpprieto]

I think we should create an issue or a discussion about zod to get it going.

I think there's enough use cases for it (analytics, seo, cart) etc

[cartogram]

I agree its worth experimenting with as it is big part of astro, and even other framework's use it for this exact purpose (like Nuxt's Head abstraction).

@frehner and I talked very briefly about it. In a previous iteration of this generate-seo-tags I used it (but it was still very rushed and did more harm than good at the time).

@juanpprieto do you want to start that discussion?

[juanpprieto]

could do [].filter(Boolean) to get rid of undefined values

[juanpprieto]

Looks great @cartogram 🥇

[lordofthecactus]

Looks good ✅

[cartogram]

@pshapiro would love your 👀 on this one to determine if this is the right amount of flexibility for merchants.

[pshapiro]

The implementation looks good to me!

[svrnhdl]

Hi all,

I was looking into this merged PR, because I'm having trouble getting my twitter cards to display on our webshop. I believe the issue is that there is no twitter-image meta tag created. All the pages have the og:image tags, but it seems like twitter needs a specific one to display.

[frehner]

@svrnhdl Please follow this guide https://shopify.dev/docs/custom-storefronts/hydrogen/seo/manage-seo#step-3-override-seo-values-on-the-product-details-page and open a new discussion or issue if you have problems. Thanks!