Use of useSearchParams is causing the addition of meta robots with noindex in the production build

[FrancisGregori]

Link to the code that reproduces this issue

https://github.com/FrancisGregori/nextjs-noindex-issue

To Reproduce

1. Use Next.js version 13.5.6-canary.3 or higher.
2. Implement the useSearchParams hook in any page.
3. Build the application for production.
4. Notice that the tag is automatically added to the generated HTML, even with the correct robots configuration in place.

Current vs. Expected behavior

Current Behavior:
When using useSearchParams in Next.js version 13 or higher, the <meta name="robots" content="noindex"/> tag is automatically added to pages in the production build. This occurs even if the robots meta tag is already configured manually, leading to erroneous addition and potential duplication of the robots meta tag.

Expected Behavior:
The <meta name="robots" content="noindex"/> tag should only be added in preview versions, not affecting the production build. Manual configurations of the robots meta tags should be respected to avoid duplication and ensure correct indexing of pages by search engines in production.

Verify canary release

• I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
  Platform: darwin
  Arch: arm64
  Version: Darwin Kernel Version 23.1.0: Mon Oct  9 21:27:24 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T6000
Binaries:
  Node: 18.17.0
  npm: 9.6.7
  Yarn: 1.22.19
  pnpm: N/A
Relevant Packages:
  next: 14.0.4-canary.2
  eslint-config-next: 14.0.1
  react: 18.2.0
  react-dom: 18.2.0
  typescript: 5.2.2
Next.js Config:
  output: N/A

Which area(s) are affected? (Select all that apply)

Metadata (metadata, generateMetadata, next/head), Routing (next/router, next/navigation, next/link)

Additional context

I have encountered this problem both locally and when deploying on Vercel. Upon further investigation, it seems that this issue started with version 13.5.6-canary.3. This observation leads me to believe that the erroneous addition of the <meta name="robots" content="noindex"/> tag in the production build when using useSearchParams might be specific to this and subsequent versions. Prior versions, including 13.5.6-canary.2 and earlier, do not exhibit this issue.

_NEXT-1853

[iskipu]

After building when i go to .next/server/app/index.html

i see that your custom configurations are not reflected (just like you mentioned).

But however when you start your server (pnpm start) and visit the page in your browser and inspect it

you can see that the meta robot tag is same as what you configured.

So i am assuming that while we try to access the page the server automatically changes it dynamically. so i think this shouldnt be a problem.

Plz correct me if i am wrong.

[FrancisGregori]

@iskipu, if you use the element inspector to search for the robots meta tag, only the one I configured will appear. However, if you check the page source, you'll see that it's still there.

With this, even though it's dynamically removed, it will always appear in the base HTML. And because of this, search engines might end up interpreting the noindex, since it is found in the HTML.

To illustrate this better, I deployed this repository on Vercel. But just like you said, using the element inspector removes the noindex, but in the HTML, it's still there.

Here's the link in case you want to check it out.
https://nextjs-noindex-issue.vercel.app/

Well, I believe that html being built with noindex is a problem. That's why I opened this issue.

[iskipu]

@FrancisGregori Afaik crawlers used by browsers use the generated DOM for SEO rather than the base HTML.

Use any of the SEO tools for meta tag analysis i am using this (https://smallseotools.com/meta-tags-analyzer/) and then paste your url (https://nextjs-noindex-issue.vercel.app/) and check the report generated and you can see that the robot meta tag is (index, follow).

So it might not be big issue but since it is reflecting differently in base html it is still a small issue like you said.

[marduc812]

I have the same issue on next.js 14.0.3. Probably the tag was added to avoid indexing of user data, but on the other hand, it prevents the indexing from Google for the whole page. Not a good way to handle it IMO. I am also looking for a solution.

[FrancisGregori]

@marduc812 , as a workaround I ended up removing the use of useSearchParams and using window.location.search with URLSearchParams.

  const searchParams = new URLSearchParams(
    typeof window !== 'undefined' ? window.location.search : '',
  );

Maybe this will help you temporarily.

[FrancisGregori]

@FrancisGregori Afaik crawlers used by browsers use the generated DOM for SEO rather than the base HTML.

Use any of the SEO tools for meta tag analysis i am using this (https://smallseotools.com/meta-tags-analyzer/) and then paste your url (https://nextjs-noindex-issue.vercel.app/) and check the report generated and you can see that the robot meta tag is (index, follow).

So it might not be big issue but since it is reflecting differently in base html it is still a small issue like you said.

@iskipu, you're right that modern search engine crawlers, like Googlebot, are capable of processing JavaScript and can analyze the generated DOM for SEO purposes. This means they can see the page as a user would after JavaScript execution, including any changes to meta tags.

However, as you pointed out, there's still a discrepancy between the base HTML and the rendered version, which can be a concern. Although advanced crawlers can interpret JavaScript, not all search engines have this capability. Plus, relying solely on JavaScript for important SEO elements can lead to unpredictability, especially if a crawler encounters issues executing the JavaScript. It's a smaller issue, but for optimal SEO, consistency between the initial HTML and the post-JavaScript-rendered HTML is recommended.

[iskipu]

@FrancisGregori Understood thanks for the clear explanation 😄

[sudoaugustin]

I faced a similar problem today after upgrading to NEXTJS 14. I had to switch to useParams(). The problem is only visible when we View Page Source. We use NEXTJS because we don't wanna look into and handle SEO in detail.

[FrancisGregori]

Indeed, the use of useParams is a good alternative @sudoaugustin. Unfortunately, in my case, I couldn't change the final URL, so I needed to continue using query params.

[ZakharPilipenko]

I had to use useSearchParams in my project as well, had the same issue of duplicated meta tag robots with noindex value, but even removing useSearchParams from a project and deleting .next folder to rebuild the app again didn't solve the issue.

[ZakharPilipenko]

I had to use useSearchParams in my project as well, had the same issue of duplicated meta tag robots with noindex value, but even removing useSearchParams from a project and deleting .next folder to rebuild the app again didn't solve the issue.

My bad, I didn't double check if I removed useSearchParams completly from my app, I found one more place where I used it, after removing it the issue with duplicated metatag gone.
This temporary solution

  const searchParams = new URLSearchParams(
    typeof window !== 'undefined' ? window.location.search : '',
  );

Is indeed good.

[jo-sip]

Took me a while to track down how this was being added to my build. Thanks @FrancisGregori for the clear workaround. My index page noindex was driving me nuts.

[FrancisGregori]

Took me a while to track down how this was being added to my build. Thanks @FrancisGregori for the clear workaround. My index page noindex was driving me nuts.

I know the feeling @jo-sip, it also took me a while to figure out what the problem was. And I'm glad the workaround helped you.

[pier7529]

"next": "14.0.3"
Add same issue with my site since few weeks. Some pages of https://wildescapesoverland.com/fr/trip/france-alpes-gr738 (view-source) has these wrong value and cannot be indexed by Googlebot :-(

Has removed useSearchParams in one typescript BUT not enough to fix the issue.
Still looking for the root cause and a fix.

[FrancisGregori]

@pier7529, I believe you haven't set up the robots meta tag, right? I saw that the robots.txt file seems correct, but usually, when we have this error due to the use of useSearchParams with the robots meta tag configured, the robots meta tag gets duplicated in the code.

So, I would suggest you configure the robots meta in your layout.tsx

import type { Metadata } from 'next'

export const metadata: Metadata = {
  robots: {
    index: true,
    follow: true,
  },
}

Here's a very simplified example, but you can check out other configurations here if you think it's necessary.

Besides that, I would also suggest doing a double check to ensure that useSearchParams is not being used or imported anywhere in your code.

[pier7529]

@FrancisGregori I've tried this :

robots: {
      index: true,
      follow: true,
      nocache: true,
      googleBot: {
        index: true,
        follow: true,
        noimageindex: true,
        'max-video-preview': -1,
        'max-image-preview': 'large',
        'max-snippet': -1,
      },
    },

In Navigator Inspector, I can see the correct values :

 <meta name="robots" content="index, follow, nocache">
<meta name="googlebot" content="index, follow, noimageindex, max-video-preview:-1, max-image-preview:large, max-snippet:-1">

BUT when I display the code source of the same page, I got this (in "npm run dev" mode) :

<!DOCTYPE html><html id="__next_error__">...
</script><meta name="robots" content="noindex"/>
<meta name="next-error" content="not-found"/>....

So don't understand.

[prettydev]

Same issue.
automatically added.

[haginus]

Setting the dynamic route config option to force-dynamic on the pages where you use useSearchParams() will prevent Nest.js from adding the <meta name="robots" content="noindex"/>.
Basically, the page will load dynamically for each request and the query params will be accessible to the server.

[FrancisGregori]

Setting the dynamic route config option to force-dynamic on the pages where you use useSearchParams() will prevent Nest.js from adding the <meta name="robots" content="noindex"/>. Basically, the page will load dynamically for each request and the query params will be accessible to the server.

@haginus, using force-dynamic in Next.js for server access to query parameters on pages with useSearchParams does have a few notable trade-offs:

Increased Server Load: It leads to server-side rendering on every request, which can gently heighten server workload and extend response times.

Caching Trade-offs: This approach bypasses some of Next.js's caching advantages, which might slightly impact site performance.

SEO Considerations: While it avoids adding a noindex meta tag, there's a subtle possibility that slower load times could affect SEO, but this can be managed.

In essence, force-dynamic is a helpful tool for a specific need, but it's good to weigh its impacts on performance and SEO against your site's overall goals.

[pier7529]

In my context, I think I solved my issue with meta no index or multiple meta robots
by moving my internal APIs to directory app\api
and generate static my pages (generateStaticParams)
I still have some few things / components to adapt (Leaflet Map) but my website is fully visible for googlebot as before with Next.js 13.x

[atrhacker]

For the last 5 days I have wondered why this meta was added while everything else was working fine. This is really concerning, especially for a framework that is supposed to be SEO-friendly. Even if it's minor, it has an impact. Don't hesitate to let us know when this is fixed. In my case, I need to use the useSearchParams in some components. Thanks @FrancisGregori for raising this issue and making me feel less alone. I'm surprised that so few people are concerned

[leerob]

Hey folks, I was not able to reproduce. I took the repo from OP, updated to the latest canary, and I do not see the issue. Is there a newer reproduction?

[github-actions]

We cannot recreate the issue with the provided information. Please add a reproduction in order for us to be able to investigate.

Why was this issue marked with the please add a complete reproduction label?

To be able to investigate, we need access to a reproduction to identify what triggered the issue. We prefer a link to a public GitHub repository (template for App Router, template for Pages Router), but you can also use these templates: CodeSandbox: App Router or CodeSandbox: Pages Router.

To make sure the issue is resolved as quickly as possible, please make sure that the reproduction is as minimal as possible. This means that you should remove unnecessary code, files, and dependencies that do not contribute to the issue. Ensure your reproduction does not depend on secrets, 3rd party registries, private dependencies, or any other data that cannot be made public. Avoid a reproduction including a whole monorepo (unless relevant to the issue). The easier it is to reproduce the issue, the quicker we can help.

Please test your reproduction against the latest version of Next.js (next@canary) to make sure your issue has not already been fixed.

If you cannot create a clean reproduction, another way you can help the maintainers' job is to pinpoint the canary version of next that introduced the issue. Check out our releases, and try to find the first canary release that introduced the issue. This will help us narrow down the scope of the issue, and possibly point to the PR/code change that introduced it. You can install a specific version of next by running npm install next@<version>.

I added a link, why was it still marked?

Ensure the link is pointing to a codebase that is accessible (e.g. not a private repository). "example.com", "n/a", "will add later", etc. are not acceptable links -- we need to see a public codebase. See the above section for accepted links.

What happens if I don't provide a sufficient minimal reproduction?

Issues with the please add a complete reproduction label that receives no meaningful activity (e.g. new comments with a reproduction link) are automatically closed and locked after 30 days.

If your issue has not been resolved in that time and it has been closed/locked, please open a new issue with the required reproduction.

I did not open this issue, but it is relevant to me, what can I do to help?

Anyone experiencing the same issue is welcome to provide a minimal reproduction following the above steps. Furthermore, you can upvote the issue using the 👍 reaction on the topmost comment (please do not comment "I have the same issue" without reproduction steps). Then, we can sort issues by votes to prioritize.

I think my reproduction is good enough, why aren't you looking into it quicker?

We look into every Next.js issue and constantly monitor open issues for new comments.

However, sometimes we might miss one or two due to the popularity/high traffic of the repository. We apologize, and kindly ask you to refrain from tagging core maintainers, as that will usually not result in increased priority.

Upvoting issues to show your interest will help us prioritize and address them as quickly as possible. That said, every issue is important to us, and if an issue gets closed by accident, we encourage you to open a new one linking to the old issue and we will look into it.

Useful Resources

• How to Contribute to Open Source (Next.js)
• How to create a Minimal, Complete, and Verifiable example
• Reporting a Next.js bug
• Next.js Triaging issues

[FrancisGregori]

Hey folks, I was not able to reproduce. I took the repo from OP, updated to the latest canary, and I do not see the issue. Is there a newer reproduction?

Hey @leerob, upon inspecting through the elements panel, everything seems to be in order. However, in this instance, we are examining the DOM which has been modified by JavaScript.

If you take a look at the source code directly(view-source:https://nextjs-noindex-issue.vercel.app/, for example) you'll notice that the robots meta tag is duplicated. The first one is a robots=noindex meta tag, and the second is the one I configured in my export const robots = {...}.

[leerob]

Thank you! We have a PR going #59531.

[FrancisGregori]

Awesome @leerob! Thanks for the heads up. Much appreciated!

[lukebelbina]

Any ETA on when the PR will make it to main, I am running into this issue and haven't been able to resolve it with the items suggested on this thread

[github-actions]

This closed issue has been automatically locked because it had no new activity for 2 weeks. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.