fix: parsing robots config erroneously returns indexable as false

[silverbackdan]

🐛 The bug

The debug process was found as this issue affects my sitemap.

nuxt-modules/sitemap#466

🛠️ To reproduce

https://www.cymrukitchens.com/sitemap_index.xml

🌈 Expected behavior

After deploying the sitemap works fine. After some time (and the time varies) the check on the robots configuration for all pages changes.

You'll see in the debug that this is the block which is triggered:

https://github.com/nuxt-modules/robots/blob/main/src/runtime/server/composables/getPathRobotConfig.ts#L51-L59

So it is reading the computed robots.txt which is pretty simple
https://www.cymrukitchens.com/robots.txt

# START nuxt-robots (indexable)
User-agent: *
Disallow: /*?s=*

# Block all from operational endpoints
User-agent: *
Disallow: /_cwa/*

# Block non helpful bots
User-agent: Nuclei
User-agent: WikiDo
User-agent: Riddler
User-agent: PetalBot
User-agent: Zoominfobot
User-agent: Go-http-client
User-agent: Node/simplecrawler
User-agent: CazoodleBot
User-agent: dotbot/1.0
User-agent: Gigabot
User-agent: Barkrowler
User-agent: BLEXBot
User-agent: magpie-crawler
Disallow: /

Sitemap: https://www.cymrukitchens.com/sitemap_index.xml
# END nuxt-robots

ℹ️ Additional context

I think it must be the robots section that blocks non-helpful bots that triggers this? I'm going through this repo now and seeing if I can reproduce locally instead of waiting a while every time it deploys to figure it out.

In my module I do configure the robots.txt file in a couple of places, here mainly on the robots:config hook.

https://github.com/components-web-app/cwa-nuxt-module/blob/main/src/runtime/server/server-plugin.ts

[silverbackdan]

It looks like perhaps the parsed robots.txt which began life in public/_robots.txt with

User-agent: *
# block search results
Disallow: /*?s=*

Could very well be the cause, why it changes after it has been deployed for a while I'm not sure yet. But I see the function getPathRobotConfig seems quite untested, yet sitemap module really relies on it.

Perhaps we can add some tests for it? I started but I'm a little tired and can't see how to mock the event reliably with a context that makes sense. I'm not sure if the bots property is going to be set wrong...

The debug actually looks to parse the disallow from the other user-agents though and at some point perhaps that's a change that the user agent internally used by Nitro is matching one of these... matching against line disallow: / suggests this part is triggered for this project...

Im deploying my application without this robots.txt wildcard and will report back.

Upon further testing and reducing my robots.txt file to this:

# START nuxt-robots (indexable)
User-agent: *
Disallow: 

# Block all from operational endpoints
User-agent: *
Disallow: /_cwa/*

Sitemap: https://localhost/sitemap_index.xml
# DEVELOPMENT HINTS:
# - Indexing is enabled from anonymous.

# END nuxt-robots

The sitemap emptying still persists.

[silverbackdan]

So far I've disabled this function from my forked sitemap repo and my sitemap appears to remain up and running.

this line here seems to check if a matched group has a property _indexable
https://github.com/nuxt-modules/robots/blob/main/src/runtime/server/composables/getPathRobotConfig.ts#L51

You'll see here that I am adding groups in from my hook:
https://github.com/components-web-app/cwa-nuxt-module/blob/dev/src/runtime/server/server-plugin.ts#L36-L59

I do not specify the _indexable property at all.

This check only checks the user agent, and not the allow or disallow paths match. therefore I have the rule added

    ctx.groups.push({
      userAgent: ['*'],
      comment: ['Block all from operational endpoints'],
      allow: [],
      disallow: ['/_cwa/*'],
    })

And I think this is being picked up for all the URLs ??

I don't think the other rules would be picked up.. or they shouldn't be I don't think as I've actually disabled them.

My next step is to debug the actual groups found in a forked version of this robots repo. And log at the point where this response would be made to say that every URL is not indexable.

If a test is started for this composable, I could easily add in my groups and checks if it is functioning properly with the rules given.

Why it works for a while and then doesn't ... I'm not sure... One theory is that the Nitro User Agent changes or disappears, or there is some inconsistency with a promise not being awaited somewhere in the code.

Like the sitemap repo, the types seem to be unresolved throughout. I've got a PR open on sitemaps to get the type checks passing now.

I'll report back with a definitive group that is being found and triggered to show all my pages as indexable: false

[silverbackdan]

Got it.
Yes the rule I add above is caught because _indexable ends up not defined.

normalizeGroup is a function here https://github.com/nuxt-modules/robots/blob/main/src/util.ts#L265-L281

This is only called in the module setup. modules:done hook here https://github.com/nuxt-modules/robots/blob/main/src/module.ts#L417

And here https://github.com/nuxt-modules/robots/blob/main/src/module.ts#L489

On line 439 here you call the hook robots:config And here https://github.com/nuxt-modules/robots/blob/main/src/module.ts#L439

This is where I set my group configs. Something must happen between the hook and line 489 where normalise is called ...

_indexable is definitely undefined when I need it to be when the sitemap calls it...

thoughts?

For now I could call the normalize function manually?

[silverbackdan]

I think this is it:
https://github.com/nuxt-modules/robots/blob/main/src/runtime/server/util.ts

The robots:config hook is called again here. There is no normalization being called after this one. So when it is called, my hook will push into the groups a new group. This group is not normalized as far as I can tell.

Thoughts for fixes welcome from you @harlan-zw - at least I have found out what it is in my implementation that is causing this.

Basically the existence of a _robots.txt in the public directory I guess will mean there is this additional call to this hook and why it only happens in 1 of my projects.

TIA for your attention to it and happy to think up ideas and be corrected if I should be extending configurations on different hooks.

[harlan-zw]

Hey @silverbackdan, I appreciate your thorough investigation into this.

So is the fix we just need to renormalize after the hook call in your opinion?

[silverbackdan]

No problem, it was a really tough one to track down. I think so, but I guess the question is making sure we don't waste time double-normalizing.

My questions would be, is there a purpose to call this hook again in this util, and is it the same use case as the other times the hook is called when the module is ready?

You'll know the internals more of why these properties are set, but is the _indexable property crucial to be set, or in the getPathRobotConfig composable, if _indexable is undefined, should it be treating it as false? Should this conditional be checking for an explicitly false value on indexable?

Once this is decided, is there a way we can implement tests for this composable? It isn't used at all within this bundle it seems and is more of an API functionality for other SEO services to call and check metadata - is this just for the sitemap module, and what is good to be checking to make sure it behaves as desired, and should we also be checking that these additional metadata properties like _indexable are always being set?

Perhaps you'll be in a better place to answer al of these, but certainly the big issue right now is when I set my groups from this hook, the _indexable property can be wiped at what appears to be a random time to a user - I'm not even sure when this other composable that wipes the property is called, but once it's cleared the _indexable property, it never comes back so the module seems to persist this additional data.

Happy to discuss - and thanks for taking the time to read what's been going on and helping work on finding a solution.

[harlan-zw]

I'll create some tests to replicate the issue and I should get a better understanding from there. Thanks for the details.

[silverbackdan]

Appreciate it! Happy to report that since I used my forked branch to skip that function call though, the site map remains live still. Happy to have found this cause. Also wondering if there's a way to add debugging to output the filtered URLs somewhere in dev, perhaps with that check failed.
As it's quite a complicated checking process, someone may end up just confused why their page isn't showing in a sitemap and that sort of debugging functionality could help them fix configs.

In my application, who knows but perhaps I could build in something so I can test URLs that a user has made and flag to them if it isn't indexable, or will be excluded from a sitemap ... that'd be VERY cool. But perhaps not as easy to do, and definitely not a priority right now compared to this bug :P