Pages are not being generated when using sources with github driver

[BernardoSM]

Environment

• Operating System: Linux
• Node Version: v16.20.0
• Nuxt Version: 3.5.3
• Nitro Version: 2.5.2
• Package Manager: yarn@1.22.19
• Builder: vite
• User Config: modules, content
• Runtime Modules: @nuxt/content@2.6.0
• Build Modules: -

Reproduction

https://stackblitz.com/edit/github-kwfyia-plfaaa?file=pages%2Findex.vue

Describe the bug

Only the first root folder is generating all nodes when using nuxt generate and nuxt content with github driver.

This is my nuxt content config within nuxt.config.ts:

...
content: {
    highlight: {
      theme: "github-light",
    },
    sources: {
      github: {
        driver: "github",
        repo: "menthorlabs/courses",
        prefix: "/",
      },
    },
    navigation: {
      fields: ["image", "description", "_id", "_dir", "areas"],
    },
  },
...

This is the content cached by github driver:

These are the pages generated after yarn generate (you can notice that is missing two pages within dist/c/ folder):

I tested a lot of things, from changing folder/file names to changing .md and .yml files content and I realized only the first folder, in my case dist/a generate pages from all nodes.

Additional context

No response

Logs

No response

[Barbapapazes]

Hello,

This behavior is normal and not related to nuxt/content but to how the Nitro crawler works.

In fact, Nitro, the server engine under Nuxt, can only generate known pages from a user point of view. By default, the crawler goes to /, fetch the HTML, get every href, goes to every href founded and start again until every known links have been visited (like a Google Bot).

You can manually add routes into the nuxt.config.ts or do it programmatically using a local Nuxt module.

In this reproduction, I added a link to a/introcducao/.... If you run npm run generate, you will see that Nitro prerender this link and still not /c/.... https://stackblitz.com/edit/github-kwfyia-fpy4ry?file=pages%2Findex.vue

Read more at https://nitro.unjs.io/config#prerender.

@danielroe, I see many people struggling with the prerender mode and how the Nitro crawler works. Do you think we could update the content, to make it sharper, in this section, https://nuxt.com/docs/getting-started/deployment#crawl-based-pre-rendering?

[danielroe]

@Barbapapazes That is a great idea 👍

[BernardoSM]

Thanks for the answer and thanks for updating the documentation. I could make it work enabling crawlLinks and adding anchor tags for the pages to be crawled.