Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: enable option to propagate arbitrary frontmatter data in globals #4492

Open
kaytwo opened this issue Mar 23, 2021 · 14 comments 路 Fixed by #4495
Open

Proposal: enable option to propagate arbitrary frontmatter data in globals #4492

kaytwo opened this issue Mar 23, 2021 · 14 comments 路 Fixed by #4495
Labels
domain: content plugin Related to content plugin emitting metadata for theme consumption feature This is not a bug or issue with Docusausus, per se. It is a feature request for the future.

Comments

@kaytwo
Copy link
Contributor

kaytwo commented Mar 23, 2021

馃殌 Feature

Expose markdown frontmatter in page globals.

Have you read the Contributing Guidelines on issues?

Yes

Motivation

I'd like to programmatically generate docs/pages based on custom metadata stored in the frontmatter of my docs. In my use case, I'm building a course website, where the docs are the descriptions of homeworks, labs, exams, etc, and I'd like to create a "calendar" page that reads release and due date metadata from the frontmatter of each of those pages. As of right now, only id, path, and sidebar are set in the plugin globals.

Pitch

While this is a very specific use case, I've typically seen frontmatter used in this way in other static site generators to create derivative pages that incorporate some or all of the important metadata from some subset of other pages.

I tried to implement something similar to this both using this approach as well as a standalone plugin that I wrote for this purpose (trying both setGlobalData and createData), but the issue is that the functionality is too tightly tied to the docs plugin itself to be easily factored out (and I couldn't figure out how to swizzle this part of @docusaurus/plugin-content-docs).

Implementation

The frontmatter can be propagated with this commit: kaytwo@ff408e1. I understand that this will increase the size of the globals which should be avoided, however this option could probably be gated behind a config option by replacing this line in docs.ts with:

    frontMatter : options.includeFrontmatter ? frontMatter : undefined
@kaytwo kaytwo added feature This is not a bug or issue with Docusausus, per se. It is a feature request for the future. status: needs triage This issue has not been triaged by maintainers labels Mar 23, 2021
@slorber
Copy link
Collaborator

slorber commented Mar 23, 2021

Hi,

I understand your motivation to create additional pages, but I don't think adding this data to the global site data is the best option here. As you noted it's going to increase the size of the whole site, including the homepage etc.

What I would suggest instead is to look at extending the original docs plugin to create those additional pages.

We don't have an official doc for this but you can take a look at: #4138 (comment)

You could probably do what you want with something like:

const docsPluginExports = require("@docusaurus/plugin-content-docs");

const docsPlugin = docsPluginExports.default;

function docsPluginEnhanced(...pluginArgs) {
  const docsPluginInstance = docsPlugin(...pluginArgs);

  return {
    ...docsPluginInstance,

    async contentLoaded(...contentLoadedArgs) {

      // Create default plugin pages
      await docsPluginInstance.contentLoaded(...contentLoadedArgs);

      // Create your additional pages
      const { actions, content } = contentLoadedArgs[0];
      const { addRoute } = actions;
      const { loadedVersions } = content;

      await Promise.all(
        loadedVersions.map(async version => {
          addRoute({
            path: loadedVersion.versionPath + "/calendar",
            exact: true,
            component: "@theme/DocsCalendarPage", // Your component
            modules: {
              // ... The props DocsCalendarPage need to receive
              calendar: await createCalendarProp(version)
            }
          });
        })
      );
    }

  };

}

module.exports = {
  ...docsPluginExports,
  default: docsPluginEnhanced
};

Does it make sense to you? Normally you'll have all the data you need from the "version" item to create the prop your custom calendar page should receive.

Only this custom calendar page would receive the data, and the homepage will not be burdened by useless frontmatter weight

@slorber slorber removed the status: needs triage This issue has not been triaged by maintainers label Mar 23, 2021
@kaytwo
Copy link
Contributor Author

kaytwo commented Mar 23, 2021

This does make sense, thanks! I built the most basic version as I could (using a super basic test component in my site at @site/src/components/calendar.jsx, I added the new plugin to both the docusaurus config and package.json, but no luck.

Here's the error:
./.docusaurus/registry.js
Module not found: Can't resolve 'contentPath' in '/home/ckanich/repos/cs361-website-next/.docusaurus'./.docusaurus/registry.js
Module not found: Can't resolve 'contentPathLocalized' in '/home/ckanich/repos/cs361-website-next/.docusaurus'./.docusaurus/registry.js
Module not found: Can't resolve 'docs' in '/home/ckanich/repos/cs361-website-next/.docusaurus'./.docusaurus/registry.js
Module not found: Can't resolve 'isLast' in '/home/ckanich/repos/cs361-website-next/.docusaurus'./.docusaurus/registry.js
Module not found: Can't resolve 'mainDocId' in '/home/ckanich/repos/cs361-website-next/.docusaurus'./.docusaurus/registry.js
Module not found: Can't resolve 'permalinkToSidebar' in '/home/ckanich/repos/cs361-website-next/.docusaurus'./.docusaurus/registry.js
Module not found: Can't resolve 'routePriority' in '/home/ckanich/repos/cs361-website-next/.docusaurus'./.docusaurus/registry.js
Module not found: Can't resolve 'sidebarFilePath' in '/home/ckanich/repos/cs361-website-next/.docusaurus'./.docusaurus/registry.js
Module not found: Can't resolve 'sidebars' in '/home/ckanich/repos/cs361-website-next/.docusaurus'./.docusaurus/registry.js
Module not found: Can't resolve 'versionEditUrl' in '/home/ckanich/repos/cs361-website-next/.docusaurus'./.docusaurus/registry.js
Module not found: Can't resolve 'versionEditUrlLocalized' in '/home/ckanich/repos/cs361-website-next/.docusaurus'./.docusaurus/registry.js
Module not found: Can't resolve 'versionLabel' in '/home/ckanich/repos/cs361-website-next/.docusaurus'./.docusaurus/registry.js
Module not found: Can't resolve 'versionName' in '/home/ckanich/repos/cs361-website-next/.docusaurus'./.docusaurus/registry.js
Module not found: Can't resolve 'versionPath' in '/home/ckanich/repos/cs361-website-next/.docusaurus'
the new plugin source:
const docsPluginExports = require("@docusaurus/plugin-content-docs");

const docsPlugin = docsPluginExports.default;

function docsPluginEnhanced(...pluginArgs) {
  const docsPluginInstance = docsPlugin(...pluginArgs);

  return {
    ...docsPluginInstance,

    async contentLoaded(...contentLoadedArgs) {

      // Create default plugin pages
      await docsPluginInstance.contentLoaded(...contentLoadedArgs);

      // Create your additional pages
      const { actions, content } = contentLoadedArgs[0];
      const { addRoute } = actions;
      const { loadedVersions } = content;
      const createCalendarProp = async (version) => {
        return Object.keys(version)
      }

      await Promise.all(
        loadedVersions.map(async version => {
          addRoute({
            path: version.versionPath + "/calendar",
            exact: true,
            component: "@site/src/components/calendar", // Your component
            modules: {
              // ... The props DocsCalendarPage need to receive
              calendar: await createCalendarProp(version)
            }
          });
        })
      );
    }
  };
}

module.exports = {
  ...docsPluginExports,
  default: docsPluginEnhanced
};
package.json:
{
  "name": "cs-361",
  "version": "0.0.0",
  "private": true,
  "scripts": {
    "docusaurus": "docusaurus",
    "start": "docusaurus start",
    "build": "docusaurus build",
    "swizzle": "docusaurus swizzle",
    "deploy": "docusaurus deploy",
    "clear": "docusaurus clear",
    "serve": "docusaurus serve",
    "write-translations": "docusaurus write-translations",
    "write-heading-ids": "docusaurus write-heading-ids"
  },
  "dependencies": {
    "@docusaurus/core": "2.0.0-alpha.72",
    "@docusaurus/preset-classic": "2.0.0-alpha.72",
    "@docusaurus/theme-classic": "^2.0.0-alpha.72",
    "@mdx-js/react": "^1.6.21",
    "clsx": "^1.1.1",
    "docusaurus-plugin-auto-sidebars": "^1.0.7",
    "docusaurus-plugin-content-docs-index": "/home/ckanich/repos/docusaurus-plugin-content-docs-index",
    "react": "^17.0.1",
    "react-dom": "^17.0.1"
  },
  "browserslist": {
    "production": [
      ">0.5%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  },
  "devDependencies": {
    "@docusaurus/module-type-aliases": "^2.0.0-alpha.72",
    "@tsconfig/docusaurus": "^1.0.2",
    "@types/react": "^17.0.3",
    "@types/react-helmet": "^6.1.0",
    "@types/react-router-dom": "^5.1.7",
    "typescript": "^4.2.3"
  }
}
docusaurus.config.js:
const path = require("path");

/** @type {import('@docusaurus/types').DocusaurusConfig} */
module.exports = {
  title: "CS 361: Systems Programming",
  tagline: "Getting to know your operating system",
  url: "https://www.cs.uic.edu/~ckanich/cs361/next/",
  baseUrl: "/",
  onBrokenLinks: "throw",
  onBrokenMarkdownLinks: "warn",
  favicon: "img/uic.svg",
  organizationName: "csatuic", // Usually your GitHub org/user name.
  projectName: "cs361-website", // Usually your repo name.
  themeConfig: {
    navbar: {
      title: "CS 361: Systems Programming",
      logo: {
        alt: "UIC Logo",
        src: "img/uic.svg",
      },
      items: [
        {
          to: "syllabus/",
          label: "Syllabus",
          position: "left",
        },
        {
          href: "https://github.com/csatuic/cs361-website/",
          label: "GitHub",
          position: "right",
        },
      ],
    },
    footer: {
      style: "dark",
      links: [
        {
          title: "Docs",
          items: [
            { label: "Home", to: "/" },
            { label: "Schedule", to: "/schedule" },
            { label: "Syllabus", to: "/syllabus" },
          ],
        },
        {
          title: "Community",
          items: [
            {
              label: "Discourse Q&A",
              href: "https://example.com",
            },
            {
              label: "Discord",
              href: "https://example.com/invite/docusaurus",
            },
          ],
        },
        {
          title: "More",
          items: [
            {
              label: "Chris Kanich",
              to: "https://www.cs.uic.edu/~ckanich/",
            },
            {
              label: "GitHub",
              href: "https://github.com/csatuic/",
            },
          ],
        },
      ],
      copyright: `Copyright 漏 ${new Date().getFullYear()} Chris Kanich. Built with Docusaurus.`,
    },
  },
  plugins: [
    [
      path.resolve(
        __dirname,
        "/home/ckanich/repos/docusaurus-plugin-content-docs-index"
      ),
      {
        routeBasePath: "/",
        sidebarPath: require.resolve("./sidebars.auto.js"),
        // Please change this to your repo.
        editUrl: "https://github.com/kaytwo/cs361-website-next/edit/main/"
      },
    ],
    "docusaurus-plugin-auto-sidebars",
  ],
  themes: [
    [
      "@docusaurus/theme-classic",
      {
        customCss: require.resolve("./src/css/custom.css"),
      },
    ],
  ],
};

Any help would be greatly appreciated, if I can figure this out it should be pretty easy to publish this plugin for others to fork to provide a similar enhancement.

@slorber
Copy link
Collaborator

slorber commented Mar 23, 2021

If you have a public github repo with a branch and your code I can see how to fix it.

You can't pass any json to "modules" directly here, but you can pass a path to a son file containing the data you want (we have a createData action for that)

Modules means something Webpack can require.

It's a bit deep in our code but you'll find this link useful:
https://docusaurus.io/docs/next/lifecycle-apis#async-contentloadedcontent-actions

async function contentLoaded({ content, actions }) {
  const { createData, addRoute } = actions;

  // Create friends.json
  const friends = ["Yangshun", "Sebastien"];
  const friendsJsonPath = await createData(
    "friends.json",
    JSON.stringify(friends)
  );

  // Add the '/friends' routes, and ensure it receives the friends props
  addRoute({
    path: "/friends",
    component: "@site/src/components/Friends.js",
    modules: {
      // propName -> JSON file path
      friends: friendsJsonPath
    },
    exact: true
  });
}

@kaytwo
Copy link
Contributor Author

kaytwo commented Mar 24, 2021

Thanks for your help on this, I got it to work - unfortunately the metadata available directly through the docsPluginEnhanced approach will allow direct access to the docusaurus metadata and the filenames which would allow for recovering the frontmatter variables but it would be a pretty roundabout approach, and the new page would still need to be themed to fit in with the rest of the site.

At the end of the day I think a prebuild script that programmatically extracts the frontmatter from chosen docs and creates a json/yaml for the calendar page to ingest will be the best approach to solving this problem. The minimum working example is at https://github.com/kaytwo/docusaurus-plugin-content-docs-index if it might be helpful or if there's an easier way to accomplish this.

@kaytwo
Copy link
Contributor Author

kaytwo commented Mar 24, 2021

Do plugins get processed in the order of the plugin array? Would this type of preprocessing be possible using a plugin? e.g. pass it a glob of files you care about, and then generate the new mdx file, which then gets processed by the docs plugin itself to create the summary page.

@slorber
Copy link
Collaborator

slorber commented Mar 24, 2021

but it would be a pretty roundabout approach

why don't you like this approach?

and the new page would still need to be themed to fit in with the rest of the site.

Yes but in any case if you are creating a page that is not handled directly by the classic theme you would have to style it anyway? You can reuse existing theme components to do so.

At the end of the day I think a prebuild script that programmatically extracts the frontmatter from chosen docs and creates a json/yaml for the calendar page to ingest will be the best approach to solving this problem.

You mean write calendar.json to disc and import it in src/pages/calendar.jsx?

That would definitively work, and some sites do that already (Jest site fetch opencollective contributors to display them on the site)

But why not using the Docusaurus infra instead?
That basically does the same in the end, it's just the json file is at a different location, and we have read the docs files for you, so you don't need to read them twice.

I admit the current API is not ideal, but it is definitively the pattern we want you to encourage to adopt, and we'll improve this API to make this easier.

The minimum working example is at https://github.com/kaytwo/docusaurus-plugin-content-docs-index if it might be helpful or if there's an easier way to accomplish this.

Your example looks good to me (apart the data in events.json)

Do plugins get processed in the order of the plugin array? Would this type of preprocessing be possible using a plugin? e.g. pass it a glob of files you care about, and then generate the new mdx file, which then gets processed by the docs plugin itself to create the summary page.

The plugins load content in parallel for efficiency, and you should avoid relying on the plugins ordering in any way.

We don't want to implement a plugin execution ordering/dependency system but rather keep plugins isolated and let you extend them more easily.

@kaytwo
Copy link
Contributor Author

kaytwo commented Mar 24, 2021

but it would be a pretty roundabout approach

why don't you like this approach?

This may be my mistake - I couldn't find where custom frontmatter was persisted within the existing data structures (only all of the metadata generated by docusaurus), so the solution I could think of would be to:

  1. capture the various @site/docs/homeworks/homework1.mdx etc. source file references in contentLoaded
  2. open those files, read their frontmatter with a yaml/markdown parser
  3. send the relevant information as props to the schedule component.

Please let me know if there's a different way to access that metadata, that would be a great help.

and the new page would still need to be themed to fit in with the rest of the site.

Yes but in any case if you are creating a page that is not handled directly by the classic theme you would have to style it anyway? You can reuse existing theme components to do so.

This may be my own personal use case, but for a smaller docs site I see the hierarchical left nav as more of a global site nav than "read about the docs in this one specific component of a project", so the best case scenario would be for the schedule to coexist within that structure and be easily accessible from the sidebar, and for that same sidebar to show up on the schedule page itself. As I've been navigating the docusaurus docs, I've often found myself having to swap between the "Docs" sidebar and the "API" sidebar by using the top global nav, and it would be nice (and hopefully not overwhelming) for everything to show up in that same hierarchy.

If there is an easy way for me to use the existing theme and the same sidebars (besides just dynamically generating a new schedule.mdx in my docs directory), please let me know.

At the end of the day I think a prebuild script that programmatically extracts the frontmatter from chosen docs and creates a json/yaml for the calendar page to ingest will be the best approach to solving this problem.

You mean write calendar.json to disc and import it in src/pages/calendar.jsx?

That would definitively work, and some sites do that already (Jest site fetch opencollective contributors to display them on the site)

But why not using the Docusaurus infra instead?
That basically does the same in the end, it's just the json file is at a different location, and we have read the docs files for you, so you don't need to read them twice.

This is definitely the crux of it (I dove into this mostly because I didn't want to read the same files twice) - I am not seeing where I can read the arbitrary frontmatter content from the already-read-in files (and I can't figure out how to style the new page as a doc).

I admit the current API is not ideal, but it is definitively the pattern we want you to encourage to adopt, and we'll improve this API to make this easier.

The minimum working example is at https://github.com/kaytwo/docusaurus-plugin-content-docs-index if it might be helpful or if there's an easier way to accomplish this.

Your example looks good to me (apart the data in events.json)

Do plugins get processed in the order of the plugin array? Would this type of preprocessing be possible using a plugin? e.g. pass it a glob of files you care about, and then generate the new mdx file, which then gets processed by the docs plugin itself to create the summary page.

The plugins load content in parallel for efficiency, and you should avoid relying on the plugins ordering in any way.

We don't want to implement a plugin execution ordering/dependency system but rather keep plugins isolated and let you extend them more easily.

This makes sense - right now I'm considering an approach like https://github.com/acrobit/docusaurus-plugin-auto-sidebars/blob/45d6723fd3a0e04641b343cf1478827587c9fd5e/src/index.ts#L42 to generate the schedule.json in a well known location, so that I can import it in a <Schedule> component. I don't think this would cause any dependency issues except perhaps during development, which I can live with.

Thanks again for your help!

@slorber
Copy link
Collaborator

slorber commented Mar 24, 2021

This may be my mistake - I couldn't find where custom frontmatter was persisted within the existing data structures (only all of the metadata generated by docusaurus), so the solution I could think of would be to:

FYI the debug plugin might be help to inspect the available content: https://docusaurus.io/__docusaurus/debug/content

You are right, and we should make sure to provide you with this info. For now the only work-around is to re-read the file again 馃槄 would be happy to accept a PR that add frontmatter to DocMetadataBase

so the best case scenario would be for the schedule to coexist within that structure and be easily accessible from the sidebar, and for that same sidebar to show up on the schedule page itself

Only docs display the sidebar currently, and if you want to display the sidebars on a non-doc page it can be a bit tricky, as you'd need to duplicate some complicated code of the DocPage component.

If there is an easy way for me to use the existing theme and the same sidebars (besides just dynamically generating a new schedule.mdx in my docs directory), please let me know.

An alternatives is implementing your own SchedulePage based on DocPage, but it duplicates a decent amount of code.

Another alternative is to override loadContent() and "inject" a new artificial doc (I mean, it wouldn't exist in your FS) in the data, so that the contentLoaded() can create a route for it, and so that DocPage can render it, but I think it can be tricky.

I think in this case the best option would be to reuse the DocPage component and make sure that your code provides it with the expected props.

For now in practice it may be simpler to generate your calendar data as a pre-build step, have a component import/render it, and then use MDX to render this calendar component in docs/calendar.mdx, this way it will be a regular doc and you could add it easily to the sidebar.

This makes sense - right now I'm considering an approach like https://github.com/acrobit/docusaurus-plugin-auto-sidebars/blob/45d6723fd3a0e04641b343cf1478827587c9fd5e/src/index.ts#L42 to generate the schedule.json in a well known location, so that I can import it in a component. I don't think this would cause any dependency issues except perhaps during development, which I can live with.

This is likely the simplest option for now.


I think we don't provide enough infra to help you do what you really want.

Let's keep this open and see how we could make what you want easier in the future

@slorber slorber reopened this Mar 26, 2021
@zhouzi
Copy link

zhouzi commented Apr 19, 2021

I have a similar use case where I want to generate docs from a GraphQL schema and have it integrated with the rest of the docs. Since the documentation would be automatically generated, I'd prefer not to write files to the disk. I first tried to hook into the docs plugin's loadContent() to add items that would be caught by contentLoaded(). But as you had foreseen, it is not trivial as the docs plugin expects a pretty complex structure that's a pain to mock. I then tried to hook into the docs plugin's components but abandonned the idea for the same reasons.

In the end I created a plugin that writes files to the disk through loadContent() and contentLoaded(). But now docusaurus is caught into an infinite reloading loop when I make a change. The issue is that I make a change -> docusaurus reloads the plugin -> it writes files to the disk, which triggers a reload again. So I added a command to the CLI to generate the docs and I will probably get rid of the runtime part. Perhaps the plugin function should only be called again if there's a change to the options or context? It could still call its loadContent() and contentLoaded() when the content itself changes. Because to workaround this issue I thought about declaring a variable caching the schema so that contentLoaded() doesn't write files to the disk if it didn't change since the last call. I failed to do so as the plugin function is called again on each reload (and the variable is thus reinitialized).

Also, the sidebar can be tricky to deal with as you can't reference files that are not written to the disk.

EDIT: I created a repository that illustrates the issue with the infinite reloading: https://github.com/Zhouzi/docusaurus-plugin I initialized a docusaurus project and added a pretty basic plugin that writes to the disk: https://github.com/Zhouzi/docusaurus-plugin/blob/main/plugin.js If I run yarn start and then edit a file in the docs folder, it triggers the loop.

@slorber
Copy link
Collaborator

slorber commented Apr 20, 2021

@zhouzi some details:

  • The docs plugin absolutely require files to be written to disk. The markdown files should be loaded through the MDX loader. In practice they turn out to be compiled to React components. You can't really create a docs plugin doc "artificially" , but this will be probably solved to integrate docs with a CMS (Extend existing content plugins (CMS integration, middleware, doc gen...)聽#4138)

  • The way you implemented your plugin is not ideal. In general, you read content from the fs in loadContent, and create appropriate routes, global data etc from that content in contentLoaded. Writing new files in contentLoaded will indeed trigger some kind of infinite loop. And one plugin should rather not mess-up with the content folder of another plugin, as plugins should rather be isolated from one another.

  • I think currently the solution we could provide is a way to write the markdown files to disk in a beforeLoadContent() hook

  • About the sidebar we have a new "autogenerated sidebar" feature in alpha 73, that might simplify things for your usecase?

Note @edno implemented something similar here: https://github.com/edno/docusaurus2-graphql-doc-generator, currently using a cli as a pre-build step, but I suggested to try extending the docs plugin to write md files to disk (graphql-markdown/graphql-markdown#144).

@zhouzi
Copy link

zhouzi commented Apr 20, 2021

Thanks for the insights 馃檹 I think the solution to #4138 would actually solve most of my issues. For now I will follow the example of docusaurus2-graphql-doc-generator and extend the cli.

@slorber
Copy link
Collaborator

slorber commented Nov 4, 2021

Related to discussion: #5856 (reply in thread)

@slorber
Copy link
Collaborator

slorber commented Mar 16, 2022

Related RFC: #6923

@Josh-Cena Josh-Cena added the domain: content plugin Related to content plugin emitting metadata for theme consumption label Mar 29, 2022
@slorber
Copy link
Collaborator

slorber commented Sep 25, 2023

The global data API is not ideal and adding data there decrease the loading perf of all pages of your site, due to having to load all that data before hydrating React.

I think in the future we'll use React Serve Components to access and solve a similar use case in a more flexible and performant way.

Track #9089

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: content plugin Related to content plugin emitting metadata for theme consumption feature This is not a bug or issue with Docusausus, per se. It is a feature request for the future.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants