Roadmap for the "projects" plugin

[wilhelmer]

I love the idea of the new projects plugin, thanks for developing this!

Currently, the plugin behaves more like mkdocs-multirepo and less like mkdocs-monorepo. Specifically, there's no merged seach index and no merged navigation.

However, you mention that you're planning to "add support for merging files". So are you envisioning this plugin to be more like monorepo in the future? Or a mix of both?

[squidfunk]

We are aware that there are several multi- and mono-repo plugins, but we're trying to solve different use cases and, as always, make it dead simple 😊 The idea of the projects plugin was born out of our own need to setup the examples repository, which we will start to fill with self-contained and downloadable examples for each and every feature that Material for MkDocs supports to show how features can be configured, and also reduce the communication overhead for us. It's part of our documentation restructuring effort.

Our requirements are:

• Build a tree of projects: of course we also want to demo the projects plugin in the examples repository, so building not only a list, but a tree of projects must be supported. It's important to note that we want to provide an easy way to build projects in the same repository (multi-repo is out of scope, but might be achievable with submodules). This is already implemented.
• Keep specific structure separate: we mainly want to build projects that are kept separate, e.g., different languages which should have separate search indexes, and also different sitemaps. This is already implemented, but we're still working on better support for building and interlinking projects, like multiple languages. Any feedback is appreciated.
• Intelligent serve mode: you start mkdocs serve from the root project and all projects will be built and served and only rebuilt when they change. This is already implemented. It should be especially useful for very large sites that can be cut into multiple pieces by using the plugin, but we might need to add additional support for interlinking projects again that we want to learn about.
• Hoist theme assets: we don't want each project to have its own copy of Material for MkDocs theme files, as this is hugely redundant and slows down deployment and loading of your site. This is already implemented and was released yesterday. We might, however, add support for hoisting author-provided files as well in the future.
• Concurrent builds: maybe one of the best things is that projects are built and rebuilt in parallel, as all new plugins that are part of Material for MkDocs that perform expensive operations! This might massively speed up build times, especially for large projects.

Ideas we plan to explore:

Modes

Languages

We want to support different modes for the projects plugin. As already mentioned, multi-language support is one of the prime candidates for the plugin (and the most upvoted and commented discussion on our GitHub repository!), and it's something that users are struggling with. The plugin should make this simpler and guide the user through setting up a multi-language installation.

When you write multi-lingual documentation, you likely don't have each page translated, but that's you goal, i.e., mirror your documentation in different languages. The plugin might help you with that by automatically copying or redirecting to fallback versions of untranslated pages, and add a custom designed banner that is provided by the author. Somehow similar how FastAPI is doing it.

It's essentially also similar to the mode of operation of the mkdocs-static-i18n plugin, but allows for maximum flexibility, as you can translate every aspect of your project without the plugin needing to add support, e.g., for translating the cookie consent. Thus, the requirement is that you can set up a new language, translate one page and deploy it, and the plugin will mark all other pages as not-translated, but you have something working right now.

Versioning

While git-based versioning is much superior to folder-based versioning in terms of storage, piggy-backing on git's beautiful tree model, some users just can't use the approach that mike implements. They want or need to separate their versions into folders. This is similar to the language case, but a little different, as each version might add new pages, and old versions are usually rarely touched.

Read about scalability concerns in Docusaurus related to folder-based versioning.

Dependencies

The projects plugin allows to build a tree of projects, but currently all projects are assumed to be independent and built in parallel. If we discover use cases where we need projects deeper in the tree to be built before other projects because of dependencies, or because of additional information the plugin might expose (e.g. a list of links, consolidated navigation, you name it), we can add a mode where projects can be built in the correct order. The current enumeration of projects is already a post-order tree traversal, so adding support for "first build A and B, when that's finished, build C" is definitely in scope.

Merging of files

We also thought about that users might want to merge files, e.g. the search index or sitemap. Developing the plugin, we made sure that this can be added when necessary, and I think it will be necessary.

Linking between projects

It will be possible to link between projects by prefix and Markdown file. The plugin will resolve to the actual file location and create an
appropriate link. This might not be necessary for multi-lingual documentation, but for consolidation of multiple projects into one.

Configuration

The projects plugin will make use of the extra.alternate feature, and should allow to configure things centrally, so that you don't have to define the language switcher (or more generally speaking "alternate version switcher") in each and every project. This might also be necessary for other configuration options as we learn about new use cases.

Additionally, solving #4835 will make it even more powerful 😊

---

If you or somebody else has other ideas to share, feel free to drop them here 😊 When we finished restructuring our documentation, we will construct and provide canonical examples for the use cases we identified so far as part of our documentation.

[wilhelmer]

Here's another use case – our use case:

Say your company is selling products A, B, and C. For each product, there's product documentation. Each product documentation should be deployed individually, e.g., to be bundled with the product.

But also, there should be a general platform where all products A, B, and C should be documented. The UX should be seamless, so users don't notice that the platform consists of several sub-projects. There should be one unified search, and one unified navigation.

This is a use case that mkdocs-monorepo serves quite well, but the project isn't that active anymore. So I was hoping your plugin could fill that hole.

[wilhelmer]

When building the main project, the navigation should be entirely consolidated in all projects. When building a subproject, only the navigation of the subproject should be displayed.

Is that feasible? Sounds tricky or even impossible to me, but maybe you have something in mind?

If not, a compromise might be to only display the parent node of the subproject's nav tree and visually indicate that clicking that link will switch to a new site with a different navigation:

And in the subproject, there could be a "Go back to main project" link.

[squidfunk]

Is that feasible? Sounds tricky or even impossible to me, but maybe you have something in mind?

Yeah, so I thought about this a little, and I think it might be possible, even without loosing parallel builds, but at the price of a limitation: no support for plugins that alter the navigation or add pages dynamically. We would need to resolve the navigation from the project prior to building the project. That would also mean that the blog plugin's generated navigation wouldn't be part of the main tree. Hmm. I will think a little more about it, maybe I can come up with something, but it's definitely a complex problem to consolidate all navigation in all projects with support for arbitrary plugins.

What might work is to run two builds – one to record the effective final navigation of a subproject after building all projects, and the second run to build them with consolidated navigation, but that's clearly not optimal. Nonetheless, a lot of ideas to explore!

[squidfunk]

Okay, so what might be possible is the following approach:

• First run, build all projects, and in the page_context event, record the navigation of each project and write it to the .cache directory. Then, perform a second run and rebuild all projects, because only now, the entire navigation is available. This would include generated pages if we make sure that the page_context handler runs latest, as it's the last event triggered before actual rendering. Compute a digest for each navigation and save that with the navigation tree to .cache.
• On the subsequent run, assume that projects did not change and use the navigation from the .cache to build all projects. Navigation changes far less often than content, so I think this would be a good trade-off. Compute the digest in each project nonetheless and if it didn't change, only a single run is needed.
• If the digest changed, mark the run as stale and rerun it again, updating the navigation in the .cache. This would be entirely transparent to the author, the plugin would take care of everything and could notify the user on the command line that another build is necessary because of stale navigation.

This would mean that navigation would be guaranteed to be correct at the expense of an additional concurrent build. However, it would also mean that projects can use plugins like the awesome pages plugin, because we would always observe the final result and cache intelligently at the expense of a second build. The second build would always need to rebuild all projects, since the navigation for all of them changed.

Of course, this mode would be opt-in, so the author would turn on a flag that consolidated navigation should be built. Normals runs of the plugin would not be impacted. Additionally, building projects themselves would not be impacted.

[squidfunk]

This will be tricky, but it should be achievable. I threw together a prototype which renders the consolidated navigation in the top-level project, taken from our examples repository:

However, it's far beyond just parsing a navigation and injecting it into the right location, because then, many of the navigation features like icons and subtitles, as well as the typeset plugin will not work. I already got an idea how we account for those, but puh, it's likely going to take some time 😅

Edit: with some additional effort, it even works for generated pages, as I previously stated. For example, we added a blog to the "Privacy" subproject for testing, and the links are correctly rendered:

[squidfunk]

By the way, unified search will not be tackled as part of the projects plugin, but as part of implementing search federation as requested in #5230. It's one of the requirements for the new search we're also working on.

[squidfunk]

We wanted to share that we made progress and decided to offer both options:

Link projects in the navigation

74dc02153 adds support for linking a project at an arbitrary location in the navigation of the top-level project. It's currently not yet possible to cross-link projects (e.g. create a link from Product A to Product B if they're siblings), but we'll tackle this as a next step.

You can now do the following in mkdocs.yml:

nav:
  - Home: index.md
  - Product A: project://product-a # -> projects/product-a/docs/index.md
  - project://product-b            # -> projects/product-b/docs/index.md, using the project's site name
  - And this:
    - Can also be:
      - Nested:
        - project://product-c      # -> projects/product-c/docs/index.md, using the project's site name

Note that this already works over arbitrary levels, so you should be able to nest projects inside projects inside projects. However, you can currently only link projects on the next level. We'll remove this limitation as well. Another cool thing is that the plugin will use the project's site name when you don't provide a link title.

We decided to go for the URL syntax with the project scheme, because it translates perfectly to creating inter-project links inside Markdown and integrates well with everything else. In the near future you'll also be able to link from a project to another one, like so:

[Product A Reference](project://product-a/reference/index.md#overview)

We'll provide runnable examples in the coming days/weeks that can be checked out.

Generate consolidated navigation

We're still working on that, but we're very convinced that it's possible to achieve, as we're already close. You'll be able to use the exact sample example as above, and turn on a configuration flag that will not link but consolidate project navigation among all projects.

[Blacksmoke16]

@squidfunk Sure, Give this a try:

9.5.12+insiders.4.53.0-project-site-url-404.zip

For me I can run mkdocs serve and see the 404s in the log output when navigating to the /MyApp page.

[squidfunk]

My fault: the info plugin in Insiders currently does not properly pack up projects. See #6750. Could you please manually pack up the archive and attach it here? We've fixed this in #6826 but it still needs to be merged released. Sorry for the extra hoops.

[Blacksmoke16]

@squidfunk Ahh, no worries. Give this a try then:

bug.zip

I had to exclude to venv and such due to size restrictions, so will need to set that up and install insiders version again, but that should be the project structure that reproduces this.

[squidfunk]

Yes, this is definitely a bug. It only seems to occur when you do not have a base path set, i.e., if you deploy your documentation on the root, so http://example.com/ triggers the problem, while setting. site_url to http://example.com/docs/ does not. Could I ask you to open a new issue, so we can tackle it? This discussion is meant for discussing the general direction of the plugin.

[squidfunk]

It's directly related to #6437. If we revert this, it works as expected.

[SaltyAimbOtter]

We have another scenario that could fit within the scope of the projects plugin, especially regarding the "versioning" part:

Our documentation is created separately for each software release using mike. Our goal is to integrate the same version of the blog seamlessly into all versions of the docs. New blog posts e.g. about releases only in the DEV version's docs aren't very helpful 😛

It would be great if this was possible without rebuilding all old releases everytime a blog change is made. Perhaps some sort of linking between a global blog folder and subfolders for each version would be possible.

[squidfunk]

Thanks for sharing! Definitely harder to pull off, because I'm not sure whether mike can deploy on a subpath, and mike it outside of MkDocs which we currently only address. I'm not sure this is solvable with the plugin, but it might be with some more legwork and a custom workflow. It's, however, definitely a valid use case. Have you discussed with the maintainer of mike?

[squidfunk]

I've looked at mike, and it looks like that this is perfectly achievable. You can use the deploy_prefix setting to specify a subpath at which your versioned docs site should live, and then only deploy the versioned site with mike, the blog with MkDocs. You would need to do separate deployments, because it's (currently) outside of the scope of the plugin, as it doesn't govern mike. Nonetheless, you should be able to preview both sites simulatenously with the projects plugin locally – you would just need to do separate deployments.

[wilhelmer]

@squidfunk Any plans for when you will continue working on this? I'd love to see the consolidated navigation, even as an early alpha.

Another suggestion: When linking projects via project://, the subprojects should inherit the main project's use_directory_urls setting. Currently, it is always true for linked projects.

[squidfunk]

I'm working hard finishing the docs restructuring in #5692. Then the next things I'll work on will be search and the projects plugin. I currently can't put an estimate on it, but I have an almost working prototype that should be finished soon.

I also thought about allowing to inherit specific settings to projects, and use_directory_urls could be one of them. As always, we want to make this as configurable as we possibly can ☺️

[squidfunk]

We've decided to settle for a more flexible way than inheriting specific configuration settings. The master of Insiders contains new functionality for the projects plugin, which allows to transform project configurations after loading them. This means you can also inherit the use_directory_urls setting. First, create a python file at the top-level of your project, e.g. projects.py:

from mkdocs.config.defaults import MkDocsConfig

def transform(project: MkDocsConfig, config: MkDocsConfig):
    project.use_directory_urls = config.use_directory_urls

Then, instruct the projects plugin to use the function with the following configuration:

plugins:
  - projects:
      projects_config_transform: !!python/name:projects.transform

Note that you must set PYTHONPATH=. when calling MkDocs, so the current directory is used for resolution of modules. A better approach is to put your transform function into a separate package that you install, but that's requires extra effort. If somebody knows how to make it even simpler to use local modules/functions without having to explicitly set PYTHONPATH, please share:

PYTHONPATH=. mkdocs serve 

We're still working on navigation consolidation, but we need to sort out some other things first.

[squidfunk]

Changes have been released as part of insiders-4.42.0. The project plugin brings two new settings:

• projects_config_files
• projects_config_transform

[Blacksmoke16]

This is looking quite promising! I wanted to get your thoughts on an additional use case of mine.

I have a project that's setup as a monorepo, with multiple components (sub-projects) nested within it. These sub-projects are leveraging the mkdocstrings-crystal and gen-files plugins to generate the documentation pages from the source code. I.e. the docs/ directory is empty of any hard-coded files. Each component is leveraging the INHERIT mkdocs configuration option to share common parts of the configuration. Ideally I think each component would be versioned independently as well. The components also link to other components, which the project:// links would be perfect for.

The main idea is wanting to have another site at the root of the monorepo that serves links to each component's API documentation, defaulting the the latest version, while also serving additional static documentation from its own root docs/ directory. The root project would use the navigation.tabs feature, while each component would not.
Based on my play testing, the projects plugin works quite well for this so far. However, what I'm trying to figure out now is a good way to get from one of the component's docs back to the root site?

For my use case, the ideal UX would be if I could make the root project's navigation tabs sticky such that it is displayed when viewing a specific components documentation. But I'm definitely option to other options/suggestions!

[squidfunk]

Thanks for sharing your use case! This sounds all manageable by the plugin.

However, what I'm trying to figure out now is a good way to get from one of the component's docs back to the root site?

You could use extra.homepage to point to the root of the site. If users click on the icon, they would get back. Other than that, you can also use a project://. link in the navigation of a component, which will link back to the root site:

nav:
  - Back to home: project://.
  - ...

We're still figuring out the optimal mechanics of interlinking and navigating between projects, so any feedback is valuable. Additionally, our documentation on leveraging the plugin to create multi-site installations is quite poor. The recent documentation restructuring by moving out the plugin section laid the ground work for improving it. We'll be adding examples and different use cases in the coming weeks. The plan is to build an ever growing list of runnable examples ☺️

For my use case, the ideal UX would be if I could make the root project's navigation tabs sticky such that it is displayed when viewing a specific components documentation. But I'm definitely option to other options/suggestions!

Once we implemented merging of navigation, this should be possible.

[Blacksmoke16]

Awesome, thanks for the info! I'll be sure to watch the changelog closely for future updates 💪.