Investigate building docs from scratch

[willmcgugan]

Investigate the experience of getting the docs to build from a fresh environment.

We've had a few issues that suggest there may be some problems there, related to dependancies.

[davep]

Related discord thread.

Pretty sure this wasn't an issue as of #1726 making it into the repo; so it looks like something in the toolchain may have changed between then and now.

[paul-ollis]

Hi.
I am the person who identified this yesterday.
I have tried this in a venv with the same results.
After poking around, my understanding is that there are 2 missing mkdocs plugins - privacy and blog. My understanding is that they are both built-in plugins, but only (currently) available to mkdocs sponsors.

[davep]

After poking around, my understanding is that there are 2 missing mkdocs plugins - privacy and blog.

Indeed, see #1726 -- this is what that work was for. It moved the dependency on the blog plugin, which is only available to mkdocs sponsors, into a separate branch of docs building Makefile targets (the blog plugin isn't useful to people generally, it's just for building out own docs which includes our blog). Without diving in, what seems to have happened here is that the privacy plugin's status seems to have changed between then and now (that's to say: I don't know that that's the case, but that's the first thing I'd check; back when doing that PR I tested building from scratch without access the the insiders' edition and had no problems).

[paul-ollis]

I tried deleting 'privacy' from mkdocs-offline.yml. That seems to allow 'make docs-build-offline' to work OK.

[davep]

As I recall (I'd need to go back over what I did in February), the problem with that is that the "offline" serving and building of docs was intended to be just that: possible to build and view without an Internet connection; and I seem to remember that the privacy plugin was important to enabling that.

I would expect removing it would remove the error, but it's likely that it reintroduces the issue that #1726 was mainly seeking to address (in part allowing GNU/Linux distributions to build and make available the docs without needing a working net connection).

Whatever the correct fix is here, that's an aspect we'll need to keep in mind.

[paul-ollis]

FYI. The offline docs I was able to generate appear to be pretty much fully functional, including the search feature. I seem to be able to use then OK without any network connection. Definitely good enough for my current needs.

The icon at the top left of each page:

does not work - the link is a directory (I assume it is missing 'index.html'). Everything else I have tried so far works as I would expect.

[TomJGooding]

Without diving in, what seems to have happened here is that the privacy plugin's status seems to have changed between then and now (that's to say: I don't know that that's the case, but that's the first thing I'd check; back when doing that PR I tested building from scratch without access the the insiders' edition and had no problems).

I might be mistaken, but looking at the mkdocs-material Insiders page, it looks like the privacy plugin features won't be merged for general availability until the $20K funding goal? Perhaps that was changed since Feb, but couldn't see anything obvious from a quick browse of their repo.

From a quick read through the posts on #631, it looks like the plugin was added to bundle some external assets used in the docs. Need to check if that's still reproducible now. It looks like mainly just fonts from what I can tell so far...

[TomJGooding]

Probably a naive approach (and I'm not too confident with my regex!) but searching "https" in docs seems to confirm mainly just fonts from the excalidraw SVGs.

grep -r -P --exclude-dir=blog '(?<!(?:f"|\]\()|\[link=)https' docs | less

[squidfunk]

I might be mistaken, but looking at the mkdocs-material Insiders page, it looks like the privacy plugin features won't be merged for general availability until the $20K funding goal? Perhaps that was changed since Feb, but couldn't see anything obvious from a quick browse of their repo.

@TomJGooding the built-in privacy plugin will be merged when the $14k goal is reached, which should cover all you need to bundle external assets ☺️ There are some additional features to the plugin that include optimization support (automatically optimize downloaded images) and handling of external links, that will be merged when the $20k goal is reached.

With GitHub auto-cancelling all PayPal subscribers, we lost a significant amount of sponsors, but we hope to reach the $12k (blog plugin etc.) and $14k (privacy plugin etc.) goals within this year. Also note that once tied to a funding goal, we always stick to it. The only thing we allow ourselves is to move features into lower goals, releasing them earlier as planned, but that only happened once so far. We will never move features into higher goals after having committed to a funding goal.

Feel free to mark this as OT.

[davep]

Looking into this, steps taken and what I've found so far:

1. Cloned a fresh copy of Textual main
2. Ran make setup
3. Ran make docs-serve-offline

The result is:

rm -rf .screenshot_cache
echo "INHERIT: mkdocs-offline.yml" > mkdocs-nav-offline.yml
grep -v "\- \"*[Bb]log" mkdocs-nav.yml >> mkdocs-nav-offline.yml
poetry run mkdocs serve --config-file mkdocs-nav-offline.yml
ERROR    -  Config value 'plugins': The "privacy" plugin is not installed
Aborted with 1 Configuration Errors!
make: *** [docs-serve-offline] Error 1

The current setup comes from #1726; the inclusion of privacy in the offline version of the documentation is done by 45af25a where I seem to have included it for the very specific purpose of helping with allowing the documentation to be built and used without a net connection; this was one of the requests made that motivated these changes.

I was pretty certain that, after work on #1726 had mostly concluded, I'd tested this with a fresh environment where the mkdocs insiders code wasn't available; curiously we've also not had any complaints from downstream either. Perhaps I'm misremembering that I tested this, or somehow managed to accidentally test in an environment that wasn't as pristine as I'd thought.

Either way: while it looks like the privacy plugin is one of the ways to go to let third parties build and read documentation offline, and also do so in a way that doesn't require a net connection at all and so maximises user privacy, it obviously doesn't fulfil that role at the moment so I'll look at removing it and seeing what impact it has on the original requirement mentioned in #631.

Hopefully it doesn't turn out that "anyone can build" and "can be built 100% offline" are mutually exclusive desires.