Create a ZIM with all versions of a given project

[benoit74]

Currently, it is only possible to create one ZIM per slug. E.g. one for one for lua5.4, one for lua5.3, one for lua~5.2 and so on.

Analyzing a bit devdocs, it showed this means we will create 716 ZIMs, with new ZIMs coming at every release of projects (lua, ...).

This is way too much, and this is not really convenient for the end-user who typically know he needs documentation of a given programming language, but usually need documentation for multiple versions.

We would like the scraper to be able to embed all versions of a given project (e.g. lua) in a single ZIM. This obviously means significant changes to the codebase, both in terms of ZIM structure (to accommodate to multiple slugs in one ZIM) but also in terms of UI (to be able to switch from one slug to the next one).

It probably also means that something needs to be done to provide custom search capabilities directly in the UI: currently suggestions and full-text search are already hard to use, but it will become even harder if we have multiple slugs in one ZIM. Or maybe this means we need to enrich the search capabilities of the libzim (e.g. ask the libzim to search only in a subset of articles ...).

To make the feature as generic as possible, I think that we should continue to use the CSV --slug parameter to specify all slugs that we want to embed, and add a --oneSingleZim (or something like this, I'm bad at choosing names) parameter to indicate we want all slugs in a single ZIM file. With this approach, it means we have lot's of flexibilities, and we can for instance embed all Vue (Vue, VueRouter, Vuex, ...) in a single ZIM file even if they are distinct project son DevDocs side. Or we can even create a "all DevDocs" ZIM with all documentation.

[benoit74]

I've opened the discussion about search issues to be (potentially) solved at libzim level in openzim/libzim#943

[josephlewis42]

I like this idea and I think it's a good thing to keep in mind when approaching an SPA.

It probably also means that something needs to be done to provide custom search capabilities directly in the UI: currently suggestions and full-text search are already hard to use, but it will become even harder if we have multiple slugs in one ZIM.

Thinking broadly here, would it make sense to push this up to the ZIM level too e.g. a ZIM that combines multiple ZIMs?

I'm thinking of the user experience of a user switching the version in the DevDocs UI and then doing a search. Would the UI have some way of propagating up facets to the ZIM reader for search?

[benoit74]

Good questions, no answers for now on my side.