eventually the index will outgrow a single JSON file, no?

[BryanWilhite]

what is the plan when the index grows to, say, 1GB or more?

what learnings are there out there to help you as you tread in deeper architectural waters?

[BryanWilhite]

paging the index by date and paging by category (and then by date) is the simplistic solution; this would require the generation of two sets of “sharded” (?) index snapshots

my ignorance insists that someone else smarter than me and this problem and did not resort to something like Redis cache or MongoDB

[BryanWilhite]

here is a new idea:

site/
    html/
    index/
    md/
    templates/

drilling into index/:

index/
    page-01.json
    page-02.json
    page-03.json
    ...
    page-<nn>.json

    groups/
        by-year-month/
            ...
            <yyyy>-<mm>.json

        by-topic/
            ...
            topic-*.json

• every page/group JSON file will contain metadata about the entire set of pages or the entire group such that loading the first file allows loading of the other files on demand
• the JSON files will hold arrays of MenuDisplayItemModel [docs]
• index/page-*.json should come directly out of md/
• index/ groups should come from page-*.json
• the flexibility promise of 11ty should handle md/ to populate html/

[BryanWilhite]

New direction: I think the Index App should be based on lunr and Bing Web Search (or some open source equivalent).

The lunr folks on pre-building large indices:

This serialized index can then be written to a file, compressed, and served along side other static assets.

https://lunrjs.com/guides/index_prebuilding.html