Scene compendium packs may often include base64 thumbnail images in their index which substantially increases the size of the initial indexing requests that are needed when the Foundry VTT client first loads.

[aaclayton]

Motivation (why we index)

Compendium packs are automatically indexed on world load. There is one driving reason for this: dynamic document links like @Compendium[dnd5e.monsters.Acolyte] won't work without the index present. Such links rely on either _id or name-based lookup and without the index they will be initially broken.

The Problem

When we index a compendium we retrieve the _id, name, img, and type of each document in the collection. Some compendium packs have an expensive index due to images which are encoded as base64 making the payload for obtaining the index unexpectedly large - this is especially noticeable on slow connections. A typical text-only index is on the order of 100 kB, but a compendium that has base64 images for all its entries could be as large as 20MB or more.

Furthermore, since each compendium pack is indexed individually, this results in a number of socket requests equal to the number of active packs to construct all the indices - with more network round trips involved.

Planned Solution

Return Compendium packs to a state of lazy indexing, but send a "minimal index" as part of the initial payload provided to each user when they connect to the world. This minimal index contains ONLY _id and name - allowing dynamic document links to fully function. The compendium is not "fully indexed" however until getIndex() is asynchronously called. We maintain a state flag, indexed so that the first time we actually view the Compendium in the UI we ensure that we obtain the full index including images and other relevant fields at that time.

Bonus Points

As a bonus, this solution expands the API allowing for modules to customize the set of fields which should be included in the "full index". For example, the dnd5e system could request to include data.details.type for Actors or data.details.school for Items to be able to add custom filters, categorization, or more for things like NPC creature type or Spell School.

---

What Would I Need To Change?

If you rely upon CompendiumCollection#index in your module or system to implement features like custom importing, compendium browsing, or other similar features you may need to make a change to manually request an expanded index prior to taking further action. By default the compendium incides will now only contain _id and name upon initial load. You can request the expanded index - or even a custom index with additional fields - by calling the getIndex() method.

---

API Examples

Loading the index with additional custom fields included

const pack = game.packs.get("dnd5e.monsters");
await pack.getIndex({fields: ["name", "type", "data.details.type"]});

Loading the expanded index if it has not already been acquired

const pack = game.packs.get("dnd5e.monsters");
if ( !pack.indexed ) await pack.getIndex();

[aaclayton]

Originally in GitLab by @anathemamask

Comment from the community devs:

It would be beneficial to include 'type' as one of the defaults for INDEX_FIELDS

[aaclayton]

marked this issue as related to #5478

[aaclayton]

Originally in GitLab by @gcolgate

Now that this is supposed to be fixed, when I run (I have comcast, so my upload speed is a miserable 5 mps) when they first log in users spend up to 5 minutes indexing compendiums. (I don't know why so many compendiums need to be indexed for players, items and spells might be the only ones necessary).

Today my internet was suffering 1 at 1 mps, (Comcast's fault, they are sending a technician) so the game had to be cancelled as it was taking 25 minutes to process the Compediums and often timed out.

Would it be possible to start the game BEFORE the compendiums have been completed? That would sove the problems as looking into compendiums is not a high priority for the end users, it only comes up when they are updating their characters.