Remove document cache after a certain inactivity

[fflorent]

Describe the problem to be solved

Currently, in a multi-workers architecture, the /persist folder is filled of documents that has been loaded. This cache is only removed after a doc worker restart.

Describe the solution you would like

It would be neat that once the document has been closed, and after a certain time of inactivity the local copy in /persist is deleted.

If in the meantime (so after a document has been closed) there is some activity, the document is loaded again and the timer for wiping the local cache is disabled.

[paulfitz]

The "Housekeeper" might be a natural place for this code to live:

grist-core/app/gen-server/lib/Housekeeper.ts

Lines 29 to 41 in ec0b885

	/**
	* Take care of periodic tasks:
	*
	* - deleting old soft-deleted documents
	* - deleting old soft-deleted workspaces
	* - logging metrics
	*
	* Call start(), keep the object around, and call stop() when shutting down.
	*
	* Some care is taken to elect a single server to do the housekeeping, so if there are
	* multiple home servers, there will be no competition or duplication of effort.
	*/
	export class Housekeeper {

It may be simpler than other housekeeping tasks, since it is scoped to an individual doc worker.

[fflorent]

I thought of this place as well. But I have rather considered adding a timer in ActiveDoc for these reasons:

• there already exists a timer for inactivity before closing the document, I thought of adding another one which would be enabled after the document is closed for inactivity;
• instead of letting each document worker inspect their documents and wipe the unused cache, the document would remove its own cache just like it closes itself;

If that makes sense, otherwise I would be glad to hear your feedback :).

[paulfitz]

I have rather considered adding a timer in ActiveDoc

Hmm you'll need to work carefully. When a document is unused for some time, its ActiveDoc will currently shut down to recover RAM, which means that the ActiveDoc no longer exists. That would be a bit early to wipe the file from disk in my opinion, since it would be common for a document to be visited say once a day. You do want to reclaim RAM in that scenario, but disk space, arguable, given that recovery from s3 can take several seconds (depending).

It is possible to add a new state to ActiveDoc, but it will need a good mental model to avoid breaking interactions with DocManager and other classes. Definitely doable but be prepared for adventures.

[paulfitz]

The bulk of RAM is held by the sandbox associated with the ActiveDoc so holding ActiveDocs around after sandboxes shut down is certainly not ruled out, just a significant change.

[fflorent]

Oh, I see, a shutdown unregisters the ActiveDoc from the DocManager, which means, if I am correct, that reopening the document after that would mean create a new instance of ActiveDoc.

I thought the ActiveDoc instance would be reused.

Now using Housekeeper makes total sense to me. Thank you @paulfitz!