Unload unused models from server memory

[ricwo]

Proposed changes:

• only ever keep one model per project in the server's memory under Project._models
• unload previous NLU model from Project when a new model is requested by the server

Status (please check what you already did):

• made PR ready for code review
• added some tests for the functionality
• updated the documentation
• updated the changelog

[wrathagom]

I haven't reviewed the code, but as I read the description this would be a big change for anyone using multiple fixed model names under the same project, which seems to be a common practice.

[ricwo]

This only affects the server's in-memory models. The change means the server's project_store only contains the latest-used model, otherwise if you keep switching models you might run out of memory.

You can of course store as many models as you like, but when you switch them (i.e. parse requests a different model than the previous parse) they are newly loaded into memory 🙂

[wrathagom]

In our use case, and several others I am aware of, we use projects as individual bots and serve multiple models simultaneously from within the project. For example, we may have a pizza bot project and then a model inside for each language English, Spanish, German, etc. We manage our own versions with a version number suffixed to the model name.

Just trying to make you aware that what I am describing is a pretty common use case that would have to be re-designed by the implementation of your proposed method.

@tmbo @amn41 any input.

[amn41]

thanks for the feedback @wrathagom ! and great that you keep an eye on the issues before we break anything :)

the case we're trying to guard against is that if you are continuously running a server and training models, the old ones never get unloaded and Rasa's memory footprint will monotonically increase.

I'm sure there's some logic we can implement that supports both use cases, possibly adding an auto_onload flag to the server config, or coming up with another scheme for loading / unloading

[ricwo]

I have made some changes that should also accommodate your use case, @wrathagom:

• fixed model names (those not beginning with model_) are unloaded iff their name differs from a newer one by a version number (the model name has to end on _X or _vX )
• default model names (model_X) are always unloaded iff the requested one differs from the one previously used

@amn41 I guess we could also consider adding a versioning parameter to the model config that increments the model name according to some versioning scheme, or even directly add a version field

[wrathagom]

@ricwo @amn41 I understand the need for it, indeed it's been a community request for awhile. Just pointing out that there was an issue open related to this #438 , specifically having to do with exposing endpoints for managing the loading/deleting of models. If I had my way that would still be the solution, but then again we're building platforms on top of Rasa not just individual agents. Maybe my team can take the code from this PR and expand it a bit to add the endpoints and contribute that back.

I spoke a bit to soon on our specific use case as ours is actually a hash, which includes among other things the date. Small Talk_294b4d80-5295-0b20-3b79-d30f686fb700

[tmbo]

I think it makes sense to do the following:

• addd a DELETE endpoint to manually unload models (no deletion on disk, just unloading from mem) - this should be simple as we already have the code for the unloading (@ricwo let me know if there are issues)
• list the loaded models in the status endpoint (right now we list all, so it is not clear which ones are loaded - whatever is easiest here (e.g. adding an attribute, creating a separate property next to available_projects))
• don't do any automatic unloading

[tmbo]

should we use a default here?

[tmbo]

lets move this default to the endpoint level

[tmbo]

I think it is fine to just return the model without the "unloaded_model" wrapper, so just return unloaded_model

[tmbo]

if interpreter is not None

[tmbo]

params.get("project", RasaNLUConfig.DEFAULT_PROJECT_NAME)

[tmbo]

please also log to logger

changes have been made :)