Track unused models and automatically delete them

[miquelduranfrigola]

Background

Most models in Ersilia require substantial disk usage. Deleting a model means removing all model files as well as the corresponding conda environment or docker container. A desirable feature would be to automatically remove models if they have not been used for e.g. a month.

Requested feature

Keep track of model last usage date, and delete it automatically if considered to be unused.

[Riyabelle25]

Hi!
I have experience with Software Development for ML tools, and would love to contribute here! I have set up Ersilia and have been going through the codebase. Can I take this up?

[GemmaTuron]

Hello @Riyabelle25,

As I mentioned, please first complete the requirements in issue #36. Then we advise you to choose one issue to start working with, either this one or issue #7 where you have also shown interest

[Amna-28]

Hi!
I have completed the Ersilia installation and have run the model. I am really interested in this issue.
May I start working on this task?

[miquelduranfrigola]

Hi @Amna-28 and @Riyabelle25! I see that both of you are showing interest in this task. Before implementing anything, I'd like to hear your thoughts, so that we are all on the same page (this issue may be a bit tricky to address). How would you like to approach this problem? (one or two lines will suffice :))

[Riyabelle25]

Hi there!
First, could you tell us from where do we delete the unused model?
In my last issue, I observed that each model is fetched from its own repo: https://github.com/ersilia-os/eos6tg8
So is it desired to delete the git repo itself?

[Amna-28]

I have been working on this task.
I have created a python script that uses python os and datetime to filter files that are not accessed for n days and deletes them and used cron job to automate the deletion after required interval.
I was searching and found that we can create cron job using shell script, but I am unsure about whether we could automate the creation of cronjob on another system.
still working on finding which specific directories to delete
@miquelduranfrigola please correct me if I am going in the wrong direction

[miquelduranfrigola]

Hi @Riyabelle25 and @Amna-28 - I am slowly catching up.

You will see that I have a ModelFullDeleter class in: https://github.com/ersilia-os/ersilia/blob/master/ersilia/hub/delete/delete.py This is the class that is called by the CLI command as well: ersilia delete ...

This is the right way to delete a model (@Riyabelle25 it will delete the local copy of the github repo indeed, among other things).

@Amna-28 I really like your cronjob approach, as you know. I have no experience in this, but perhaps we could try this python library? https://pypi.org/project/python-crontab/

[Rufaida98]

Hi @miquelduranfrigola can we do DVC on this issue to track the files and then remove it. I am still learning and searching to how make work remotely. Ease guide me :)

[miquelduranfrigola]

Hi @Rufaida98 this is a great suggestion. We will certainly use DVC in the context of model data management (https://github.com/ersilia-os/isaura), but I feel the current solution, with cron jobs is good enough, because models are not being updated or anything - they just need to be removed after some time.

[Rufaida98]

@miquelduranfrigola oh I see thank you for the feedback 👍

[sonali-sk]

Hi @miquelduranfrigola, I have a couple of doubts, If a model is already fetched and is in the local system,

1. If we fetch the same model again, will the model's repo along with other related files be fetched again from git-lfs or from the local system?
2. Also when we know that a model is already in the local system, should that model be fetched again or is it safe to go ahead serving?
   Thanks.

[miquelduranfrigola]

Hi @sonali-sk thanks for the two questions.

At the moment, by default when you fetch a model that is currently available in your system, the model is deleted and we fetch it again. We can certainly optimize this, perhaps just by using git functionalities.

[sonali-sk]

@miquelduranfrigola Is it being deleted so that the repo's last modify date is changed? Also, what I noticed initially is that when we try to fetch an already fetched model, it took lesser time compared to the current fetch mechanism, at that time was it fetching from the local system, due to which it took relatively less time?

[miquelduranfrigola]

Good points, @sonali-sk. Fetching a second time takes less time in this case, probably because conda has already cached some packages.

I do think that checking for the "last modified" data point in the repo would be a good way of avoiding unnecessary extra downloads. We haven't implemented it yet. Will keep you updated.

[miquelduranfrigola]

Hi folks:
I am closing this issue since we've done quite a lot of progress since May.
The fetch command is now, in principle, ready to allocate checkpoints.