Peroidic tasks to remove incomplete file uploads

[yaswanthsaivendra]

Is your feature request related to a problem? Please describe.
Currently, I believe we don't have a logic for deleting the incomplete s3 file uploads.

Describe the solution you'd like
Create a periodic_task which handles the deletion of incomplete s3 file uploads.

soln I'm thinking of - the function would be deleting the files which are created till yesterday and are having upload_completed=False

[yaswanthsaivendra]

Hello team,
Lemme know if its a relevant issue, if so I would like to work upon it!

[rithviknishad]

Not sure how relevant or required this is.

But what comes to my mind at first glance are the following:

1. We could archive it instead maybe? (set is_archived=True)
2. Do this only for records with created_date older than 1 day? (to avoid running this task over records that are being uploaded at the moment)
3. When we upload a file, is the front end aware that the file upload to S3 has failed? And, should we support retrying the upload?

@gigincg @vigneshhari what do you think about this?

[gitone912]

i want to resolve this issue, can i start ?

[yaswanthsaivendra]

i want to resolve this issue, can i start ?

No this one is still need to be discussed. Meanwhile , you can work on other issues.

[Ashesh3]

@rithviknishad Regarding the relevancy of this issue, I noticed that we do have a pending TODO request about this behavior.

care/care/facility/models/file_upload.py

Line 23 in ed9361b

# TODO : Periodic tasks that removes files that were never uploaded

So, maybe we can have a periodic task to check upload_completed and delete incomplete FileUpload objects with a 1 day grace period. Let's still get a second opinion.

[rithviknishad]

Just adding the stats here of our staging file uploads.