Temporary Upload Files Should be Deleted

[elad-levi-cyberark]

Describe the bug
Files are not being deleted after upload (though the content is indexed to OpenSearch and the file is not required anymore)

To Reproduce
Steps to reproduce the behavior:

1. Use dftimewolf plaso_ts recipe
2. Wait for the entire process to end
3. Go to upload persistent volume
4. The file is not deleted

Expected behavior
The upload file should be deleted after the process ended successfully.

Additional context
I'm running Timesketch using docker compose.

[jkppr]

It is a feature not a bug. Depending on the setup some teams prefer to have a copy of the uploaded files.

Therefore it is left to the instance admin to decide what to do with the files in the upload folder. If you want to delete them, I recommend to add a nightly cron job to clean out the upload folder.

[elad-levi-cyberark]

It is a feature not a bug. Depending on the setup some teams prefer to have a copy of the uploaded files.

Therefore it is left to the instance admin to decide what to do with the files in the upload folder. If you want to delete them, I recommend to add a nightly cron job to clean out the upload folder.

Understood.
How about linking the uploaded file with the sketch so that when the sketch is deleted, the uploaded file is removed as well?

[jkppr]

Summoning @jaegeral who is working on the deletion topics. What do you think about this idea?

The path is already linked with the datasource object. So it would be only about the deletion part that is missing.

[jaegeral]

Yes I think that is a valid feature request to have, maybe with a Config file parameter to enable / disable the deletion of files during Sketch delete.

[JakePeralta7]

I've also noticed that my OpenSearch volume is getting bigger and bigger even if I delete all of the sketches, is it possible that the indexed data is kept when the sketch is deleted?

[jkppr]

Yes this is correct. Deleting a sketch does not remove the data from the searchindex. It just closes it to save resources.
We are working on a PR that will introduce a real wipeout deletion soon and make this more transparent to the user.

[jaegeral]

I've also noticed that my OpenSearch volume is getting bigger and bigger even if I delete all of the sketches, is it possible that the indexed data is kept when the sketch is deleted?

You can now try the delete feature in the backend.

[JakePeralta7]

I've also noticed that my OpenSearch volume is getting bigger and bigger even if I delete all of the sketches, is it possible that the indexed data is kept when the sketch is deleted?

You can now try the delete feature in the backend.

Does the released Docker image include the feature?