Duplicate thumbnail files created

[etj]

When creating a dataset, 2 thumbnail files are created, e.g.:

/mnt/volumes/statics/uploaded/thumbs/dataset-3b05df02-9f97-41b4-a425-864ec2e87dfb-thumb-ab219ad8-dd53-4d93-b078-4c4eaa31d7c7_JSXE9QL.jpg
/mnt/volumes/statics/uploaded/thumbs/dataset-3b05df02-9f97-41b4-a425-864ec2e87dfb-thumb-ab219ad8-dd53-4d93-b078-4c4eaa31d7c7.jpg

These are the content of the thumbnail related fields in the db:

• thumbnail_url: http://localhost/uploaded/thumbs/dataset-3b05df02-9f97-41b4-a425-864ec2e87dfb-thumb-ab219ad8-dd53-4d93-b078-4c4eaa31d7c7.jpg
• thumbnail_path: thumbs/dataset-3b05df02-9f97-41b4-a425-864ec2e87dfb-thumb-ab219ad8-dd53-4d93-b078-4c4eaa31d7c7.jpg

This seems to be a longstanding issue in GeoNode.

[mattiagiupponi]

@sijandh35 take a 2 hours timebox for the investigation

[sijandh35]

Hi @mattiagiupponi @etj

I checked why this is happening.

Currently in save_thumbnail, a file is saved — here, here, and here:

1. The raw image is first stored via the storage manager.
2. im.save() writes the optimized image to a local temp path, but since that path is the same as the stored file, on local storage it overwrites the raw one in place with the optimized one.
3. The storage manager saves again to the same path. Since the path is now taken, the backend appends a random suffix — which is the extra ..._XXXXXXX.jpg file we see.

So, due to this reason we can see the 2 duplicated thumbnail files.

Proposed solution

Instead of saving multiple times, we can:

1. Read the image from memory.
2. Optimize the image.
3. Save it once with the storage manager.

I can open a PR for this if it looks good to you.

[mattiagiupponi]

Hi @sijandh35
Two points:

1. save in memory can be a problem at the moment there are a different users uploading data in GeoNode, don't we risk to go in OOO?
2. If the img.save() already saves in the right spot, can't we just make it obvious and avoid the point3? or just make the image.save() save under the /tmp path and then move it in the right place?

Before proceeding, how much is the estimation on this?

[sijandh35]

1. save in memory can be a problem at the moment there are a different users uploading data in GeoNode, don't we risk to go in OOO?

Hi @mattiagiupponi , my plan is to reuse the existing image variable which is on memory already, optimize it, and write it to a temp path like you suggested, then do a single storage_manager.save() at the end.

2. If the img.save() already saves in the right spot, can't we just make it obvious and avoid the point3? or just make the image.save() save under the /tmp path and then move it in the right place?

img.save() stores the image locally, so I think point 3 (the storage manager) was kept to also handle the remote storage case.

Before proceeding, how much is the estimation on this?

Estimation: 2-3 hrs.

[mattiagiupponi]

Ok sijan proceed