Purge static files on delete

[quaxsze]

No description provided.

[quaxsze]

After testing it is possible to create resource with distant link instead of local fs file, as well as an organization or reuse without a logo image. I added try except on delete or None verification.

[abulte]

Solid work 💪 A few comments below :p

There's one more case we need to handle I think: community resources files not associated to a dataset — this happens regularly because our creation process is f****. So we should:

• either strengthen the upload process: if a dataset is not selected, delete the file (not sure we can do that, it's pretty stateless)
• make a purge scheduled script for community resources files

[abulte]

[abulte]

maybe delete the resource from the index at this point, but totally optional.

[abulte]

💡 what about when we replace a ressource? not sure we handle that case.

[quaxsze]

Script creatinf indexes for deletion

def main():
    app = create_app()
    init_app(app)
    with app.app_context():
        print('Processing resources and community resources.')

        datasets = Dataset.objects()
        community_resources = CommunityResource.objects()
        with open('resource_fs.txt', 'w') as f:
            for dataset in datasets:
                for resource in dataset.resources:
                    if resource.url.startswith('https://static.data.gouv.fr'):
                        fs_name = resource.url[38:]
                        f.write(f'{fs_name}\n')

            for community_resource in community_resources:
                if community_resource.url.startswith('https://static.data.gouv.fr'):
                    fs_name = community_resource.url[38:]
                    f.write(f'{fs_name}\n')

        print('Processing organizations logos and users avatars.')
        orgs = Organization.objects()
        users = User.objects()
        with open('avatar_fs.txt', 'w') as f:
            for org in orgs:
                if org.logo.filename:
                    split_filename = org.logo.filename.split('.')
                    f.write(f'{split_filename[0]}\n')

            for user in users:
                if user.avatar.filename:
                    split_filename = user.avatar.filename.split('.')
                    f.write(f'{split_filename[0]}\n')

        print('Processing reuses logos.')
        reuses = Reuse.objects()
        with open('image_fs.txt', 'w') as f:
            for reuse in reuses:
                if reuse.image.filename:
                    split_filename = reuse.image.filename.split('.')
                    f.write(f'{split_filename[0]}\n')

        print('Completed.')


if __name__ == "__main__":
    main()

Deletion bash script

#!/bin/bash

resources_path="instance/fs/resources"
avatars_path="instance/fs/avatars"
images_path="instance/fs/images"

echo "Processing resources"
resources_counter=0
for f in $(find -type f "$resources_path")
do
    foo=${f#"$resources_path/"}
    if ! grep -Fxq "$foo" resource_fs.txt
    then
        echo $f >> resources_deletion.txt
        rm $f
        (( resources_counter++ ))
    fi
done

echo "Resources completed"
echo "Resources deletion: ${resources_counter}"

echo "Processing avatars"
avatars_counter=0
for f in $(find -type f "$avatars_path")
do
    match=0
    while read p
    do
        if  [[ $f == *$p* ]]
        then
            match=1
            break
        fi
    done < avatar_fs.txt
    if [ $match -eq 0 ]
    then
        echo $f >> avatars_deletion.txt
        rm $f
        (( avatars_counter++ ))
    fi
done
echo "Avatars completed"
echo "Avatars deletion: ${avatars_counter}"

echo "Processing images"
images_counter=0
for f in $(find -type f "$images_path")
do
    match=0
    while read p
    do
        if  [[ $f == *$p* ]]
        then
            match=1
            break
        fi
    done < image_fs.txt
    if [ $match -eq 0 ]
    then
        echo $f >> images_deletion.txt
        rm $f
        (( images_counter++ ))
    fi
done
echo "Images completed"
echo "Images deletion: ${images_counter}"

[quaxsze]

Resources completed
Resources deletion: 82783
Processing avatars
Avatars completed
Avatars deletion: 17854
Processing images
Images completed
Images deletion: 7420

real    29m53.011s
user    22m29.135s
sys     6m31.937s

[abulte]

Suggested change

	datasets = Dataset.objects()
	datasets = Dataset.objects().no_cache()

Using this may avoid the current migration failure on demo. Use it on every query set below too.