New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
added management command #912
Merged
Merged
Changes from all commits
Commits
Show all changes
29 commits
Select commit
Hold shift + click to select a range
6beb3a7
added management command
jrief f1fc437
Flake8 complained, used isort
jrief 2671d79
Use the storage.location as intended
jrief c358316
Merge remote-tracking branch 'origin/master' into feature/orphaned_files
jrief 63827c3
change interface according to @yakky’s annotations
jrief f720d7d
add management command ‘filer_check’ plus tests
jrief 0c0bb7b
Merge branch 'develop' into feature/orphaned_files
jrief 0681366
Use verbosity flag <3 to only render filenames
jrief 3af5929
Merge branch 'feature/orphaned_files' of github.com:jrief/django-file…
jrief 09b3984
fix indention
jrief 6424896
use ‘input’ from PY2/3 compatibility layer
jrief 47d8b68
isort complained for that extra line
jrief 0861042
Merge branch 'develop' into feature/orphaned_files
jrief 6f019a0
Merge remote-tracking branch 'origin/patch-1' into feature/orphaned_f…
jrief 1c968e8
Merge branch 'develop' into feature/orphaned_files
jrief 4cbfdb7
fix typos
jrief 4d635a6
remove translations from managment commands
jrief ea63a68
aborting a managment command does not raise an error
jrief 192fe45
Merge branch 'develop' into feature/orphaned_files
jrief c3f7954
fix typo
jrief 8743c29
Merge branch 'develop' into feature/orphaned_files
jrief f7b27f4
Merge remote-tracking branch 'divio/develop' into feature/orphaned_files
jrief 87c68c0
Merge branch 'feature/orphaned_files' of github.com:jrief/django-file…
jrief 33ebe23
Merge branch 'develop' into orphaned_files
jrief c40a0a5
fix flake8 complaint
jrief 95bbf60
fix isort complaints
jrief 1f54b5d
fix all tests regarding filer_check
jrief 16c9751
fix flake8 complaint
jrief 9a612be
Merge branch 'develop' into feature/orphaned_files
FinalAngel File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,118 @@ | ||
# -*- coding: utf-8 -*- | ||
from __future__ import unicode_literals | ||
|
||
import os | ||
|
||
from django.core.files.storage import DefaultStorage | ||
from django.core.management.base import BaseCommand | ||
from django.utils.module_loading import import_string | ||
from django.utils.six.moves import input | ||
|
||
from filer import settings as filer_settings | ||
|
||
|
||
class Command(BaseCommand): | ||
help = "Look for orphaned files in media folders." | ||
storage = DefaultStorage() | ||
prefix = filer_settings.FILER_STORAGES['public']['main']['UPLOAD_TO_PREFIX'] | ||
|
||
def add_arguments(self, parser): | ||
parser.add_argument( | ||
'--orphans', | ||
action='store_true', | ||
dest='orphans', | ||
default=False, | ||
help="Walk through the media folders and look for orphaned files.", | ||
) | ||
parser.add_argument( | ||
'--delete-orphans', | ||
action='store_true', | ||
dest='delete_orphans', | ||
default=False, | ||
help="Delete orphaned files from their media folders.", | ||
) | ||
parser.add_argument( | ||
'--missing', | ||
action='store_true', | ||
dest='missing', | ||
default=False, | ||
help="Verify media folders and report about missing files.", | ||
) | ||
parser.add_argument( | ||
'--delete-missing', | ||
action='store_true', | ||
dest='delete_missing', | ||
default=False, | ||
help="Delete references in database if files are missing in media folder.", | ||
) | ||
parser.add_argument( | ||
'--noinput', | ||
'--no-input', | ||
action='store_false', | ||
dest='interactive', | ||
default=True, | ||
help="Do NOT prompt the user for input of any kind." | ||
) | ||
|
||
def handle(self, *args, **options): | ||
if options['missing']: | ||
self.verify_references(options) | ||
if options['delete_missing']: | ||
if options['interactive']: | ||
msg = "\nThis will delete entries from your database. Are you sure you want to do this?\n\n" \ | ||
"Type 'yes' to continue, or 'no' to cancel: " | ||
if input(msg) != 'yes': | ||
self.stdout.write("Aborted: Delete missing file entries from database.") | ||
return | ||
self.verify_references(options) | ||
|
||
if options['orphans']: | ||
self.verify_storages(options) | ||
if options['delete_orphans']: | ||
if options['interactive']: | ||
msg = "\nThis will delete orphaned files from your storage. Are you sure you want to do this?\n\n" \ | ||
"Type 'yes' to continue, or 'no' to cancel: " | ||
if input(msg) != 'yes': | ||
self.stdout.write("Aborted: Delete orphaned files from storage.") | ||
return | ||
self.verify_storages(options) | ||
|
||
def verify_references(self, options): | ||
from filer.models.filemodels import File | ||
|
||
for file in File.objects.all(): | ||
if not file.file.storage.exists(file.file.name): | ||
if options['delete_missing']: | ||
file.delete() | ||
msg = "Delete missing file reference '{}/{}' from database." | ||
else: | ||
msg = "Referenced file '{}/{}' is missing in media folder." | ||
if options['verbosity'] > 2: | ||
self.stdout.write(msg.format(str(file.folder), str(file))) | ||
elif options['verbosity']: | ||
self.stdout.write(os.path.join(str(file.folder), str(file))) | ||
|
||
def verify_storages(self, options): | ||
from filer.models.filemodels import File | ||
|
||
def walk(prefix): | ||
child_dirs, files = storage.listdir(prefix) | ||
for filename in files: | ||
relfilename = os.path.join(prefix, filename) | ||
if not File.objects.filter(file=relfilename).exists(): | ||
if options['delete_orphans']: | ||
storage.delete(relfilename) | ||
msg = "Deleted orphaned file '{}'" | ||
else: | ||
msg = "Found orphaned file '{}'" | ||
if options['verbosity'] > 2: | ||
self.stdout.write(msg.format(relfilename)) | ||
elif options['verbosity']: | ||
self.stdout.write(relfilename) | ||
|
||
for child in child_dirs: | ||
walk(os.path.join(prefix, child)) | ||
|
||
filer_public = filer_settings.FILER_STORAGES['public']['main'] | ||
storage = import_string(filer_public['ENGINE'])() | ||
walk(filer_public['UPLOAD_TO_PREFIX']) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,73 @@ | ||
# -*- coding: utf-8 -*- | ||
from __future__ import absolute_import | ||
|
||
import os | ||
import shutil | ||
|
||
from django.core.files.uploadedfile import SimpleUploadedFile | ||
from django.core.management import call_command | ||
from django.test import TestCase | ||
from django.utils.module_loading import import_string | ||
from django.utils.six import StringIO | ||
|
||
from filer import settings as filer_settings | ||
from filer.models.filemodels import File | ||
|
||
from tests.helpers import create_image | ||
|
||
|
||
class FilerCheckTestCase(TestCase): | ||
def setUp(self): | ||
# ensure that filer_public directory is empty from previous tests | ||
storage = import_string(filer_settings.FILER_STORAGES['public']['main']['ENGINE'])() | ||
upload_to_prefix = filer_settings.FILER_STORAGES['public']['main']['UPLOAD_TO_PREFIX'] | ||
if storage.exists(upload_to_prefix): | ||
shutil.rmtree(storage.path(upload_to_prefix)) | ||
|
||
original_filename = 'testimage.jpg' | ||
file_obj = SimpleUploadedFile( | ||
name=original_filename, | ||
content=create_image().tobytes(), | ||
content_type='image/jpeg') | ||
self.filer_file = File.objects.create( | ||
file=file_obj, | ||
original_filename=original_filename) | ||
|
||
def tearDown(self): | ||
self.filer_file.delete() | ||
|
||
def test_delete_missing(self): | ||
out = StringIO() | ||
self.assertTrue(os.path.exists(self.filer_file.file.path)) | ||
file_pk = self.filer_file.id | ||
call_command('filer_check', stdout=out, missing=True) | ||
self.assertEqual('', out.getvalue()) | ||
|
||
os.remove(self.filer_file.file.path) | ||
call_command('filer_check', stdout=out, missing=True) | ||
self.assertEqual("None/testimage.jpg\n", out.getvalue()) | ||
self.assertIsInstance(File.objects.get(id=file_pk), File) | ||
|
||
call_command('filer_check', delete_missing=True, interactive=False, verbosity=0) | ||
with self.assertRaises(File.DoesNotExist): | ||
File.objects.get(id=file_pk) | ||
|
||
def test_delete_orphans(self): | ||
out = StringIO() | ||
self.assertTrue(os.path.exists(self.filer_file.file.path)) | ||
call_command('filer_check', stdout=out, orphans=True) | ||
# folder must be clean, free of orphans | ||
self.assertEqual('', out.getvalue()) | ||
|
||
# add an orphan file to our storage | ||
storage = import_string(filer_settings.FILER_STORAGES['public']['main']['ENGINE'])() | ||
filer_public = storage.path(filer_settings.FILER_STORAGES['public']['main']['UPLOAD_TO_PREFIX']) | ||
orphan_file = os.path.join(filer_public, 'hello.txt') | ||
with open(orphan_file, 'w') as fh: | ||
fh.write("I don't belong here!") | ||
call_command('filer_check', stdout=out, orphans=True) | ||
self.assertEqual("filer_public/hello.txt\n", out.getvalue()) | ||
self.assertTrue(os.path.exists(orphan_file)) | ||
|
||
call_command('filer_check', delete_orphans=True, interactive=False, verbosity=0) | ||
self.assertFalse(os.path.exists(orphan_file)) |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
management commands shouldn't mark strings for translation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed