Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

filebrowser_safe does not work with Amazon S3 (and presumably other services of the same kind) #530

Closed
mrlundis opened this Issue · 18 comments

9 participants

@mrlundis

Trying to open the Media Library fails with the following exception when assets are hosted externally (in this case using django-storages on S3):

Stacktrace (most recent call last):
  File "mezzanine/pages/middleware.py", line 45, in process_view
    return view_func(request, *view_args, **view_kwargs)
  File "filebrowser_safe/views.py", line 76, in browse
    raise ImproperlyConfigured, _("Error finding Upload-Folder. Maybe it does not exist?")

This is a major issue when running Mezzanine on a PaaS like Heroku where persistent file system access is limited or completely unavailable.

The issue has already been discussed throughly here:
https://groups.google.com/d/topic/mezzanine-users/kjsAw6WFqMY/discussion
and here: sehmaschine/django-filebrowser#40

There is already a fix in the upstream version of django-filwbrowser, although some other minor issues with thumbnail generation still remain.

If the fix cannot easily be incorporated in the existing fork, maybe it's time to consider abandoning the *_safe-forks and use the newer versions of grappelli and filebrowser instead. Doing so would have the added benefit of

  • Not having to maintain these packages separately and instead focus on the core of Mezzanine,
  • Maybe fix some existing issues (I'm thinking #62)
  • Gaining access to the progress and other improvements made since forking. Grappelli is looking really good now...

Keep up the good work!

@stephenmcd
Owner

Until this is resolved, I just thought I'd point out in the thread you linked to, there's a work-around described:

"I've solved the ImproperlyConfigured: Error finding Upload-Folder. you have to collectstatic to upload to the bucket and check the directory (media/uploads) is declared public!"

@natea

I made the media/uploads in my S3 bucket public, but I'm still getting this error:

Environment:


Request Method: GET
Request URL: http://www.coderaising.org/admin/media-library/browse/

Django Version: 1.4.3
Python Version: 2.7.3
Installed Applications:
['mezzanine.boot',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.redirects',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.sitemaps',
 'django.contrib.staticfiles',
 'mezzanine.conf',
 'mezzanine.core',
 'mezzanine.generic',
 'mezzanine.blog',
 'mezzanine.forms',
 'mezzanine.pages',
 'mezzanine.galleries',
 'mezzanine.twitter',
 'mezzanine.accounts',
 'mdown',
 'gunicorn',
 'storages',
 'debug_toolbar',
 'filebrowser_safe',
 'south',
 'grappelli_safe',
 'django.contrib.admin',
 'django.contrib.comments']
Installed Middleware:
['debug_toolbar.middleware.DebugToolbarMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.redirects.middleware.RedirectFallbackMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'mezzanine.core.request.CurrentRequestMiddleware',
 'mezzanine.core.middleware.TemplateForDeviceMiddleware',
 'mezzanine.core.middleware.TemplateForHostMiddleware',
 'mezzanine.core.middleware.AdminLoginInterfaceSelectorMiddleware',
 'mezzanine.core.middleware.SitePermissionMiddleware',
 'mezzanine.pages.middleware.PageMiddleware']


Traceback:
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  111.                         response = callback(request, *callback_args, **callback_kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/admin/views/decorators.py" in _checklogin
  16.             return view_func(request, *args, **kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  89.         response = view_func(request, *args, **kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/filebrowser_safe/views.py" in browse
  76.             raise ImproperlyConfigured, _("Error finding Upload-Folder. Maybe it does not exist?")

Exception Type: ImproperlyConfigured at /admin/media-library/browse/
Exception Value: Error finding Upload-Folder. Maybe it does not exist?

Here you can see that I made both 'media' and 'uploads' public.

Screen Shot 2013-02-04 at 6 46 50 PM

@natea

Here's what I've set up for the S3 storage in my settings.py:

    DEFAULT_FILE_STORAGE = "storages.backends.s3boto.S3BotoStorage"
    STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

    AWS_ACCESS_KEY_ID = os.environ.get("AWS_ACCESS_KEY_ID", "")
    AWS_SECRET_ACCESS_KEY = os.environ.get("AWS_SECRET_ACCESS_KEY", "")
    AWS_STORAGE_BUCKET_NAME = os.environ.get("AWS_STORAGE_BUCKET_NAME", "")

    AWS_HEADERS = {
        "Cache-Control": "public, max-age=86400",
    }

    AWS_S3_FILE_OVERWRITE = False
    AWS_QUERYSTRING_AUTH = False
    AWS_S3_SECURE_URLS = False
    AWS_REDUCED_REDUNDANCY = False
    AWS_IS_GZIPPED = False

    STATIC_URL = 'http://' + AWS_STORAGE_BUCKET_NAME + '.s3.amazonaws.com/'
    MEDIA_URL = STATIC_URL + 'media/'
    ADMIN_MEDIA_PREFIX = STATIC_URL + 'grappelli/'
@shurik

Hi all,

A solution I've been using is to uninstall Filebrowser altogether. Mezzanine appears to simply revert to using Django's stock FileField and ImageField and so works fine with S3. I recently gave a presentation and put together an example repo on GitHub.

https://github.com/shurik/freesnakes

Would love to hear criticisms or ideas, bug reports about this approach.

Cheers,

Sasha

@mrlundis

I also tried to make the uploads dir public without success. Will give the workaround by @shurik a shot though, the remaining functionality would still be sufficient for my needs.

Thanks for sharing Sasha!

@synasius

Any news on this issue? I found a workaround to bypass "Error finding Upload-Folder" which is to create an uploads folder on the root of my S3 bucket (not in the media folder)... now I can upload files but I got problem on thumbnails generation..

@stephenmcd
Owner

Good thread recently covering some of the S3 issues and solutions:

https://groups.google.com/group/mezzanine-users/browse_thread/thread/904ef36615ad5ff

@littleq0903

to @synasius:

I also have the thumbnails generation problems, I'm trying deploy my mezzanine onto GAE, I think the possible problem is mezzanine do the trick by using "os.makedirs" directly, but not go through your Storage module, so the folder is created on your server but not S3, then cause this problem.

possible workaround should be use Storage functions instead of os.makedirs

on GAE it's worse, GAE doesn't provide the file operation functions under os.

@natea

As per this discussion, the code that does the thumbnail creation assumes that it has access to filesystem which is not the case with S3 and GAE.

The thumbnail generation on line 327 (of latest version in GitHub at time of writing) of mezzanine_tags.py assumes access to the local file system:

 ImageFile.MAXBLOCK = image.size[0] * image.size[1]
    try:
        image = ImageOps.fit(image, (width, height), Image.ANTIALIAS)
        image = image.save(thumb_path, filetype, quality=quality, **image_info)
        # Push a remote copy of the thumbnail if MEDIA_URL is
        # absolute.
        if "://" in settings.MEDIA_URL:
            with open(thumb_path, "r") as f:
                default_storage.save(thumb_url, File(f))
    except Exception:
        # If an error occurred, a corrupted image may have been saved,
        # so remove it, otherwise the check for it existing will just
        # return the corrupted image next time it's requested.
        try:
            os.remove(thumb_path)
        except Exception:
            pass
        return image_url
    return thumb_url

In particular this line:

image = image.save(thumb_path, filetype, quality=quality, **image_info)

The solution as described in the post is this:

    thumb_io = StringIO.StringIO()
    img.save(thumb_io, format='JPEG')

    # Create a new Django file-like object to be used in models as ImageField using
    # InMemoryUploadedFile.  If you look at the source in Django, a
    # SimpleUploadedFile is essentially instantiated similarly to what is shown here
    thumb_file = InMemoryUploadedFile(thumb_io, None, 'foo.jpg', 'image/jpg',
                                      thumb_io.len, None)

This allows 'saving' the result of PIL operations to an in memory file, ready to be saved straight to one's default storage, without the need to store it locally first. Is that something that could be put into Mezzanine?

@stephenmcd
Owner

@natea yep I think that's correct.

So long as it doesn't regress in any way with local storage backends I'm keen to get this in.

If anyone wants to get this working and tested with a pull request that'd be awesome.

@natea
@littleq0903

@stephenmcd yeah, I will start working on that after passed the midterm exam, it's on Tue.

But I'm not really using Mezzanine so maybe I will need some help or advise from you to avoid doing thing in the wrong way.

Just wanna get this awesome framework working on PaaS.

@synasius

After reading several articles on this issue I can say the following settings works for me on Heroku:

DEFAULT_FILE_STORAGE = "storages.backends.s3boto.S3BotoStorage"
STATICFILES_STORAGE = 's3utils.StaticS3Backend'

STATIC_URL = 'http://' + AWS_STORAGE_BUCKET_NAME + '.s3.amazonaws.com/static/'
STATIC_ROOT = STATIC_URL

MEDIA_URL = 'http://' + AWS_STORAGE_BUCKET_NAME + '.s3.amazonaws.com/'
MEDIA_ROOT = ''

ADMIN_MEDIA_PREFIX = STATIC_URL + 'grappelli/'

You have to define StaticS3Backend in a s3utils.py module at root of your mezzanine project.

from storages.backends.s3boto import S3BotoStorage

class StaticS3Backend(S3BotoStorage):
    location = 'static'
@madteckhead

Did this make it into any version yet? Just running into the thumbnail issue now. These may be of interest:
sehmaschine/django-filebrowser#103
sehmaschine/django-filebrowser#40

@dangayle

@synasius Does that fix the thumbnail issue also?

@madteckhead
@stephenmcd stephenmcd closed this
@river-jade

I got this working (ie filebrowser-safe and django-storages on s3) with two small changes to each package:

stephenmcd/filebrowser-safe#37
https://bitbucket.org/david/django-storages/pull-request/96/if-entrylast_modified-is-none-set-to/diff

Static files, media uploads and thumbnails all work perfectly.

@stephenmcd
Owner

Merged now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.