AttributeError: 'cStringIO.StringO' object has no attribute 'name' #100

Closed
amccloud opened this Issue Aug 8, 2011 · 23 comments

Comments

Projects
None yet

amccloud commented Aug 8, 2011

The example CachedS3BotoStorage raises this exception when I try to do collectstatic.

I'm using compressor@develop
Django@trunk

Traceback (most recent call last):
  File "manage.py", line 11, in <module>
    execute_manager(settings)
  File "/Users/andrew/Labs/something/src/django/django/core/management/__init__.py", line 442, in execute_manager
    utility.execute()
  File "/Users/andrew/Labs/something/src/django/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/andrew/Labs/something/src/django/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/andrew/Labs/something/src/django/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/Users/andrew/Labs/something/src/django/django/core/management/base.py", line 351, in handle
    return self.handle_noargs(**options)
  File "/Users/andrew/Labs/something/src/django/django/contrib/staticfiles/management/commands/collectstatic.py", line 116, in handle_noargs
    handler(path, prefixed_path, storage)
  File "/Users/andrew/Labs/something/src/django/django/contrib/staticfiles/management/commands/collectstatic.py", line 243, in copy_file
    self.storage.save(prefixed_path, source_file)
  File "/Users/andrew/Labs/something/something/../something/contrib/storages/__init__.py", line 11, in save
    self.local_storage._save(name, content)
  File "/Users/andrew/Labs/something/src/django/django/core/files/storage.py", line 198, in _save
    for chunk in content.chunks():
  File "/Users/andrew/Labs/something/src/django/django/core/files/base.py", line 65, in chunks
    counter = self.size
  File "/Users/andrew/Labs/something/src/django/django/core/files/base.py", line 39, in _get_size
    elif os.path.exists(self.file.name):
AttributeError: 'cStringIO.StringO' object has no attribute 'name'

The exact code for the backend is:

from django.core.files.storage import get_storage_class
from storages.backends.s3boto import S3BotoStorage

class CachedS3BotoStorage(S3BotoStorage):
    def __init__(self, *args, **kwargs):
        super(CachedS3BotoStorage, self).__init__(*args, **kwargs)
        self.local_storage = get_storage_class('compressor.storage.CompressorFileStorage')()

    def save(self, name, content):
        name = super(CachedS3BotoStorage, self).save(name, content)
        self.local_storage._save(name, content)
        return name

And my settings:

MEDIA_ROOT = os.path.join(PROJECT_DIR, 'media')
MEDIA_URL = '//something.s3.amazonaws.com/media/'

STATIC_ROOT = os.path.join(PROJECT_DIR, 'static')
STATIC_URL = '//something.s3.amazonaws.com/static/'

STATICFILES_DIRS = (
    os.path.join(PROJECT_DIR, 'static_media'),
)

STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
    'compressor.finders.CompressorFinder',
)

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_STORAGE = 'something.contrib.storages.CachedS3BotoStorage'

COMPRESS_STORAGE = STATICFILES_STORAGE
COMPRESS_OUTPUT_DIR = 'compressed'
COMPRESS_CSS_FILTERS = (
    'compressor.filters.css_default.CssAbsoluteFilter',
    'compressor.filters.cssmin.CSSMinFilter',
)
COMPRESS_JS_FILTERS = (
    'compressor.filters.jsmin.JSMinFilter',
)
Owner

jezdez commented Aug 19, 2011

Can you try it with the recent develop branch?

Owner

jezdez commented Sep 7, 2011

Okay, I'm going to close this since this is obviously a bug in the S3BotoStorageFile (https://bitbucket.org/david/django-storages/src/11bef0c2a410/storages/backends/s3boto.py#cl-285) which shouldn't return a cStringIO object but a ContentFile instead. Please file a bug report upstream.

jezdez closed this Sep 7, 2011

Hey I'm pretty sure this issue is not upstream. Even if ContentFile is used, you are still doing:

elif os.path.exists(self.file.name)

And here self.file will be a StringIO which has no attribute name indeed. Using a ContentFile does not change this.

I am having the same issue. Django-storages and s3boto work perfectly on their own for me. Only when I use CachedS3BotoStorage with AWS_IS_GZIPPED = True do I run into any trouble.

jezdez reopened this May 3, 2012

Owner

jezdez commented May 3, 2012

Turns out this is a problem in Django, not handling those file objects correctly, see https://code.djangoproject.com/changeset/17871 If anyone has a suggestion how to fix this for older Django installations, I'm listening.

Owner

jezdez commented May 3, 2012

So it seems like use something like this (backport of the changes mentioned above) when saving the content with the local storage:

from django.core.files.base import File

class ForgivingFile(File):

    def _get_size(self):
        if not hasattr(self, '_size'):
            if hasattr(self.file, 'size'):
                self._size = self.file.size
            elif hasattr(self.file, 'name') and os.path.exists(self.file.name):
                self._size = os.path.getsize(self.file.name)
            elif hasattr(self.file, 'tell') and hasattr(self.file, 'seek'):
                pos = self.file.tell()
                self.file.seek(0, os.SEEK_END)
                self._size = self.file.tell()
                self.file.seek(pos)
            else:
                raise AttributeError("Unable to determine the file's size.")
        return self._size

    def _set_size(self, size):
        self._size = size

    size = property(_get_size, _set_size)

    def chunks(self, chunk_size=None):
        """
        Read the file and yield chucks of ``chunk_size`` bytes (defaults to
        ``UploadedFile.DEFAULT_CHUNK_SIZE``).
        """
        if not chunk_size:
            chunk_size = self.DEFAULT_CHUNK_SIZE

        if hasattr(self, 'seek'):
            self.seek(0)

        while True:
            data = self.read(chunk_size)
            if not data:
                break
            yield data

Use it like this:

from django.core.files.storage import get_storage_class
from storages.backends.s3boto import S3BotoStorage

class CachedS3BotoStorage(S3BotoStorage):
    def __init__(self, *args, **kwargs):
        super(CachedS3BotoStorage, self).__init__(*args, **kwargs)
        self.local_storage = get_storage_class('compressor.storage.CompressorFileStorage')()

    def _save(self, name, content):
        content = ForgivingFile(content)
        name = super(CachedS3BotoStorage, self).save(name, content)
        self.local_storage._save(name, content)
        return name

stuross commented Jun 6, 2012

This worked for me, but you should be careful if you have gzip turned on. when you collectstatic it will put the gzip'd file in the static folder.

I'm using Django 1.3. This works partially for me because it cannot compress css files for some reason. JS files are combined and gzipped however. Do you have any idea?

saulshanabrook referenced this issue in jamstooks/django-s3-folder-storage Dec 30, 2012

Closed

Added caching to S3 storages #3

Contributor

ulyssesv commented Jan 3, 2013

I packed these fixes into http://pypi.python.org/pypi/django-cached-s3-storage if it helps.

nkeilar commented May 21, 2013

@stuross I know it was awhile back, but I'm running into the issue your eluded to above. When I have GZIP enabled then compressed files are saved to static folder, over the top of the original collect static files. Next time compressor runs it borks at the gibberish in the css or js files as they've been replaced by the compressed content. Did you have an idea for a work around?

Here is how I implemented it, hope this helps: http://pastebin.com/EGRFEPCg

nkeilar commented May 21, 2013

@noelph Thanks. Still no dice. Should gzipped files be uploaded with a .gz extension? I just ask because for me they aren't with the setup in jezdez#404.

Seems when with this type of backend

class CachedS3BotoStorage(S3BotoStorage):
    """
    S3 storage backend that saves the files locally, too.
    See http://django_compressor.readthedocs.org/en/latest/remote-storages/
    """
    def __init__(self, *args, **kwargs):
        super(CachedS3BotoStorage, self).__init__(*args, **kwargs)
        self.local_storage = get_storage_class(
            "compressor.storage.CompressorFileStorage")()

    def save(self, name, content):
        name = super(CachedS3BotoStorage, self).save(name, content)
        self.local_storage._save(name, content)
        return name

the compress files are saved in without the gz extension, both locally in cache/js/ or cache/css and in the s3 bucked at cache/js/ or cache/css. For some reason this then causes:

CommandError: An error occured during rendering /home/user/workspace/noshlyfs/templates/theme_bootstrap/less_base.html: UnicodeDecodeError while processing '/home/user/workspace/noshlyfs/static_root/cache/js/a1576bd8c653.js' with charset utf-8: 'utf8' codec can't decode byte 0x8b in position 1: invalid start byte

compressed

Yup, the extension is still "js" but the content encoding is gzip. Make sure you're using https://github.com/iserko/django-storages

BTW, I'm not pre-compressing them, haven't tried that. I simply do collectstatic, then restart the web server. The static files are automatically compressed and saved upon request of the page.

nkeilar commented May 22, 2013

@noelph thanks very much for confirming your setup. Turns out it was working, but just not using the compress --force which I think is a bug (see the issue I created earlier). Unfortunately for me I was assuming this would be able to test the setup, no so. Seems that command is accessing the local compressed files after compressing them, then borking because they are no longer utf8.

At the very least we made it easier for the next reader 👍 Thanks.

nkeilar commented May 22, 2013

@noelph also i followed the breadcrumbs about the diff with https://github.com/iserko/django-storages as far as I could tell the issue has been resolved upstream now. :)

Good to know its working. Thanks for the update re django-storages.

Owner

jezdez commented May 27, 2013

Can I close this then?

nkeilar commented May 28, 2013

Well, my issue was mostly resolved - still seems unstable, but that might be in part due to cache with cloud-front. While I was primarily concerned with GZIP I might have hijacked this thread.

We are using Django 1.4 in one of our projects with S3, django-storages and django-compressor. Recently we decided to use Gzip compression too for JS and CSS. And got into the same issue, but only when running 'collectstatic' and not when running 'compress' (offline compression).

Ie, If I set AWS_IS_GZIPPED = False then run 'collectstatic' static files would be collected (without gzip) and then set AWS_IS_GZIPPED = True then run 'compress' those collected files would be compresses and gzipped without any error. But obviously no on want to change their settings every time they run 'compress', So I and using the following class as my compressed storage:

class GzippedS3BotoStorage(storages.backends.s3boto.S3BotoStorage):
    gzip = True

And my storage settings are:

STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
COMPRESS_STORAGE = '<some_module>.GzippedS3BotoStorage'

Haven't tested this with online compression, but I think would work with that too.

diox removed the bug label May 25, 2014

Contributor

karyon commented Sep 25, 2015

the issue @jezdez identified as the root cause here #100 (comment) was fixed in django 1.5. since 1.4 is nearing EOL, i vote for closing this.

Owner

diox commented Sep 25, 2015

Agreed. We're about to remove official support for 1.4 (since it's nearing EOL) in compressor and there are enough workarounds available anyway.

diox closed this Sep 25, 2015

@iamwithnail iamwithnail pushed a commit to iamwithnail/treatout that referenced this issue Mar 9, 2016

Chris Update files to reflect where collect static has put them; issue with…
… running it on production settings, as uses django_storages, with problem. related django-compressor/django-compressor#100
ced3dea
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment