OfflineGenerationError: You have offline compression enabled but key "%s" is missing from offline manifest.

[RemyAlves]

Hi guys,

I'm posting here because im out of idea, and tried almost everything. I'm trying to use django-compressor with S3BOTO to serve all my static and media file directly from S3. Everything almost work, when I do the offline compression I get all my files on S3, same with collect static, but then I get this famous error the missing key in the manifest. I tried to search on a lot of different forum, website, tutorial, but couldnt help myself, and we have to deliver soon.
Thank you for your help !

Settings.py :

"""
Django settings for Califun_travel_agent project.

Generated by 'django-admin startproject' using Django 1.9.7.

For more information on this file, see
https://docs.djangoproject.com/en/1.9/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.9/ref/settings/
"""

import os
import dj_database_url
#from boto.s3.connection import SubdomainCallingFormat

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
PROJECT_ROOT = os.path.dirname(os.path.abspath(__file__))

# Allow all host headers
ALLOWED_HOSTS = ['*']
FULL_DOMAIN = 'hidden'

# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.9/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'hidden'

# SECURITY WARNING: don't run with debug turned on in production!
if os.environ.get('DEBUG') == "False" :
    DEBUG = False
else :
    DEBUG = True

if os.environ.get('QBSAVE') == "True" :
    SAVE_IN_QUICKBOOKS = True
else :
    SAVE_IN_QUICKBOOKS = False

# Application definition

INSTALLED_APPS = [
    'public.apps.PublicConfig',
    'travelagent.apps.TravelagentConfig',
    'califun.apps.CalifunConfig',
    'product.apps.ProductConfig',
    'furnisher.apps.FurnisherConfig',
    'order.apps.OrderConfig',

    'compressor',
    'storages',

    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'whitenoise.runserver_nostatic',
    'django.contrib.staticfiles',
]

MIDDLEWARE_CLASSES = [
    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'Califun_travel_agent.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
                'public.navbar_context.navbar_context',
            ],
        },
    },
]

WSGI_APPLICATION = 'Califun_travel_agent.wsgi.application'


# Database
# https://docs.djangoproject.com/en/1.9/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}


# Password validation
# https://docs.djangoproject.com/en/1.9/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]


# Internationalization
# https://docs.djangoproject.com/en/1.9/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_L10N = True

USE_TZ = False


# Update database configuration with $DATABASE_URL.
db_from_env = dj_database_url.config(conn_max_age=500)
DATABASES['default'].update(db_from_env)

# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.9/howto/static-files/

STATIC_HOST = 'https://califuntravel.s3.amazonaws.com'
STATIC_URL = STATIC_HOST + '/'
STATIC_ROOT = os.path.join(PROJECT_ROOT, 'staticfiles')

STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',

    'compressor.finders.CompressorFinder',
)

COMPRESS_PRECOMPILERS = (
    ('text/x-scss', 'django_libsass.SassCompiler'),
)
COMPRESS_OFFLINE=True
COMPRESS_ENABLED=True

COMPRESS_PARSER = 'compressor.parser.HtmlParser'
COMPRESS_CSS_FILTERS = [
    'compress_filters.CustomCssAbsoluteFilter',
]
COMPRESS_ENABLED = True

# Extra places for collectstatic to find static files.
STATICFILES_DIRS = [
    os.path.join(PROJECT_ROOT, 'static'),
]

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = "hidden"
AWS_SECRET_ACCESS_KEY = "hidden"
AWS_STORAGE_BUCKET_NAME = "califuntravel"
#AWS_CALLING_FORMAT = SubdomainCallingFormat()
STATICFILES_STORAGE = 's3utils.CachedS3BotoStorage'

SESSION_EXPIRE_AT_BROWSER_CLOSE = True


EMAIL_BACKEND = 'django_mailgun.MailgunBackend'
MAILGUN_ACCESS_KEY = 'key-hidden'
MAILGUN_SERVER_NAME = 'hidden'


USE_S3 = True
if USE_S3:
    DEFAULT_FILE_STORAGE = 's3utils.MediaRootS3BotoStorage'
    THUMBNAIL_DEFAULT_STORAGE = 's3utils.MediaRootS3BotoStorage'
    MEDIA_URL = STATIC_HOST + '/media/'
    COMPRESS_STORAGE = 's3utils.CompressorS3BotoStorage'

s3utils.py :

from django.core.files.storage import get_storage_class
from storages.backends.s3boto import S3BotoStorage

class CachedS3BotoStorage(S3BotoStorage):
    def __init__(self, *args, **kwargs):
        super(CachedS3BotoStorage, self).__init__(*args, **kwargs)
        self.local_storage = get_storage_class(
            'compressor.storage.CompressorFileStorage')()

    def save(self, name, content):
        name = super(CachedS3BotoStorage, self).save(name, content)
        self.local_storage._save(name, content)
        return name

    # HERE is secret to dont generating multiple manifest.json and to delete manifest.json in Amazon S3
    def get_available_name(self, name):
        if self.exists(name):
            self.delete(name)
        return name


CompressorS3BotoStorage = lambda: CachedS3BotoStorage(location='compressor')
MediaRootS3BotoStorage = lambda: S3BotoStorage(location='media')

[karyon]

the only thing i can see is that you have STATICFILES_STORAGE and COMPRESS_STORAGE set to different values, the docs suggest to set it to the same.

apart from that, you could check whether the manifest.json exists in the place you'd expect it to and whether it contains anything useful at all.

if nothing else helps, you'd need to properly debug this i guess :(

[RemyAlves]

I tried but not very successfull. I get this thing : '%s' isn't accessible via COMPRESS_URL ('%s') and can't be compressed.

What do you mean by properly debug ?

[RemyAlves]

Yeah the manifest.json is good I think. When I do compress he upload it to S3, I have the same file on S3 and in my local staticfiles. The file look good too, he even have the empty compress block I've put on the page I'm trying to load.

{ "7249a26948744b0991f783fe71e11cab": "<link rel=\"stylesheet\" href=\"https://califuntravel.s3.amazonaws.com/CACHE/css/f297911ca140.css?Signature=wiS7w%2FY2iysvlbD5CON%2Fk4HUjjU%3D&Expires=1473362486&AWSAccessKeyId=AKIAJV5QIY4UWVSXCGTA\" type=\"text/css\" />", "2f8b7631fb4098c5a1232c42c82920e7": "<script type=\"text/javascript\" src=\"https://califuntravel.s3.amazonaws.com/CACHE/js/188194e5bfe5.js?Signature=RctM4zsyWoNjweNaOlB4HhYUQBU%3D&Expires=1473362486&AWSAccessKeyId=AKIAJV5QIY4UWVSXCGTA\"></script>", "fb83404c40c0548030c5c607aad09b20": "<link rel=\"stylesheet\" href=\"https://califuntravel.s3.amazonaws.com/CACHE/css/65f3b8915b6e.css?Signature=ZYOB%2FF3DZXD6tAVeOQmcmEcS9AA%3D&Expires=1473362487&AWSAccessKeyId=AKIAJV5QIY4UWVSXCGTA\" type=\"text/css\" />", "34a02411f0cf6a51c05e52294f33c116": "<script type=\"text/javascript\" src=\"https://califuntravel.s3.amazonaws.com/CACHE/js/5e3121bdd44e.js?Signature=RIGIoreRofobRgWzsnFq0U0ZM3w%3D&Expires=1473362487&AWSAccessKeyId=AKIAJV5QIY4UWVSXCGTA\"></script>", [...] "68b329da9893e34099c7d8ad5cb9c940": "", [...] }

[RemyAlves]

When I refresh the page with the error, the key change all the time. If I understood right it's not suppose to change at every refresh, only at every compress. Feel like he can't find the manifest...

[diox]

If the key is changing every time it probably means there is something dynamic in your templates that compress depends on (such as a variable or a method call inside {% compress %} blocks).

IIRC the fact that in your manifest URLs have expiry times (and not very far-future ones) is also an indicator that something is mis-configured but I haven't played with storages in a very long time.

[RemyAlves]

I removed ALL my compress tag except one on the page im trying to load. The page look like this :

{% load staticfiles %}
{% load compress %}

{% compress css %}
<link rel="stylesheet" type="text/x-scss" href="{% static 'califun/css/style.scss' %}" />
{% endcompress %}

[...]

The manifest look like this then :

{
  "37c4674d8679c48b926159f4904985b9": "<link rel=\"stylesheet\" href=\"https://califuntravel.s3.amazonaws.com/CACHE/css/25c77c744141.css?Signature=M2j4gTeNPJ6PrJ8gPnYF%2FebXiRc%3D&Expires=1473365110&AWSAccessKeyId=AKIAJV5QIY4UWVSXCGTA\" type=\"text/css\" />"
}

And the 25c file look like this :

/* line 7, /home/remy/Documents/Califun_travel_agent/Califun_travel_agent/califun/static/califun/css/style.scss */
.page_content.califun_admin_team .add_agent_button {
  margin-bottom: 20px; }
  /* line 11, /home/remy/Documents/Califun_travel_agent/Califun_travel_agent/califun/static/califun/css/style.scss */
  .page_content.califun_admin_team .add_agent_button:before {
    font-family: FontAwesome;
    content: '\f067';
    margin-right: 5px; }
[...]

They all both exist and look the same in local and in S3. Look like the compression go well. But then when loading the page he still ask me for random key.

[diox]

I can't see your HTML inside the {% compress %} blocks because of github overzealous HTML filtering. But again, my guess is that your storage class needs to be configured to output URLs that do not change, without expiry information. That will in turn make the compress block contents remain the same across requests, and fix offline compression key.

[RemyAlves]

Oh yeah I see what you mean now. That could be a probable cause. I'm trying to find a way to achieve that. I don't think it come from django compressor, more from S3Boto no ?

[diox]

Right.

[RemyAlves]

Do you have any idea where this can possibly happen ? Im digging in the compressor and boto source code but I really don't see at what point, and most important why, he change the url like that. I don't see anything on the web about it. Is this a S3 config ? Or a compressor / s3boto config ? Something like "DONT_TOUCH_MY_URL_PLEASE = True" ?

[RemyAlves]

AWS_QUERYSTRING_AUTH = False

Oh... my...
I need further testing, put everything back together, but look like perhaps it solved it. Will keep you updated.

[karyon]

any progress?

[RemyAlves]

Hey, sorry I had still some test to do but weekend. But of what I tested it work fine now, @diox were right. Thanks to both of you, I own you on this one. If you come around LA one day I would love to offer you a coffee or a beer, you deserved it !

[yspanchal]

@karyon @RemyAlves I am getting same error with django-storage and django-compressor==4.3.1
is there any way to make it working with AWS_QUERYSTRING_AUTH=True
S3 bucket is not public.