1.4.2 requires absolute paths support from static storage

[rjagielski]

I'm using the following static storage:

class S3PipelineStorage(PipelineMixin, S3BotoStorage):
    """S3 storage with pipeline magic"""
    pass

Which is almost the same as the one from pipeline docs (even simpler because it doesnt use CachedFilesMixin) .
It's fine with 1.3.x but breaks with 1.4.2 because it tries to run sass compiler using absolute path from s3. S3BotoStorage doesn't support absolute paths and throws this long exception:

/home/bender/myapps/foo/manage.py in <module>()
      8     from django.core.management import execute_from_command_line
      9 
---> 10     execute_from_command_line(sys.argv)

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/core/management/__init__.pyc in execute_from_command_line(argv)
    383     """
    384     utility = ManagementUtility(argv)
--> 385     utility.execute()

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/core/management/__init__.pyc in execute(self)
    375             sys.stdout.write(self.main_help_text() + '\n')
    376         else:
--> 377             self.fetch_command(subcommand).run_from_argv(self.argv)
    378 
    379 

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/core/management/base.pyc in run_from_argv(self, argv)
    286         handle_default_options(options)
    287         try:
--> 288             self.execute(*args, **options.__dict__)
    289         except Exception as e:
    290             if options.traceback or not isinstance(e, CommandError):

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/core/management/base.pyc in execute(self, *args, **options)
    336                     not options.get('skip_checks')):
    337                 self.check()
--> 338             output = self.handle(*args, **options)
    339             if output:
    340                 if self.output_transaction:

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/core/management/base.pyc in handle(self, *args, **options)
    531         if args:
    532             raise CommandError("Command doesn't accept any arguments")
--> 533         return self.handle_noargs(**options)
    534 
    535     def handle_noargs(self, **options):

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.pyc in handle_noargs(self, **options)
    166             raise CommandError("Collecting static files cancelled.")
    167 
--> 168         collected = self.collect()
    169         modified_count = len(collected['modified'])
    170         unmodified_count = len(collected['unmodified'])

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.pyc in collect(self)
    112             processor = self.storage.post_process(found_files,
    113                                                   dry_run=self.dry_run)
--> 114             for original_path, processed_path, processed in processor:
    115                 if isinstance(processed, Exception):
    116                     self.stderr.write("Post-processing '%s' failed!" % original_path)
/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/pipeline/storage.pyc in post_process(self, paths, dry_run, **options)
     28         from pipeline.packager import Packager
     29         packager = Packager(storage=self)
---> 30         for package_name in packager.packages['css']:
     31             package = packager.package_for('css', package_name)
     32             output_file = package.output_filename

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/pipeline/packager.pyc in pack_stylesheets(self, package, **kwargs)
     94         return self.pack(package, self.compressor.compress_css, css_compressed,
     95                          output_filename=package.output_filename,
---> 96                          variant=package.variant, **kwargs)
     97 
     98     def compile(self, paths, force=False):

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/pipeline/packager.pyc in pack(self, package, compress, signal, **kwargs)
    103         if self.verbose:
    104             print("Saving: %s" % output_filename)
--> 105         paths = self.compile(package.paths, force=True)
    106         content = compress(paths, **kwargs)
    107         self.save_file(output_filename, content)

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/pipeline/packager.pyc in compile(self, paths, force)
     97 
     98     def compile(self, paths, force=False):
---> 99         return self.compiler.compile(paths, force=force)
    100 
    101     def pack(self, package, compress, signal, **kwargs):

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/pipeline/compilers/__init__.pyc in compile(self, paths, force)
     54         else:
     55             with futures.ThreadPoolExecutor(max_workers=multiprocessing.cpu_count()) as executor:
---> 56                 return list(executor.map(_compile, paths))
     57 
     58     def output_path(self, path, extension):

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/concurrent/futures/_base.pyc in map(self, fn, *iterables, **kwargs)
    578             for future in fs:
    579                 if timeout is None:
--> 580                     yield future.result()
    581                 else:
    582                     yield future.result(end_time - time.time())

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/concurrent/futures/_base.pyc in result(self, timeout)
    398                 raise CancelledError()
    399             elif self._state == FINISHED:
--> 400                 return self.__get_result()
    401 
    402             self._condition.wait(timeout)

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/concurrent/futures/_base.pyc in __get_result(self)
    357     def __get_result(self):
    358         if self._exception:
--> 359             reraise(self._exception, self._traceback)
    360         else:
    361             return self._result

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/concurrent/futures/_compat.pyc in reraise(exc, traceback)
    105     def reraise(exc, traceback):
    106         locals_ = {'exc_type': type(exc), 'exc_value': exc, 'traceback': traceback}
--> 107         exec('raise exc_type, exc_value, traceback', {}, locals_)
/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/concurrent/futures/thread.pyc in run(self)
     59 
     60         try:
---> 61             result = self.fn(*self.args, **self.kwargs)
     62         except BaseException:
     63             e, tb = sys.exc_info()[1:]

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/pipeline/compilers/__init__.pyc in _compile(input_path)
     34                 if compiler.match_file(input_path):
     35                     output_path = self.output_path(input_path, compiler.output_extension)
---> 36                     infile = self.storage.path(input_path)
     37                     outfile = self.output_path(infile, compiler.output_extension)
     38                     outdated = compiler.is_outdated(input_path, output_path)

/home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/core/files/storage.pyc in path(self, name)
     85         accessed using open() should *not* implement this method.
     86         """
---> 87         raise NotImplementedError("This backend doesn't support absolute paths.")
     88 
     89     # The following methods form the public API for storage systems, but with

NotImplementedError: This backend doesn't support absolute paths

[ciokan]

+1 on this

[brad]

@bender314 @ciokan I think I've got a fix for this (see #409). It seems to be working well for us. If you could try it out yourself, that would be helpful.

[hwkns]

@brad for me, this fix only delays the error; after copying all the static files to S3, the local .scss file gets compiled into a .css file in the same directory on the local filesystem, instead of compiling the .scss file in S3 to a new .css file in S3. Then, during the concatenate_and_rewrite step, the .css file is not found on S3 and I hit an exception thrown by the storage backend (storages/backends/s3boto.py):

OSError: File does not exist: admin/OpenLayers/theme/default/style.css

[brad]

Strange, we are not experiencing that, though we are using Less, not Sass

[lundberg]

Experiencing the exact same problem using mixins mentioned in docs...

class S3PipelineStorage(PipelineMixin, CachedFilesMixin, S3BotoStorage):
    pass

After debugging, the real problem is that the configured compiler(s) gets called with infile/outfile params as storage related paths, looked up using storage.path() here

So, when not using local file storage, like S3, the paths used by the compiler, i.e. LessCompiler, the paths are storage specific and can't be found by the local less compiler cli command.

Changing path lookup with finders.find() instead of storage.path() like in #409 does not solve the problem later in the chain as @hwkns points out. It only solves the paths for the local cli command.

For other chained mixins, like CachedFilesMixin, to not break and work as expected, the compiled outfile has to be able to be found by the storage.

As far as I can see this can be solved in two ways.

1.
Making it possible configure a Package storage other than self when instantiating here
and then pass along the same storage with the paths here.
Then the compiler should be able to use local paths for the cli command, and later mixins will respect and use related/passed storage for that outfile when reading content.

2.
Create a new CompilerBase.compile() method that instead determines the infile/outfile paths and not in the Compiler.compile._compile. This allows you to override that method in a custom compiler implementation and use finders.find() to get the paths right for the cli compiler.
But, by using this approach, later mixins wont find the compiled file using the storage and you have to save the compiled outfile output using self.storage (S3) last in your compler.compile_file. Maybe implement a compiler.post_process could be a nice solution to handle the storage.save part.

[devangmundhra]

+1 for same error

[cyberdelia]

Fixed in #409.

[hwkns]

@cyberdelia did you read the comments here from myself and @lundberg ?

[lundberg]

@hwkns I've made a pull request that implements some of my idéas in commented solution 2. Could you try it and see if it solves even your problem?

@cyberdelia I'm still using your partial fix from #409 in respect of previous merge.

[hwkns]

We move fast and this was holding us back, so we use Gulp for static assets now instead of django-pipeline. If I can find the time, I will roll back to an old commit and try to test this out, but it looks good to me @lundberg.

[adambratt]

Yeah this issue is killing us. Thinking about switching off django-pipeline and over to grunt or gulp as well.

I can't believe there aren't any tests for this as I know quite a few people using S3 with their django static assets.

[rjagielski]

Everything works fine again with pipeline 1.5 (tested with django 1.8). Closing.