Okay I really don't know what else to do...
Everything seems to be correct, and works perfectly on development
but on production the images are all broken.
I have this on my Gemfile
gem 'rails', :git => 'https://github.com/rails/rails.git', :branch => '3-1-stable'
gem "sprockets", "2.0.0.beta.10"
<%= image_tag( 'categories/default_admin_thumb.png' ) %>
On my log
Started GET "/assets/categories/default_admin_thumb-f3b01ba52d3e16ea3f7393d751ccaaf7.png" for 127.0.0.1 at 2011-06-22 13:53:42 -0300
Served asset /categories/default_admin_thumb-f3b01ba52d3e16ea3f7393d751ccaaf7.png - 200 OK (1ms) (pid 54202)
On a curl -I request
HTTP/1.1 200 OK
Cache-Control: public, max-age=31536000
Last-Modified: Wed, 15 Jun 2011 17:53:11 GMT
Date: Wed, 22 Jun 2011 16:48:36 GMT
Server: WEBrick/1.3.1 (Ruby/1.9.2/2011-02-18)
Apparently content-length is zero. Please I could even try and fix it, but I really don't even know where.
Also, I tried with gem "sprockets", "2.0.0.beta.10" and gem "sprockets", "2.0.0.beta.11"
gem "sprockets", "2.0.0.beta.11"
+1 having the same issues on ubuntu production server
I'm also seeing this. It has to do with config.action_controller.perform_caching. Setting that to false in production.rb fixes the problem, but obviously not a long term solution. Still digging to find out why this happens.
+1, same issue here. I really can't figure out what is causing this. Also, setting config.action_controller.perform_caching = false didn't fix the issue for me.
I have another application using a very similar configuration (same gem versions, except for a few unrelated ones such as devise), which is working completely fine in production, even with caching set to true. I've been comparing both apps but I just can't find any differences. I put the other application up for production a few days ago, so it must be something that broke in the current rails code since then.
Anyone have any luck finding this? I think it's probably related to this sprockets issue but not sure:
I just updated my app with the latest 3-1-stable/beta.10 and it seems to be working just fine now
Can someone confirm this?
Yes, it's working for me as well.
Apparently some commit on 3-1-stable resolved this issue.
Hmm, I'm still seeing the issue. Would you mind posting the relevant portion of your Gemfile for comparison?
Hm, I haven't changed anything in my gemfile.
gem 'rails', :git => "git://github.com/rails/rails.git", :branch => "3-1-stable"
I've switched from webrick to thin, not sure if that's in any way related to the issue.
Mine has the same rails reference, but gem "sprockets", "2.0.0.beta.11" on the Gemfile too
@markdillon Please let me know if it works with sprockets beta10 and rails 3-1-stable
I upgraded an existing app to RC4 and images work fine. Moving the app up to 3-1-stable (still sprockets beta.10) all images have 0 length.
I too am still seeing this issue with both rc4 and edge using sprockets beta 10. I can't use sprockets beta 11 as it has not been published and therefore can't be found by bundler. I tried using it using :git and :branch but the dependencies don't allow me to activate it.
OK, I have created a new Rails app off 3.1.0.rc4. Images seem to work fine. Updating to 3-1-stable and updating the Gems things still seem to run fine.
All very vexing as this suggests something is interacting with code in our existing apps (yours were upgrades??). l've ready spent a couple of days on this, and running out of ideas.
Mine is an upgrade from 3.0.3 if i remember correctly (havent got access to the stuff right now). There is no way I rebuild my rails app from scratch for this release.
mark, you can do gem 'sprockets', '= 2.0.0.beta.11', :git => 'git://github.com/sstephenson/sprockets.git'
Sadly, you cannot do this as Rails 3-1-stable depends on sprockets (= 2.0.0.beta.10), and it won't bundle. :-(
Oh, there is a rails 3.1 stable? That is newer then rc4? Thats's what I'm using with sprockets 2.0.0.beta.11. Although I do also have some issues with the asset pipeline... I haven't got it working yet with heroku, ruby 1.9.2, and the cedar stack.
It doesn't even work in development for me... Without precompiling nothing works. And even the precompile task sometimes doesn't do anything...
Hmm, the stable branch is still on rc4. Maybe there were some recent changes that break
gem 'sprockets', '= 2.0.0.beta.11', :git => 'git://github.com/sstephenson/sprockets.git'
Yep, but only on upgraded apps as far as I can tell. Yours is an upgraded app?
Haha tripple post.. anyway. If i chose the stable branch it basically installs rails 3.1.rc4... I havent tried it with a new app, I will do that tomorrow. Obviously it must work otherwise it wouldnt be in the rc. But as long as there isnt a proper way of upgrading from a 3.0.x version I cant really see why this would be rc..
This shouldn't be an issue using Rails 3-1-stable and the latest sprockets gem, if you still see something wrong please add a comment and I will reopen
Adding a comment then!
A new app seems to work fine. I have upgraded an existing app and image serving broke between RC4 and 3-1-stable (both Sprockets beta.10). I suspect some interaction with a Gem and other stuff in the app. I ran out of ideas and time, but will look again if you suggest where to look.
Others in this thread appear to have the same problem - existing apps break - and it would be good to get the bottom of it before the final release.
@markdillon @desaperados Are these upgraded apps and if so from which version/s?
Mine is from late 2.x -> 3.0 -> 3.1.
Perhaps there is some config or initialization missing somewhere?
Mine is upgraded from 3.0.X to 3.1.X. I've also tested with a freshly generated app and am having the same problem.
OK, I think I might have found the problem - a stale cache.
I have deleted the cache directories in rails_root/tmp/cache and it now works. In my case I had been running in production on my dev machine while I was investigating the behavior of the asset pipeline for docrails. I have been upgrading with each RC.
At this stage I suspect a change in the cache file format (no proof yet), or some other cache mischief.
Can I suggest that you look in tmp/cache, remove any directories you have in there, and report back.
If you have upgraded through several RCs (and Sprockets versions) this might support the theory, but I can hardly be bothered going to look at the code changes with the amount of time I have spent on this. :-)
@spastorino Is there a chance that the format of the cache files changed? Will this likely be an issue for upgraders if they happen to have been on edge (and have and existing cache), and then upgrade to a later RC?
+1 - If I turn off action_controller caching via: config.action_controller.perform_caching = false
my assets work fine, otherwise they will not.
Running locally in production mode and tried on heroku.
I just discovered exactly the same. The asset pipeline broke horribly for me when caching was on. I found out the issue had todo with webrick. Switching to thin works fine here and on Heroku (Cedar stack).
Hmm still no cake for me while using thin. :(
Uh, okay. Hmm, what is the error?
Not sure if it helps, I have gem 'rails', '3.1.0.rc4' in my gemfile and no sprockets.
I've tried every permutation that was outlined in this thread to fix it.. still not working.
Basically, I'm trying to get CKEditor WYSIWUG editor working. It's packaged as a rails engine so that it's automatically in the assets compilation.
GET http://s3.amazonaws.com/rccl_roduction/logos/3/thumb/club_logo.gif?1309989920 403 (Forbidden)
config.jsGET http://localhost:3000/member/clubs/3/config.js?t=B5GJ5GG 404 (Not Found)
editor.cssGET http://localhost:3000/member/clubs/3/skins/kama/editor.css?t=B5GJ5GG 404 (Not Found)
en.jsGET http://localhost:3000/member/clubs/3/lang/en.js?t=B5GJ5GG 404 (Not Found)
Like I said previously, If i turn off controller_caching it works fine.
I think im about ready to find another solution or something, this is taking way to much time
You could try to have CKEditor in your public folder, just like traditionally. I'm using it also in the project I'm currently upgrading, but havn't gotten around trying it out with the rails 3.1 setup. So far the ckeditor folder just sits in public.
@ncri thanks alot for your help.. I will try now.
Btw.. are you using CKeditor as the gem via gem 'ckeditor-rails'
are you just using it normally without the gem
I'm not using the gem so far, wasn't aware there is one before you mentioned it. looks quite interesting!
I am pretty sure that this is the commit that caused this to fail:
Issue 1761 documents the issues this patch alleges to fix, and how the commit is tied in with the pipeline.
+1 Issue occurs on deployment to heroku. Tried all suggested solutions but still failing.
@salbito: what stack are you on heroku and which ruby version do you use?
The issue still shows up in the latest Rails 3.1 stable branch for me. Running Ruby 1.9.2 with :branch => '3-1-stable' in my Gemfile. I didn't add the sprockets beta gem as it seems to be a dependency of actionpack.
:branch => '3-1-stable'
I don't see the problem on Mac OS X both in development and production mode and with Rails 3.1 RC4 or the stable branch. Everything works as expected here. The problem does show up on my Ubunutu production server with Passenger. Could it be that Passenger has a different way of caching those assets?
Setting config.cache_classes = false seems to fix the problem for now on Ubuntu but clearly isn't the desired solution.
config.cache_classes = false
@ncri I'm on the cedar beta stack. This is running on top of MRI 1.9.2.
I'm also targeting the rails 3-1-stable branch in my Gemfile and am currently on ref 5c591e5.
Bundler did bundle sprockets-2.0.0.beta.10
gem 'rails', :git => 'git://github.com/rails/rails.git', :branch => '3-1-stable'
gem 'sass-rails', "~> 3.1.0.rc"
gem 'mongoid', :git => 'git://github.com/mongoid/mongoid.git'
gem 'devise', :git => 'git://github.com/plataformatec/devise.git'
gem 'redis', '2.2.0'
gem 'redis-store', '1.0.0.rc1', :git => 'git://github.com/jodosha/redis-store.git'
gem 'redis-namespace', '0.10.0', :require => false
group :test, :development do
gem 'rr', :git => 'git://github.com/btakita/rr.git'
gem 'mongoid-rspec', :git => 'git://github.com/evansagge/mongoid-rspec.git'
gem 'resque_spec', :require => false
@salbito: Which Server do you use in Production? I use thin with gem 'rails', '3.1.0.rc4' and caching switched on works. I'm also on cedar with Ruby 1.9.2.
@ncri unfortunately I am away from my development laptop. If it helps any I have not specified a server outside of the default on heroku. It was a simple run and gun deployment of a toy application.
Forgot to mention that currently my application has configured assets as follows:
config.assets.enabled = true
config.assets.js_compressor = :uglifier
config.assets.css_compressor = :scss
I have tried multiple solutions and this is the outcome of one of them. I'll follow up when I get home.
Rake task run on deployment:
After doing this I have tried to view the image directly and I am presented with an image stating something akin to this:
"We cannot display the image as it has errors"
I am unfortunately writing this off memory but I wanted you to have something to go on until I am back at home to give you the nitty gritty on everything else
Try using thin as a webserver. That solved many issues for me. You need to create a Procfile for that. See the Heroku docs.
I dont use asset compression yet. So maybe try removing
config.assets.js_compressor = :uglifier
config.assets.css_compressor = :scss
if it still doesn't work with thin.
That image error is weird. Have never seen it before. I do precompile my images for heroku when in development. That works fine.
The issue occurred before and after configuration of the compressor. Images are broken either way. I will try out thin when I get home. I will also take screenshots of the image error.
Also using thin as the server is not the greatest solution to this issue. I am trying to pinpoint the issue and a lot of it leads me to sprockets being in a broken state. looking at the travis ci reports for sprockets it appears that the build has been failing for 1.9.2 for quite awhile.
The issue is in 3-1-stable. If you want to use the pipeline now, then stick with RC4, where it appears to work. @spastorino has reopened the bug, so someone will be looking into it.
If you need to change the prefix config option this is broken in RC4, so you'll need to wait until this bug is resolved.
RC4 also has this issue. There is currently no sha I can track that has the asset pipeline working in production deployments. Ill try to dig in and figuire it out this afternoon. You can only try so many shas before you give up and decide to avoid the asset pipeline period. Granted this is RC.
Well, it works for me in production as I said. One slight issue I have though is that assets are not compiled in the correct order, which is working correctly in development. However, I can avoid this issue by precompiling locally in development. This is desired anyway for heroku as you can't write to the filesystem there.
I had an image issue in production (Heroku's cedar stack), but resolved it by setting
config.action_dispatch.x_sendfile_header = "X-Accel-Redirect"
in my production.rb environment file. The reason, as it appeared to me, is that the Cedar stack cannot handle X-Sendfile, but it can handle X-Accel-Redirect. Not sure if this helps anyone out, but I sure hope so.
@matthewbennink is correct, I did some digging this evening and it's caused by using the X-Sendfile header. As you can see from this line in rack:
When you use X-Sendfile it sets the Content-Length to 0. One of the reasons some people above believe it is a caching issue is that when 'perform_caching = true', that Content-Length header is saved to the cache and re-served every time that asset is requested. So even if you change the x_sendfile_header to 'X-Sendfile', and then restart your server, the Content-Length of 0 might still be in the cache.
So the fix would be to switch to X-Accel-Redirect and wipe whatever cache you have (the default is filestore, so just remove the tmp/cache dir).
This still seems like a bug though.
I don't quite get it, so using X-Sendfile is the default, but Heroku cannot handle it? And what is actually the difference of using X-Sendfile vs. X-Accel-Redirect? And why does switching to thin from webrick or mongrel solved the issue for me, without even touching config.action_dispatch.x_sendfile_header? Well, still many open questions. ;-)
Okay, this page http://wiki.nginx.org/XSendfile says about X-Sendfile:
"Lighttpd has this feature and there is a mod_xsendfile for Apache2. Nginx also has this feature, but implemented a little bit differently. In Nginx this feature is called X-Accel-Redirect."
So if Heroku used Nginx on their cedar stack it would make sense that X-Sendfile doesn't work, but they don't afaik.
So maybe there is an issue with XSendfile and mongrel/webrick in production?
None of the workarounds work for me. Changing config.action_controller.perform_caching, config.action_dispatch.x_sendfile_header and clearing tmp/cache has no effect: the X-Sendfile header always gets sent on static assets.
My config: rails 3.1.0-rc4 on ruby 1.9.2, passenger+nginx, and since I use mongodb Active Record is disabled.
Never mind, it magically started working fine. Not sure why...
I'd say that this is not an issue but there are some weird things (like returning 200 OK and content-length 0 for images that are not compiled).
The thing is: in production is required that you always manually run rake assets:precompile (manually for now, we are thinking about other approaches).
If you're running in production with an HTTP server that all. But if you're starting your server from the console using rails s RAILS_ENV=production you will also need
config.serve_static_assets = true
Doing all the things below and running with Rails 3-1-stable and Sprockets from master shouldn't show any issue.
Anyways let me know what happens to you guys.
+1 for X-Accel-Redirect on heroku w/cedar! and
config.serve_static_assets = true
@spastorino 200 with 0 length appears to be correct with X-Sendfile headers. Sprockets has a to_path method which is used in Rails middleware to get the path to send. If this failed the web server would be the one sending a bad gateway 502.
Isn't it correct that in production, you don't have to precompile? Sprockets can serve the fingerprinted assets, and using the X-Sendfile headers only has to pass the filepath upstream after mapping the request for the fingerprinted file to a real file?
At the moment though it seems to grab files out of app/assets, but I suspect it should grab them from tmp/cache/assets instead.
It is also odd that it does not do this for CSS and JSS files. I wonder if these should be X-Sendfiles from the cache store.
For compiling, you could do this on the first request for the asset, but isn't the point of precompiling that you do it during deployment to avoid all those first request misses? :-) I suppose you could allow Sprockets to serve that first request, and save the result. The thing to watch is that the first request would then have far-future headers, whereas subsequent requests (unless it is set on the server) won't.
I think the key thing is that the end user (at the moment) needs to have some understanding of how they all work together.
Does that make any sense?
Having the same issue running RC4 without Sprockets in my Gemfile on Heroku Cedar. Turned caching off, didn't change anything.
Playing with my gemfile (newer sprockets, rails 3-1-stable) didn't help.
@olivierlacan action_pack has a dependency on sprockets so taking it out of your Gemfile doesn't actually get rid of sprockets (although it might change the version).
I meant without fetching Sprockets straight from Sam's github repo, on 2.0.0 beta 10, and not beta 11. Sorry for the confusion.
This is not working on the latest rails stable 3-1 branch + sass-rails 3-1 branch.
The rendered css file just displays "asset_url" instead of the url(path)
background: asset_url("top-main-shadow.png") no-repeat; }
Can someone summarize the problem along with some concise reproduction steps? I want to help, but the number of comments, opens, and closes is confusing me. :-(
The problems seems to be people caught out by the docs being a bit thin, mixed in with a couple of bugs and issues with sprockets beta.11.
Assuming 3-1-stable, Sprockets beta.10, and a quiet thread:
Seems to be working fine. (running in prod for me :-) )
In my case I had X-Sendfile header set in application.rb in an upgraded app, which meant development and production were broken. Moving it into production.rb fixed that. Doh!
(Rails sends zero-length files along with the full file path for the upstream webserver when the header is on, which is correct)
In some other cases people seem to be using production servers that do not support X-Sendfile headers, or require the alternate X-Accel-Redirect header.
It appears the Sprockets had cached files when these headers were used, and these cached files (presumably 0 length - I did not check) cause problems in debugging. Probably also made worse by far-future headers jamming up any post-server caching.
When upgrading apps there are a bunch of settings you have to get Just Right (TM) or it just doesn't work. The interaction of various layers of cache on the server and client-side makes it hard to debug. The many settings, the new locations for assets, file extension chains, sass versus rails helpers, and preprocessors/compressors makes it more complicated (then /public).
I've spent a bit of time on the asset pipeline docs to make sure all the settings are documented, so hopefully that should help. There are some slightly inaccurate blog posts in the wild that may be adding to the confusion.
I think that the X-Sendfile header should be off be default in new apps, given the grief it has caused - you may not want large numbers of people cut-and-pasting it into existing apps or using it in new apps without a better understanding of what is required on the server-side.
The pipeline docs could do with more instructions/examples for specific upgrading use-cases. I will have a crack at this over the next few days if I have time.
Appears to be working fine, but setup is complicated, and there may be edge cases where stuff still does not work. Some internals may not be as robust as they could be, IMHO.
@draiken It is supposed to work without having to make it an erb file. That's is what sass-rails is supposed to do.
@tenderlove Here is how I am reproducing it. Basically none of the asset helpers mentioned in sass-rails are working. https://github.com/rails/sass-rails
Here is how I can reproduce the bugs and the actual issue.
Look at the last file first on the GIST https://gist.github.com/1103436
I have also seen the issue to be intermittent (which i am not sure why). Sometimes it actually does render the proper asset urls.
@dodeja I just tried it out using the scss you provided. Everything seems to work fine here. Here is a video of me not reproducing the error.
I'm going to close this ticket for now. If you're still having troubles, please open a new ticket with more detailed reproduction steps. Thanks!
@tenderlove: I think @dodeja sort of hijacked this ticket and reported a problem that wasn't related the original issue's problem. The issue has to do with image assets and the X-Sendfile header, and it only is a production issue. If you look at my first comment (#1822 (comment)) it provides a good summary of the issue. I good executive summary for this issue would be:
Using 'X-Sendfile' as the action_dispatch.x_sendfile_header in production running under Ngnix causes image assets to be served by the asset pipeline with a Content-Length value of 0.
@tenderlove Here is an app that reproduces the issue:
Pull and down and run bundle exec rails s -e production. Then make a request to localhost:3000/users and the rails.png image will have a content-length of 0.
@rubymaverick thanks for the concise reproduction. I'll have a look.
I agree that this is a documentation problem. Currently, the only thing that's telling users to switch X-Sendfile to X-Accel-Redirect is here:
# Specifies the header that your server uses for sending files
# (comment out if your front-end server doesn't support this)
config.action_dispatch.x_sendfile_header = "X-Sendfile" # Use 'X-Accel-Redirect' for nginx
Trying to use X-Sendfile with nginx also led to problems when uploading files to S3 via CarrierWave. So I think the best solution would be turn off X-Sendfile by default and encourage people to read over production.rb before deploying.
on Heroku you should set x_sendfile_header to nil or just remove the sendfile middleware.
@hone please let us know if you're going to automatically do that on Heroku.
This should close the issue, but please let us know if you still see some problem
This issue was closed three times and reopened so one more time should bother anyone :).
Please let us know if you still have issues and we will reopen.
So the default generated rails 3.1 app, with an unchanged production.rb, will not serve images correctly on the arguably most popular rails host? Where does the default production.rb actually even work?
@tenderlove I have tried that on three different computers and I can't render the images correctly.
@rubymaverick they (Heroku?) should just patch it themselves like they do with the other plugins. It's their job to know that nginx handles the headers correctly. This is a server configuration issue at this point. Nothing to do with Rails.
@rubymaverick it's possible that we could change the WEBrick handler to understand what X-Sendfile is. The problem is if someone is proxying Apache to WEBrick, then WEBrick would be serving up files rather than Apache.
I think the default production.rb file is supposed to be targeted at web servers that understand what X-Sendfile is. I'd be willing to change the WEBrick handler to deal with X-Sendfile, as long as we document which production setups we support.
Works for me!
@scanferla that worked for me too, thank you!
config.action_dispatch.x_sendfile_header = "X-Accel-Redirect" in the config/environments/production.rb fixed this issue for me on Heroku as well. Thanks guys for the clarification.
@scanferia @atd @olivierlacan you should use config.action_dispatch.x_sendfile_header = nil in Heroku
@spastorino I am deploying to a local server with Apache 2.2.17 and Passenger 3.0.7
@atd if you are using Apache you should set it to 'X-Sendfile', doesn't this work for you?
You need to compile and configure mod_xsendfile module read here
@spastorino you are right, installing libapache2-mod-xsendfile fixes the issue with 'X-Sendfile'.
I was lost because it worked with rc4, but it did not with rc5
@spastorino too hasty. I had the images cached. It does not work with mod_xsendfile module loaded.
@atd ok will take a look
@atd actually all the credit goes to @matthewbennink as he was the one who gave us the solution :)
config.action_dispatch.x_sendfile_header = "X-Accel-Redirect" works on cedar stack heroku thanks!
Interestingly enough rails 3.1 rc4 runs fine for me on the heroku cedar stack with the default config.action_dispatch.x_sendfile_header = "X-Sendfile", (thin, precompiled assets).
works well for me.
I use rails3.1.rc5 ,thin and nginx
Ah, well, actually with precompiled assets config.action_dispatch.x_sendfile_header = "X-Sendfile" shouldn't matter anyway, as in htat case rails simply links to the percompiled assets in the public dir. I will test without precomplilation at some point. Haven't done that yet though, as I think one always wants to precompile in production anyway...
Funnily, asset_path doesn't add a footprint when called inside my stylesheet.css.erb files. That ways all files referenced from styles are actually wrongly served by the asset pipeline, not from public/assets. This seems to be another bug, but proofs that config.action_dispatch.x_sendfile_header = "X-Sendfile" does in deed work on Heroku cedar.
I'm running Ruby 1.9.2 on prod under Apache and on dev under Webrick.
Image assets were served properly on dev and on prod when I was using Rails 3.1.4. However when I updated the project to Rails 3.1.0 rc5, I started to see the problem on the prod (only): Images were broken on the page and 304's appeared in the Rails log.
Removing the cache dir on prod and restarting the app does not help.
413pro: Yep, can confirm that: 3.1.0 rc5 breaks assets on production for me too. See also: #2299 (comment)
This eff7fdd should fix some of the issues you were having. I'd like to know if using 3-1-stable including this commit I've just done, is some of you hitting some issue?
@rubymaverick cool thanks :)
@spastorino should we reopen this ticket?
@tenderlove bro I don't get issues like this one and neither like #1810. I'm asking people to provide repro steps.
Others are getting JRuby issues like #2383, but I have no idea if current 3-1-stable code + sprockets master have some known issue
@spastorino I seem to be able to reproduce this using heroku. I will look more tomorrow.
I've talked with @tenderlove and showed how things are working for me with current 3-1-stable and sprockets master, possibly after I made config.action_dispatch.x_sendfile_header default to nil.
I've made a new app with a post scaffold and added two lines on application.html.erb in order to show the result of asset_path and image_tag.
Take a look at the app https://github.com/spastorino/asset_pipelining_test and test it on heroku http://empty-stream-704.herokuapp.com/posts.
I'm going to close this issue until someone is able to provide a steps which shows new issues :).