Loading assets in production? #260

Closed
mhenrixon opened this Issue Jun 20, 2012 · 43 comments

Comments

Contributor

mhenrixon commented Jun 20, 2012

No idea if anyone cares about this but I can't get the assets to load in production wich makes the admin page pretty fugly. Any suggestions to how I might make it compile or use the assets needed?

Owner

mperham commented Jun 20, 2012

I use sidekiq in production every day and it works fine, as do plenty of users. Can you be more specific about what fails to load?

Contributor

mhenrixon commented Jun 20, 2012

this is what production http://url/sidekiq looks like proof

Owner

mperham commented Jun 22, 2012

No info => closed issue.

@mperham mperham closed this Jun 22, 2012

Contributor

mhenrixon commented Jun 22, 2012

Eh? What info do you need? Did you even see that image?

Owner

mperham commented Jun 22, 2012

Yes I see the image. The CSS isn't loading. You'll need to debug why the CSS fails to load for you. You are the only one reporting this issue so I feel pretty safe in saying it's an issue in your environment or network.

Contributor

mhenrixon commented Jun 22, 2012

I think it suffers from the same problem as resque does when run under a rails 3.1 app. https://github.com/defunkt/resque/issues/418 and I am far from the only one that experienced that issue.

Owner

mperham commented Jun 22, 2012

Sounds like this is an application configuration issue, not something that Sidekiq can solve as a gem. The application needs to be configured properly to serve gem assets, whether resque, sidekiq or whatever. We are using the asset pipeline in Rails 3.2 with sidekiq mounted behind nginx and everything works fine. This is our entire asset pipeline configuration in production and we are just using the standard capistrano/bundler integration to deploy:

  config.assets.compress = true
  config.assets.compile = false
  config.assets.digest = true
Contributor

mhenrixon commented Jun 22, 2012

I feel like a rittard and by the looks of things our asset hijacking protection was the cause of the problems. That's what I get for trying to be smart.

#taskk.it.conf
## Stop Image and Document Hijacking
location ~* (\.jpg|\.png|\.css)$ {
  if ($http_referer !~ ^(http://taskk.it) ) {
    return 405;
  }
}

I comment out all that code and let the world steal our assets for now which in turn makes the Sidekiq assets load properly. I'll post a solution here if I found one where I can still protect our assets from being referenced from 3d party websites.

inetufo commented Aug 30, 2012

The same issue in production mode.Environment is nginx + passenger + rails3.2.8. Don't know how to solve...

Contributor

mhenrixon commented Aug 30, 2012

Post your nginx config and I'll tell you how to fix it.

inetufo commented Aug 30, 2012

And when i request http://url/sidekiq/assets/application.css, nothing responsed.

geoffw8 commented Aug 30, 2012

@mhenrixon Same thing is happening to me. Although I'm on Apache. Do you think its definitely an webserver setting? In the logs I can see a request is made for the files, then I get a timeout shortly after. Like you say, it looks pretty fugly!

Owner

mperham commented Aug 30, 2012

See FAQ: Why does the Sidekiq Web UI look terrible / not render correctly in production but works fine in development?

Contributor

mhenrixon commented Aug 30, 2012

@geoffw8 I am positive it's a webserver setting.

Contributor

mhenrixon commented Aug 30, 2012

What @mperham meant is on the bottom of https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting there is a section on why but not how to solve it. If your webserver (like mine) drops all requests to assets that aren't originating from the same folder as the web app is running in those requests will end up with a 404 and the web ui will look like shit. As I wrote in my previous entry check for something like:

# Stop Image and Document Hijacking
location ~* (\.jpg|\.png|\.css)$ {
  if ($http_referer !~ ^(http://taskk.it) ) {
    return 405;
  }
}

inetufo commented Sep 2, 2012

@mhenrixon here is my nginx config
server {
listen 80;
server_name localhost;
root /home/railsapp/public;
passenger_enabled on;
rails_env production;

    location ~ ^/(assets)/ {
            expires     1y;
            add_header  Cache-Control public;

            add_header ETag "";
            break;
    }
Contributor

mhenrixon commented Sep 2, 2012

The below is a production domain.com config we are using without problems. Admittedly this is for unicorn though so passenger milage may vary.

server {
  listen 80 default deferred; # for Linux

  client_max_body_size 4G;
  server_name domain.com ;
  root /var/www/domain.com/current/public;
  keepalive_timeout 5;

  try_files $uri/index.html $uri.html $uri @app;

  location ~ ^/(assets)/  {
    gzip_static on;
    expires     max;
    add_header  Cache-Control public;
  }

  location @app {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;

    proxy_pass http://app_server;
  }

  error_page 500 502 503 504 /500.html;
  location = /500.html {
    root /var/www/domain.com/current/public;
  }
}

Let me assure you it's not a sidekiq issue. If you are running into problems you need to check the documentation for passenger and or nginx/apache and search for the problem there.

I spent half a day with this problem, and the root cause was that I was running an old version of dalli, 1.0.3. dalli apparently was not connecting to memcached correctly. It threw a warning, but I am not really relying on much caching, so I kind of ignored it that error.

Well, wouldn't you know it, this was the cause of /sidekiq/application.js|css not being served. I guess sprockets uses the Rails.cache for these assets.

The fix was either commenting out config.cache_store or upgrading dalli

geoffw8 commented Oct 10, 2012

Hi,

Thanks everyone for still looking at this. If I'm honest, I have no idea. I
have 2 servers, with the EXACT same setup (I followed the SAME tutorial),
one prod works, one local works, the second prod doesn't.

....I'm just gonna live with it, bigger fish to fry.

If anyone works it out, please do let me know!

Thanks again for your efforts!

Regards,

Geoff

On Wed, Oct 10, 2012 at 8:33 PM, Ian Warshak notifications@github.comwrote:

I spent half a day with this problem, and the root cause was that I was
running an old version of dalli, 1.0.3. dalli apparently was not connecting
to memcached correctly. It threw a warning, but I am not really relying on
much caching, so I kind of ignored it that error.

Well, wouldn't you know it, this was the cause of
/sidekiq/application.js|css not being served. I guess sprockets uses the
Rails.cache for these assets.

The fix was either commenting out config.cache_store or upgrading dalli


Reply to this email directly or view it on GitHubhttps://github.com/mperham/sidekiq/issues/260#issuecomment-9316346.

BR,

Geoff Wright
Founder & CEO

w: weartolook.com
t: @weartolook http://www.twitter.com/weartolook

weartolook LTD
St John's House,
54 St John's Square,
London, EC1V4JL

@mperham I'm not sure why you keep pushing this aside as an individual user server config issue (and in your FAQ). Isn't the way sidekiq-web handles these assets completely the opposite of the way assets are normally handled?

The typical configuration is to have your assets precompiled in production, with your server set up to serve as static assets in order to avoid hitting your rails app for CSS/JS requests. For the server to let sidekiq serve the assets directly from the gem, you are requiring people to set up a location block specifically for your gem. I.e. I was having the same problems, and this fixes it in Nginx:

  location ~ ^/(assets)/  {
    expires     max;
    add_header  Cache-Control public;
    access_log off;
  }

  location ~ ^/sidekiq/(assets)/ {
    try_files $uri @unicorn;
  }

You have to specifically pass the /sidekiq/assets location to your rails app to be served out the gem.

There really should be an option to copy the assets out of the gem and package into /public/assets/ in production. We shouldn't be hitting the app server to request CSS & JS files.

My $0.02.

Owner

mperham commented Oct 26, 2012

@brandonparsons Our assets work fine and we don't have any sidekiq-specific nginx configuration. That's your user-configuration issue. If you do the simplest thing in nginx, it will work. People try to be smart and break this edge case.

The simplest thing is this:

  1. try to have nginx serve the asset from the filesystem
  2. if it does not exist, pass the request to Rails.

That's it. If it doesn't work, it's because you aren't doing #2.

@mperham Can you share your nginx config?

On Fri, Oct 26, 2012 at 11:24 AM, Mike Perham notifications@github.comwrote:

@brandonparsons https://github.com/brandonparsons Our assets work fine
and we don't have any sidekiq-specific nginx configuration. That's your
user-configuration issue.
If you do the simplest thing in nginx, it will
work. People try to be smart and break this edge case.

The simplest thing is this:

  1. try to have nginx serve the asset from the filesystem
  2. if it does not exist, pass the request to Rails.

That's it. If it doesn't work, it's because you aren't doing #2#2
.


Reply to this email directly or view it on GitHubhttps://github.com/mperham/sidekiq/issues/260#issuecomment-9820478.

Owner

mperham commented Oct 26, 2012

Here's what we use (with some pruning of sensitive content). I think the last line is critical as it encompasses step 1 and 2 above.

# unicorn
upstream clymb {
  server 127.0.0.1:8080 fail_timeout=0;
}

# http
server {
  listen 80 default_server;
  server_name _;
  send_timeout 5m;

  location ~ ^/(images|javascripts|stylesheets)/  {
    expires 24h;
    add_header Cache-Control public;
  }

  location ~ ^/assets/ {
    expires 1y;
    add_header Cache-Control public;

    add_header ETag "";
    break;
  }

  location @clymb {
    proxy_pass  http://clymb;
    proxy_redirect     off;
    proxy_read_timeout 5m;

    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto  http;
  }

  try_files $uri @clymb;
}

@mperham Cheers - thanks! Appreciate it.

geoffw8 commented Nov 1, 2012

I use Apache, I have this issue. Two servers, EXACT same setup. Works on
one, not the other...

On Fri, Oct 26, 2012 at 6:43 PM, brandonparsons notifications@github.comwrote:

@mperham https://github.com/mperham Cheers - thanks! Appreciate it.


Reply to this email directly or view it on GitHubhttps://github.com/mperham/sidekiq/issues/260#issuecomment-9821070.

BR,

Geoff Wright
Founder & CEO

w: weartolook.com
t: @weartolook http://www.twitter.com/weartolook

weartolook LTD
St John's House,
54 St John's Square,
London, EC1V4JL

nragaz commented Nov 12, 2012

Another option for those struggling with this issue: I couldn't figure out a clean way to get Passenger to hit Rails and request these assets, so I just symlink the assets into the public folder during deploy.

My Capistrano task looks like this:

task :link_sidekiq_assets, roles: :web do
    bundle_path = capture "cd #{release_path}; bundle show sidekiq"
    bundle_path = "#{bundle_path.strip}/web/assets"
    run "ln -nfs #{bundle_path} #{release_path}/public/sidekiq"
end
Owner

mperham commented Nov 12, 2012

I added a FAQ entry for this issue. https://github.com/mperham/sidekiq/wiki/FAQ

chetan commented Dec 15, 2012

I have this problem as well. My nginx config is almost identical to the one posted by @mperham and I get the following rails error:

ActionController::RoutingError (No route matches [GET] "/home/chetan/.rvm/gems/ruby-1.9.3-p327/gems/sidekiq-2.5.4/web/assets/stylesheets/application.css")

Oddly, when I hit the ruby server (unicorn in this case) directly, the asset is served just fine, but when I hit nginx, I get the above error. Haven't been able to debug it any further than that, and don't want to spend much more time on it, but if you have any ideas, I can try to get more info.

The symlink posted by @nragaz works nicely for me, so that's what I'm going with for now.

Do you have to do anything special to make it work, if I have sidekiq under another namespace e.g. /admin/sidekiq? I can't get either the standard nginx config or @nragaz's example to work even though I changed the dir to #{release_path}/public/admin/sidekiq

@NielsKSchjoedt I was able to make this work by modifying config/environments/production.rb and uncommenting this line config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect' (be sure to comment the line # config.action_dispatch.x_sendfile_header = "X-Sendfile") and clear your browser cache.

BANG! And like that, you solved it! Yes, I had forgotten to change that, after I moved to nginx. This made the default behavior work flawlessly :-D

@nragaz symlinking the assets from the gem into public worked for us on Apache

@nragaz your idea helped me... I wrote a automated script to pull the assets from Gem to public with Capistrano

namespace :deploy do 
  task :copy_sidekiq_assets, :roles => :app, :only => { :primary => true } do 
   sidekiq_asset_path = capture("cd #{current_path} && bundle show sidekiq").split("\n")[0] + "/web/assets/."
   run <<-CMD
   cp -R "#{sidekiq_asset_path}" "#{current_path}/public/sidekiq"
   CMD
 end
end

babaru commented Jul 9, 2013

I've tried @mperham 's nginx configuration file and it works for me.

Ran into this same issue, and making sure to use X-Accel-Redirect instead of X-Sendfile as mentioned by @kfalconer fixed it for me, added to Problems-and-Troubleshooting wiki page.

Took me a while to realize that if

config.action_dispatch.x_sendfile_header = "X-Sendfile"

is enabled, then for Sidekiq assets to work, Apache needs the XSendFile module to be installed (for Ubuntu: libapache2-mod-xsendfile), along with this in your vhost container:

XSendFile on
XSendFilePath /path/to/bundled/gems
# or if you have XSendFile 0.9:
XSendFileAllowAbove on

@emptyflask - that worked for my apache installation... thank you.

NoICE commented Feb 19, 2014

I have this problem too and it was indeed by using X-Accel-Redirect. If I comment the config.action_dispatch.x_sendfile_header line out, the app servers the assets correctly.

BUT! I'm using NGINX and I use X-Accel-Redirect to serve other files. Here is snippet of my nginx config:

        passenger_set_cgi_param HTTP_X_ACCEL_MAPPING /var/www/shared/storage/=/storage/;
        passenger_pass_header X-Accel-Redirect;

        location /storage/ {
          root /var/www/shared;
          internal;
        }

As you can see in this question on stack overflow: http://stackoverflow.com/questions/6237016/message-x-accel-mapping-header-missing-in-nginx-error-log, this configuration is needed to get it to work without errors. NGINX documentation says the x-accel-mapping is required.

So now I have to somehow specify multiple paths in my x-accel-mapping to get it to work (or somehow whitelist the path to sidekiq web gem).

How can it work for you guys with default nginx configuration (i.e. without setting x-accel-mapping) is beyond mystery to me.

Basically, it is completely wrong to let nginx serve assets directly from some gem path. Which it does now, because of x-accel-redirect. I don't blame sidekiq for this, because this is just not-so-well-working combination of features, not someones fault. I ended up copying sidekiq assets to my app public directory, so I don't need to mess with nginx configuration every time I update some gem/ruby to different version.

avit commented Sep 2, 2014

I'm seeing the request for Started GET "/sidekiq/javascripts/dashboard.js" in my rails log, followed immediately by Started GET "/full/path/to/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.2/web/assets/javascripts/dashboard.js"

I have config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect' enabled. Is there a good way to see what headers are being sent between nginx and the proxied app to debug this?

Owner

mperham commented Sep 2, 2014

@avit That's probably a better question for another project (nginx? sinatra?) I don't know the answer.

avit commented Sep 2, 2014

I was hoping some of the good people here would know, since it seems to be a common setup; some people are working around it by copying/symlinking the sidekiq assets to /public/ on deploy, which doesn't seem ideal either.

Collaborator

seuros commented Sep 2, 2014

I suppose you are using Heroku or a hosting that use STDOUT/STDERR for logging.
The file is send only once, the first time it rails's log, the second is Sinatra's.
Since sidekiq web is an administrative tool, that won't impact your server unless you have few thousand admins in the same time :)

keysen commented Oct 14, 2014

I found two solutions to fix this problem, the first easy one is to add a location in your nginx.conf with the path of your sidekiq admin... For example:

#FIX: sidekiq assets because it uses Sinatra and it's not compatible with our HTTP_X_ACCEL_MAPPING already defined
location /admin/sidekiq {
    passenger_enabled on;
    expires max;
}

Or the other one, you can play with regular expressions and add something that uses the assets present inside the gem:
http://airbladesoftware.com/notes/rails-nginx-x-accel-mapping/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment