Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Loading assets in production? #260

Closed
mhenrixon opened this Issue · 43 comments
@mhenrixon

No idea if anyone cares about this but I can't get the assets to load in production wich makes the admin page pretty fugly. Any suggestions to how I might make it compile or use the assets needed?

@mperham
Owner

I use sidekiq in production every day and it works fine, as do plenty of users. Can you be more specific about what fails to load?

@mhenrixon

this is what production http://url/sidekiq looks like proof

@mperham
Owner

No info => closed issue.

@mperham mperham closed this
@mhenrixon

Eh? What info do you need? Did you even see that image?

@mperham
Owner

Yes I see the image. The CSS isn't loading. You'll need to debug why the CSS fails to load for you. You are the only one reporting this issue so I feel pretty safe in saying it's an issue in your environment or network.

@mhenrixon
@mperham
Owner

Sounds like this is an application configuration issue, not something that Sidekiq can solve as a gem. The application needs to be configured properly to serve gem assets, whether resque, sidekiq or whatever. We are using the asset pipeline in Rails 3.2 with sidekiq mounted behind nginx and everything works fine. This is our entire asset pipeline configuration in production and we are just using the standard capistrano/bundler integration to deploy:

  config.assets.compress = true
  config.assets.compile = false
  config.assets.digest = true
@mhenrixon

I feel like a rittard and by the looks of things our asset hijacking protection was the cause of the problems. That's what I get for trying to be smart.

#taskk.it.conf
## Stop Image and Document Hijacking
location ~* (\.jpg|\.png|\.css)$ {
  if ($http_referer !~ ^(http://taskk.it) ) {
    return 405;
  }
}

I comment out all that code and let the world steal our assets for now which in turn makes the Sidekiq assets load properly. I'll post a solution here if I found one where I can still protect our assets from being referenced from 3d party websites.

@inetufo

The same issue in production mode.Environment is nginx + passenger + rails3.2.8. Don't know how to solve...

@mhenrixon

Post your nginx config and I'll tell you how to fix it.

@inetufo

And when i request http://url/sidekiq/assets/application.css, nothing responsed.

@geoffw8

@mhenrixon Same thing is happening to me. Although I'm on Apache. Do you think its definitely an webserver setting? In the logs I can see a request is made for the files, then I get a timeout shortly after. Like you say, it looks pretty fugly!

@mperham
Owner

See FAQ: Why does the Sidekiq Web UI look terrible / not render correctly in production but works fine in development?

@mhenrixon

@geoffw8 I am positive it's a webserver setting.

@mhenrixon

What @mperham meant is on the bottom of https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting there is a section on why but not how to solve it. If your webserver (like mine) drops all requests to assets that aren't originating from the same folder as the web app is running in those requests will end up with a 404 and the web ui will look like shit. As I wrote in my previous entry check for something like:

# Stop Image and Document Hijacking
location ~* (\.jpg|\.png|\.css)$ {
  if ($http_referer !~ ^(http://taskk.it) ) {
    return 405;
  }
}
@inetufo

@mhenrixon here is my nginx config

server {
listen 80;
server_name localhost;
root /home/railsapp/public;
passenger_enabled on;
rails_env production;

    location ~ ^/(assets)/ {
            expires     1y;
            add_header  Cache-Control public;

            add_header ETag "";
            break;
    }
@mhenrixon

The below is a production domain.com config we are using without problems. Admittedly this is for unicorn though so passenger milage may vary.

server {
  listen 80 default deferred; # for Linux

  client_max_body_size 4G;
  server_name domain.com ;
  root /var/www/domain.com/current/public;
  keepalive_timeout 5;

  try_files $uri/index.html $uri.html $uri @app;

  location ~ ^/(assets)/  {
    gzip_static on;
    expires     max;
    add_header  Cache-Control public;
  }

  location @app {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;

    proxy_pass http://app_server;
  }

  error_page 500 502 503 504 /500.html;
  location = /500.html {
    root /var/www/domain.com/current/public;
  }
}

Let me assure you it's not a sidekiq issue. If you are running into problems you need to check the documentation for passenger and or nginx/apache and search for the problem there.

@iwarshak

I spent half a day with this problem, and the root cause was that I was running an old version of dalli, 1.0.3. dalli apparently was not connecting to memcached correctly. It threw a warning, but I am not really relying on much caching, so I kind of ignored it that error.

Well, wouldn't you know it, this was the cause of /sidekiq/application.js|css not being served. I guess sprockets uses the Rails.cache for these assets.

The fix was either commenting out config.cache_store or upgrading dalli

@geoffw8
@brandonparsons

@mperham I'm not sure why you keep pushing this aside as an individual user server config issue (and in your FAQ). Isn't the way sidekiq-web handles these assets completely the opposite of the way assets are normally handled?

The typical configuration is to have your assets precompiled in production, with your server set up to serve as static assets in order to avoid hitting your rails app for CSS/JS requests. For the server to let sidekiq serve the assets directly from the gem, you are requiring people to set up a location block specifically for your gem. I.e. I was having the same problems, and this fixes it in Nginx:

  location ~ ^/(assets)/  {
    expires     max;
    add_header  Cache-Control public;
    access_log off;
  }

  location ~ ^/sidekiq/(assets)/ {
    try_files $uri @unicorn;
  }

You have to specifically pass the /sidekiq/assets location to your rails app to be served out the gem.

There really should be an option to copy the assets out of the gem and package into /public/assets/ in production. We shouldn't be hitting the app server to request CSS & JS files.

My $0.02.

@mperham
Owner

@brandonparsons Our assets work fine and we don't have any sidekiq-specific nginx configuration. That's your user-configuration issue. If you do the simplest thing in nginx, it will work. People try to be smart and break this edge case.

The simplest thing is this:

  1. try to have nginx serve the asset from the filesystem
  2. if it does not exist, pass the request to Rails.

That's it. If it doesn't work, it's because you aren't doing #2.

@brandonparsons
@mperham
Owner

Here's what we use (with some pruning of sensitive content). I think the last line is critical as it encompasses step 1 and 2 above.

# unicorn
upstream clymb {
  server 127.0.0.1:8080 fail_timeout=0;
}

# http
server {
  listen 80 default_server;
  server_name _;
  send_timeout 5m;

  location ~ ^/(images|javascripts|stylesheets)/  {
    expires 24h;
    add_header Cache-Control public;
  }

  location ~ ^/assets/ {
    expires 1y;
    add_header Cache-Control public;

    add_header ETag "";
    break;
  }

  location @clymb {
    proxy_pass  http://clymb;
    proxy_redirect     off;
    proxy_read_timeout 5m;

    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto  http;
  }

  try_files $uri @clymb;
}
@brandonparsons

@mperham Cheers - thanks! Appreciate it.

@geoffw8
@nragaz

Another option for those struggling with this issue: I couldn't figure out a clean way to get Passenger to hit Rails and request these assets, so I just symlink the assets into the public folder during deploy.

My Capistrano task looks like this:

task :link_sidekiq_assets, roles: :web do
    bundle_path = capture "cd #{release_path}; bundle show sidekiq"
    bundle_path = "#{bundle_path.strip}/web/assets"
    run "ln -nfs #{bundle_path} #{release_path}/public/sidekiq"
end
@mperham
Owner

I added a FAQ entry for this issue. https://github.com/mperham/sidekiq/wiki/FAQ

@chetan

I have this problem as well. My nginx config is almost identical to the one posted by @mperham and I get the following rails error:

ActionController::RoutingError (No route matches [GET] "/home/chetan/.rvm/gems/ruby-1.9.3-p327/gems/sidekiq-2.5.4/web/assets/stylesheets/application.css")

Oddly, when I hit the ruby server (unicorn in this case) directly, the asset is served just fine, but when I hit nginx, I get the above error. Haven't been able to debug it any further than that, and don't want to spend much more time on it, but if you have any ideas, I can try to get more info.

The symlink posted by @nragaz works nicely for me, so that's what I'm going with for now.

@NielsKSchjoedt

Do you have to do anything special to make it work, if I have sidekiq under another namespace e.g. /admin/sidekiq? I can't get either the standard nginx config or @nragaz's example to work even though I changed the dir to #{release_path}/public/admin/sidekiq

@kfalconer

@NielsKSchjoedt I was able to make this work by modifying config/environments/production.rb and uncommenting this line config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect' (be sure to comment the line # config.action_dispatch.x_sendfile_header = "X-Sendfile") and clear your browser cache.

@NielsKSchjoedt

BANG! And like that, you solved it! Yes, I had forgotten to change that, after I moved to nginx. This made the default behavior work flawlessly :-D

@tribalvibes

@nragaz symlinking the assets from the gem into public worked for us on Apache

@ankit8898

@nragaz your idea helped me... I wrote a automated script to pull the assets from Gem to public with Capistrano

namespace :deploy do 
  task :copy_sidekiq_assets, :roles => :app, :only => { :primary => true } do 
   sidekiq_asset_path = capture("cd #{current_path} && bundle show sidekiq").split("\n")[0] + "/web/assets/."
   run <<-CMD
   cp -R "#{sidekiq_asset_path}" "#{current_path}/public/sidekiq"
   CMD
 end
end
@babaru

I've tried @mperham 's nginx configuration file and it works for me.

@stevestmartin

Ran into this same issue, and making sure to use X-Accel-Redirect instead of X-Sendfile as mentioned by @kfalconer fixed it for me, added to Problems-and-Troubleshooting wiki page.

@emptyflask

Took me a while to realize that if

config.action_dispatch.x_sendfile_header = "X-Sendfile"

is enabled, then for Sidekiq assets to work, Apache needs the XSendFile module to be installed (for Ubuntu: libapache2-mod-xsendfile), along with this in your vhost container:

XSendFile on
XSendFilePath /path/to/bundled/gems
# or if you have XSendFile 0.9:
XSendFileAllowAbove on
@marknadig

@emptyflask - that worked for my apache installation... thank you.

@NoICE

I have this problem too and it was indeed by using X-Accel-Redirect. If I comment the config.action_dispatch.x_sendfile_header line out, the app servers the assets correctly.

BUT! I'm using NGINX and I use X-Accel-Redirect to serve other files. Here is snippet of my nginx config:

        passenger_set_cgi_param HTTP_X_ACCEL_MAPPING /var/www/shared/storage/=/storage/;
        passenger_pass_header X-Accel-Redirect;

        location /storage/ {
          root /var/www/shared;
          internal;
        }

As you can see in this question on stack overflow: http://stackoverflow.com/questions/6237016/message-x-accel-mapping-header-missing-in-nginx-error-log, this configuration is needed to get it to work without errors. NGINX documentation says the x-accel-mapping is required.

So now I have to somehow specify multiple paths in my x-accel-mapping to get it to work (or somehow whitelist the path to sidekiq web gem).

How can it work for you guys with default nginx configuration (i.e. without setting x-accel-mapping) is beyond mystery to me.

Basically, it is completely wrong to let nginx serve assets directly from some gem path. Which it does now, because of x-accel-redirect. I don't blame sidekiq for this, because this is just not-so-well-working combination of features, not someones fault. I ended up copying sidekiq assets to my app public directory, so I don't need to mess with nginx configuration every time I update some gem/ruby to different version.

@avit

I'm seeing the request for Started GET "/sidekiq/javascripts/dashboard.js" in my rails log, followed immediately by Started GET "/full/path/to/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.2/web/assets/javascripts/dashboard.js"

I have config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect' enabled. Is there a good way to see what headers are being sent between nginx and the proxied app to debug this?

@mperham
Owner

@avit That's probably a better question for another project (nginx? sinatra?) I don't know the answer.

@avit

I was hoping some of the good people here would know, since it seems to be a common setup; some people are working around it by copying/symlinking the sidekiq assets to /public/ on deploy, which doesn't seem ideal either.

@seuros
Collaborator

I suppose you are using Heroku or a hosting that use STDOUT/STDERR for logging.
The file is send only once, the first time it rails's log, the second is Sinatra's.

Since sidekiq web is an administrative tool, that won't impact your server unless you have few thousand admins in the same time :)

@keysen

I found two solutions to fix this problem, the first easy one is to add a location in your nginx.conf with the path of your sidekiq admin... For example:

#FIX: sidekiq assets because it uses Sinatra and it's not compatible with our HTTP_X_ACCEL_MAPPING already defined
location /admin/sidekiq {
    passenger_enabled on;
    expires max;
}

Or the other one, you can play with regular expressions and add something that uses the assets present inside the gem:
http://airbladesoftware.com/notes/rails-nginx-x-accel-mapping/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.