Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

support for the new cedar stack #26

Open
abhishiv opened this Issue · 51 comments
@abhishiv

Scaling on the new cedar stack is via the heroku scale commoand, and it doesnt seem to work with the current implementation.

@meskyanichi
Owner

How does this stack differ from any other stack? Got any info/docs?

@meskyanichi
Owner

Oh I see you asked the question over at Twitter. I'm not sure what the Cedar stack is to be honest. Is that similar to Bamboo and Aspen stacks? It looks like they locked down their documentation on it:

http://devcenter.heroku.com/tags/stack

@abhishiv

Hey there!

Basically the stack uses the Procfile model to declare app types. Our Procfile looks like

web:     bundle exec rails server thin -p $PORT
gmailworker:  bundle exec rake resque:work QUEUE=gmail
linkedworker:  bundle exec rake resque:work QUEUE=linkedin

So basically what it means is that you can have multuple type of workers, which could be scaled by using the heroku scale command

heroku scale web=3 gmailsworker=5

So basically as you see it's a great model, we switched to it instantly although it's still in beta. However the current implementation of hirefire might be incompatible. I actually went through the source, and basically the way the number of workers are retrieved from the heroku api needs to be changed, as well a way to declare the number and type of workers in the configure block.

Hope that helps!

@meskyanichi
Owner

Seems like that'd be quite an overhaul. Also since it's still in beta they might make a swift move and turn it in to another direction, thus breaking functionality again.

This Procfile thing seems quite interesting. But does this mean you're spinning up 3 dyno's and 5 workers?

heroku scale web=3 gmailworker=3 linkedworker=2

Thanks for the info!

@abhishiv

yup, although i talked to the support at heroku before migrating, and seems like they are quiet committed to this stack. You might also wanna check out this: http://adam.heroku.com/past/2011/5/9/applying_the_unix_process_model_to_web_apps/

and yeah that means it's a total of 5 resque workers, however you can also spin up non-web, non-worker processes as well, and you can name them anything. So basically you need to tell hire-fire which processes to monitor, and which type they are(resque, dj etc) while configuring.

@meskyanichi
Owner

Thanks. I haven't really taken an in-depth look yet but I actually came across DDollars's other gem called Foreman. I suspect it originates from that, or the other way around. It also uses a Procfile and you can turn the quantity up or down per process group. Though, Foreman is used on VPS's not Heroku, I think it's just a port of their Cedar stack.

All in all, I'll have to see. It'll take some significant changes to get this implemented. Not sure if I'll do it anytime soon for either HireFire open-source or HireFire hosted-service, but I'll keep an eye on it and see if anything can be done to implement it in a good way for both projects.

I'm loving Foreman and I can see Cedar stack being very useful. By the way, does this mean you can run different app-servers on Heroku, instead of just Thin? What if you define this in your Profile:

bundle exec unicorn -p $PORT -E production

Would you be able to spawn a unicorn instance instead of Thin? That'd be pretty sweet.

@abhishiv

yeah, unicorn works perfectly on heroku, although i haven't been seeing any performance gains by switching to unicorn.

Also for reference, heroku workers command doesnt work anymore(both client and cli), so to get the number of workers you have to issue heroku ps, and then filter the array for processes that issues resque/dj commands.

@meskyanichi
Owner

Nah - Unicorn isn't about performance. I'm only interested in it's threading capabilities which Thin is extremely bad at.

Also, you're saying the following doesn't work?

client.set_workers('myapp', 3)

Does no longer work? http://hirefireapp.com/ is still using it without issues. Or was that not what you meant?

@abhishiv

ah sorry, i should have been specific.

I meant it doesnt work with the cedar stack.

@meskyanichi
Owner

Gotcha. Yeah the Cedar stack seems interesting. I'll have to check that out some time. I might incorporate it at a later time. Thanks for the info!

@meskyanichi
Owner

@abhishiv After my article on Celadon Cedar with Unicorn (see: http://michaelvanrooijen.com/articles/2011/06/01-more-concurrency-on-a-single-heroku-dyno-with-the-new-celadon-cedar-stack/ ) I added support for Celadon Cedar on http://hirefireapp.com/ in case you're still interested.

Check out this page in the knowledge base for more information (see Use Celadon Cedar Stack heading) and to see if it fits your needs.

@abhishiv

Hey looks great! However i can see one issue.

Most of the people switch to cedar because it's makes it possible to run different type of processes(workers). Right now the idea that i get from the hirefire documentation is that it is only gonna scale the processes named "workers". That I think will be an issue for most of the people.

@meskyanichi
Owner

Yup. Though I wonder if it's really possible to monitor all different processes because that would mean customization on the client-side (hacking in the middleware/gem). If someone has custom worker processes that have A and B in the queue, my system would need to be aware of that, and would also have to query based on that etc. Every app is more or less unique so that'd mean everyone would have different needs when it comes to various processes. I don't think there really is a way to elegantly handle both the server as well as the client side.

Also, I assume that if you run Delayed Job, you're not also going to run Resque, and vice-versa. At most you'd write custom daemons (like i did with HireFireApp) which wouldn't be compatible in the first place.

Though, maybe im missing the broader picture. What were your intentions (and what would your Procfile look like) if you moved (or have already moved) to Celadon Cedar?

Thanks!

@abhishiv

Hey, yeah if you are using resque probably you wouldn't want to use dj as well. However for most practical purposes you would want have multiple types of resque workers, so you can scale them independently. For example, our Prcofile looks like this:

web:     bundle exec rails server thin -p $PORT
gmailworker:  bundle exec rake resque:work QUEUE=gmail
linkedworker:  bundle exec rake resque:work QUEUE=linkedin

There are different type of workers, because gmailworker needs to be near realtime; so I would configure hirefire to spin up a lot of processes for it. While a different strategy would be used for linkedinworker, as it doesnt have that high a priority.

What do you mean when you say that if it's possible to monitor all different processes? Using the heroku api, you mean?

@meskyanichi
Owner

I see. Nah, I know you can use heroku scale a=b c=d e=f etc but how would the client transfer this information (for both DJ as well as Resque) to the server, which would then be able to determine what to scale and how much?

For example, with Delayed Job you can only set "priority" iirc to for example min 10, max 10 and then you enqueue everything you want to run on that worker with the priority of 10. Then on the client-side you'd have to be able to fetch all the jobs and convert the results to JSON and sent it back to the server as {'10' => 45}, {'5' => 20} and then from the server-side the user has to configure each and every priority and worker/job ratio and it might work.

Not sure if this would be too complex for the average user to go and configure properly.

Then comes Resque, you'd have to be able to count the jobs for each "queue" and convert that to JSON and send that back to the server.

I think that as long as each worker type's ODM/ORM/KVS is able to basically gather information for each queue, then it might work. For example in DJ:

sorted_documents = Hash.new
Delayed::Job.where(:run_at.lte => Time.now).each do |document|
  sorted_documents[document.min_priority] ||= 0
  sorted_documents[document.min_priority]  += 1
end

sorted_documents.to_json would look something like this:

{
  # Priority => Job Count
  "10" => 45,
  "5"  => 30
}

So now you've managed to get the proper priority and job count for that priority, but how
would you know that "10" means "gmailworker"? That's another problem. With Resque that might
be less tricky. Not sure how you query for particular queues but I think it's easier because
you have to specify the name of the queue in the worker class.

Your thoughts?

@abhishiv

Hey sorry, have been out for the last few days.

Ah that's complex. Didn't think about it like this. I haven't worked with dj a lot, but as you say with Resque it'll be much easier.

As I see it:

1) User defines he wants to add a worker for each 10 jobs on the gmailworker queu.

2) hirefire gets the number of gmailworkers active using heroku's api.

3) It then queries redis about the number of jobs in the gmailworker queue.

4) Then it increases/decreases the number of workers.

Does it make sense? Not sure how it would look like on delayed job.

@meskyanichi
Owner

Yeah, that's the tricky part, cause people will eventually want more and more worker library support. Had a request for one I've never heard of before. I'll have to think about this some more because I don't want anything to become brittle. Celadon Cedar is a great stack, can do some good stuff with it. But due to it's freedom it becomes harder (as you know) to find the right way to go about things while being able to utilize any worker library (or at least the popular ones, DJ and Resque).

It's quite a smart move on Heroku's part, cause adding the flexibility makes stuff like this harder, and they can easily generate more income from this since people tend to spawn more workers. I love Cedar though, it's awesome, but it's the $35/mo/process that's a pain.

I'll have to think about this and see if there is a simple solution to solve this. In the mean time if people want to use Cedar for other reasons and still want to use hirefireapp.com they can, but are only limited to monitoring all workers via the worker process.

@zefer

Hi. This is a quick & dirty patch to get this to work on Cedar. It's not pretty, and it's tied to a single process-type named "worker' but it might help someone out: https://gist.github.com/1082673

@krzkrzkrz

+1 for hirefire to work on the Cedar stack. Would like to see this working soon. Keep up the good work!

Any ideas as to when a Cedar version of the gem will be released?

@thinkgareth

Another +1

Cedar is pushed as the de-facto stack for all apps using Rails 3.1 so support is essential. Adding just 'worker' support is fine for now however.

Wish I could find the time to assist.

G

@thinkgareth

The patch from @zefer seems to make it work temporarily, in my tests, it will only scale up once or twice. I wonder if it's because sometimes they error on quit rather than shutdown cleanly:

2011-08-19T13:51:06+00:00 app[worker.1]: [2011-08-19 13:51:06][HireFire] All queued jobs have been processed. Firing all workers.
2011-08-19T13:51:06+00:00 heroku[worker.1]: State changed from up to stopping
2011-08-19T13:51:07+00:00 heroku[api]: Scale to web=1 by mail@thinkgareth.com
2011-08-19T13:51:08+00:00 heroku[worker.1]: Stopping process with SIGTERM
2011-08-19T13:51:17+00:00 heroku[worker.1]: Error R12 (Exit timeout) -> Process failed to exit within 10 seconds of SIGTERM
2011-08-19T13:51:17+00:00 heroku[worker.1]: Stopping process with SIGKILL
2011-08-19T13:51:18+00:00 heroku[worker.1]: Process exited

I believe there is a use case for having one worker constantly on. Similar to or even part-of the Clock style worker for resque_scheduler.

That worker could check on the queue at much shorter intervals than once a minute and send the signal to boot other workers when required.

Could this solve some of the issues?

G

@thinkgareth

^^ This appears to be the issue mentioned above: https://github.com/defunkt/resque/issues/319

@jaimeiniesta

+1 to add cedar stack support to this gem.

There's a similar gem that currently supports the cedar stack, it's called workless:

https://github.com/lostboy/workless

I'm trying it at a pet project of mine, http://w3clove.com and it does seem to work fine, although it just switches between 1 and 0 workers, you can't configure it to scale it to more workers.

@mugwump

Now I'm confused: Is hirefire supposed to be working on cedar?! I just moved from bamboo to cedar, added a

worker: bundle exec rake jobs:work

to my procfile. The delayed jobs are enqueued properly, but hirefire does not start the worker. Is there anything else I need to do in order to make hirefire spin up workers on cedar?! Is the above patch needed or not needed? The readme states that hirefire also works on cedar, there is no mention of this patch...

@candland

@mugwump I'm still using the patch at https://gist.github.com/1082673

@rajagopals

I'm trying to use hirefire (with the patch) on the cedar stack, but it doesn't work. Its not spinning up workers at all. I'm using dj 3.0.0, and Rails 3.1.3, btw.

@pawel2105

Using Resque 1.2 and Rails 3.0.9 on the Cedar stack, with the patch and it's not working.

@atnmorrison

I ran into issues as well, is the cedar stack only going to be supported in the web service app?

@mugwump

ok, I finally got around to trying to fix this: The patch https://gist.github.com/1082673 is working almost: Apparently, the ENV['APP_NAME'] does not work on cedar. I'm currently using

return client.ps(ENV['NEW_RELIC_APP_NAME']).select {|p| p['process'] =~ /worker.[0-9]+/}.length

but you need to have the new-relic-plugin installed in order for this to work. Still trying to find out a better way to get to the app-name.

By some strange reaons I had

config.environment      =  nil

in my hirefire-initializer, which - for even stranger reasons - worked on bamboo, but it does not work on cedar: there is simply nothing at all happening if the environment is not set to heroku. Adding the proper environment for production solved this issue:

if Rails.env.production?
  HireFire.configure do |config|
    config.environment      = :heroku 
    ...

I forked hirefire (https://github.com/mugwump/hirefire) and the patch is currently living in the development branch - but use it at your own risk: It is working for me now, but I still have to find a way to test this properly...

@jaimeiniesta

@mugwump nice!

have you tried setting the ENV['APP_NAME'] yourself in heroku? Like this:

heroku config:add APP_NAME=my_awesome_app
@mugwump

No, I actually did not think about that, that would have been to easy :) Silly me, I just forgot that I had to set the app-name manually on bamboo too. I still think that there should be a more canonical way to find out the name of the app you are running on without having to set the var manually, but for now I changed the env-var back to APP_NAME and added the variable.

@zefer

Hey Guys. I'm still running fine using the patch, hirefire (0.1.4), resque (1.17.1), rails (3.1.0), Cedar stack. I have APP_NAME set as a Heroku config (env) var.

@mugwump

yes, same here now, works also fine with dj3.01 and rails 3.2. And yes, adding the env-var helps a lot :)

@pawel2105

I've edited the patch to fix the issues of workers not being hired if there are stuck or stale workers in the queue. Let me know if this works for those of you that haven't gotten it working yet. My patch fork here.

@jaimeiniesta

It's working fine for me on cedar, rails 3.1.3, delayed_job 2.1.4, and the patch from @pawel2105 on config/initializers

Thanks!

@jaimeiniesta

mmm, not so fast, the patch works, but it sometimes get stuck on 0 workers (my mininum default), and it doesn't hire more workers even when new jobs are created.

It wakes up if I scale manually the workers.

@jaimeiniesta

after upgrading to DelayedJob 3.0.0, it's been working fine for more than a week now.

@frunns

So I just put this in my lib/, set the env-variable and just hoped for the best, but that's not how it works, is it? What do you do with the patch?

@josegrad

@frunns You have to put the patch in initializers/patches

I just tried Cedar with Rails 3.0.10 and Delayed_job 2.1.4 and right now managed to get a worker starting up as expected. Not sure how stable yet but started after some struggle.

Had to use this:

HireFire.configure do |config|
  puts 
  if Rails.env.production? or Rails.env == "stagingcedar"
    config.environment      = :heroku 
  else
    config.environment      = :noop 
  end

and this in the patch:

["production", "stagingcedar"].include? ::Rails.env ? 'Heroku' : 'Noop'

Then it started the worker. Lets see tomorrow how it goes with more tests.
Obviously stagingcedar is a custom environment mode I have created for these tests.

Cheers.

@jeffreybiles

+1 for switching to workless. It's being actively maintained and works on local, bamboo/aspen, AND cedar. https://github.com/lostboy/workless

@jaimeiniesta

@jeffreybiles does workless support scaling to more than 1 worker? The good thing about hirefire is being able to scale depending on the number of jobs on the queue.

@jeffreybiles
@pawel2105

Workless is only for DJ, not Resque.

@jaimeiniesta

@jeffreybiles I looked at the workless source code and it doesn't seem like it allows scaling to more than 1 worker.

I'm planning to add this feature, though:

lostboy/workless#23

@jaimeiniesta

workless gem now can scale to multiple workers based on work load just like hirefire does, on this branch:

https://github.com/jaimeiniesta/workless/tree/multiple_workers

I've been using it on a heroku cedar app for over a month and has worked like a charm. I sent a pull request.

@neersighted

Bump.

@DenizOkcu

Hi guys,
i was annoyed by the deprecation warning in the heroku logs

app[worker.1]:  !    DEPRECATED: Heroku::Client#deprecate is deprecated, please use the heroku-api gem.

so i changed a few lines in the patch (https://gist.github.com/1082673).

class HireFire::Environment::Heroku
  require 'heroku-api'
  private
  def workers(amount = nil)
    heroku = Heroku::API.new(:api_key => ENV['HEROKU_API_KEY']) 
    if amount.nil?
      # return client.info(ENV['APP_NAME'])[:workers].to_i
      # return client.ps(ENV['APP_NAME']).select {|p| p['process'] =~ /worker.[0-9]+/}.length
      return heroku.get_ps(ENV['APP_NAME']).body.select {|p| p['process'] =~ /worker.[0-9]+/}.length
    end
    # client.set_workers(ENV['APP_NAME'], amount)
    # return client.ps_scale(ENV['APP_NAME'], {"type" => "worker", "qty" => amount})
    return heroku.post_ps_scale(ENV['APP_NAME'], "worker", amount) 
  rescue RestClient::Exception
    HireFire::Logger.message("Worker query request failed with #{ $!.class.name } #{ $!.message }")
    nil
  end
end

don't forget to add this to your gemfile:

gem 'heroku-api'
@neersighted

Bump, the sequel.

@phildow

A friendly bumpity bump.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.