Basic Foreman Support (not ready) #221

Closed
wants to merge 9 commits into
from

Conversation

Projects
None yet

This branch isn't ready yet, but I've needed a place to consolidate discussion of this change, and especially to get feedback to improve it and hear what's working and what isn't for people.

The goal is to make Pow launch Foreman in a way that's as close to foreman start as possible, so that it's super easy to work with multiple Heroku apps on my box in development.

Currently, it requires some changes to Foreman that I've submitted to Foreman (ddollar/foreman#70), but that haven't yet been accepted (and which may not be if they just don't fit the character of Foreman, which is fine and would imply a fairly different approach is needed in Pow which may not be perfectly compatible with command-line use of Foreman).

Also, as @lackac rightly pointed out on #154 where I originally requested a pull, I'm still missing good error handling which will give some reasonable message when things like a compatible version of Foreman aren't found.

Most of the other feedback I've had has come from @lackac, @clee, and @josh on my first draft (timshadel/pow@28862b7) of Foreman support in Pow. I could definitely use more.

I may wait until ddollar/foreman#70 is decided.

Here's a running list of stuff I think should be fixed or added:

  • Better error handling when Foreman isn't present or isn't compatible with this version of Pow
  • Look at changes in node-http-proxy and fix constructor to be compatible with it (and make a separate, but similar fix for the proxy portion of Pow and submit that as a separate pull request so it doesn't get mixed up with this one).
  • Make the amount of time we'll wait for each web worker to start configurable, and have a more sane default (30s is likely way too long...)
  • Offer config option of Foreman before Rack or vice versa.
  • Implement idle timeouts to shutdown apps not being used.
  • Really test app crashes which occur after launch and make sure we're rock solid on restarts
  • Consider killing child processes (just in case) when Foreman master dies; usually Foreman takes care of this really well, but I've seen it not work occasionally and I suspect I'm doing something wrong which should probably be fixed instead of just covered over with a "just-in-case kill"...

Done. I think.

  • Consider allowing Foreman apps with no web proctype to be linked into Pow. Nope. No way to handle idle time well, and plenty of other ways to launch your services (launchctl, or a Procfile.dev that's also referenced in your app's .foreman file).

timshadel added some commits Sep 8, 2011

Initial support for Foreman applications
This should handle:
  * Single and multiple concurrency under Foreman
  * Proxy requests to random "web.N" process behind Foreman, like Heroku
  * .powenv can override the Procfile used to launch Foreman
  * .foreman will decide the concurrency
  * Pow chooses a random Foreman base port between 20000-35000
Collaborator

lackac commented on 28862b7 Sep 13, 2011

Nice job! Haven't tried it yet, but it looks good on paper. :)

Owner

timshadel replied Sep 14, 2011

Big hole with this version: it pulled the PORT values from my sample apps in my test fixtures. Next version fixes it, but requires change to the Foreman gem (pull request mentioned in the commit message).

clee replied Sep 14, 2011

Yeah, I was going to say... it doesn't seem to work for me with real nodejs applications because the output isn't being flushed. Looks like you've got fixes in the pipeline for that though :)

Owner

timshadel replied Sep 14, 2011

Yeah. It's really only at first draft quality at this point; It needs some help getting up to snuff (restart mechanism, the port choice is subpar , warn if you don't define web as a proctype , etc been busy). But with some eyes I think it can make it.

Bah random ports. Listen on 0 to get a free port from the system.

Collaborator

lackac replied Sep 13, 2011

And how will we know which port to proxy to?

Owner

timshadel replied Sep 14, 2011

Yeah. I don't like it yet either. First draft stuff. @lackac hit the problem: Foreman forks the subprocess, and the Heroku contract is that the subprocess will listen on PORT. That implies that the platform knows which port to tell it to use, and there's also no requirement for the web process to report back which port the system chose for it. While I can alter Foreman to pass PORT=0 to each subprocess, back in Pow I can't learn which one was chosen... Using Unix sockets could guarantee no collisions, but then we still break the PORT contract.

Owner

timshadel replied Sep 15, 2011

So I took a stab at fixing Foreman to support -p 0 and still be able to know the port before spawning the child process. This let me subsequently ditch the random port choice in Pow. All the tests still worked. It's now a matter of if that functionality is useful to Foreman or an edge case or impl they don't think belongs there...

Collaborator

lackac replied Sep 15, 2011

I checked out the implementation today to give it a test run, and it worked wonderfully. I hope your Foreman contribution will be merged in.

Is this correct? (Testing to make sure that the Procfile.dev is being used and not this file?)

Owner

timshadel replied Sep 14, 2011

Yeah. Before I fixed up the POW_PROCFILE use, it would always launch this other app. I wanted to test that Pow chose the correct Procfile to use.

timshadel added some commits Sep 14, 2011

Grab the port from Foreman instead of test app (oops!)
The side effect of this change is that I'll know the port number before
the app may have started to listen on it. To fix that, I introduced a
'spawning' state after I know the ports, but before I've confirmed
they're working. I switch to 'ready' once all the ports are up.

This requires changes to Foreman, though. :-(  Watch ddollar/foreman#70.
Add tmp/restart.txt support
Support for tmp/always_restart.txt is probably better handled with
`web: supervisor app.js`.
Collaborator

lackac commented on src/http_server.coffee in 28862b7 Sep 15, 2011

In this order any application with a config.ru and a Procfile will be run with foreman. While I like this behavior, I think it would be a good idea to make this configurable.

Owner

timshadel replied Sep 16, 2011

Yeah. I like that idea.

timshadel added some commits Sep 19, 2011

Ask the system to choose a base port
Listen on port 0 to let the system choose a port number for us. Then use
that as the base port for Foreman.

Signed-off-by: Tim Shadel <github@timshadel.com>
Give apps 30 seconds to startup
Signed-off-by: Tim Shadel <github@timshadel.com>
HttpProxy 0.7.0 breaks stuff
Signed-off-by: Tim Shadel <github@timshadel.com>
Contributor

lackac commented Sep 21, 2011

I think your second point is covered by #220. You do need to make the same changes to your usage of http-proxy, but those are easy to do base on that pull request.

Contributor

lackac commented Sep 21, 2011

Also, I wonder what the best way is for me to contribute to this pull request. Do I just create a branch in my fork off of this and commit with mentioning the issue number?

Awesome. :-) I just granted you rights to my repo while we're working on this. Push away! :-) When we're nearly done, we'll probably want to squash and leave a clean set of commits for the final pull request.

Contributor

lackac commented Sep 22, 2011

Great! I'll have some time on the weekend to work on it.

Contributor

josh commented Sep 27, 2011

One thought, why do we need to use the ruby foreman lib at all? Can't we implement the Procfile spec in node? nack has a few work arounds to avoid needing to state extra gems for your application. Depending on on a foreman binary in you path and installed with the correct ruby sounds like a pain. It also seems weird that we'd need to fork a ruby process from node then fork more web processes from there. The current implementation doesn't seem to function that much differently than pow's proxy to port with a foreman server.

(hinting at some node-foreman project)

Contributor

josh commented Sep 28, 2011

I thought of a name so I had to create a repo for it. I think thats why I start all projects.

https://github.com/josh/norman

Contributor

lackac commented Sep 28, 2011

That's right. I thought of the same while working on the error handling for the current implementation. (Nothing pushed yet because it's half-baked.) My impression was that a solution like this would be more difficult to build and maintain. Although you're right that a dependency on foreman installed for the correct ruby is not optimal either. My idea for a name was thirteen. :)

Contributor

lackac commented Sep 28, 2011

Now I see that @timshadel agrees with you, @josh. See ddollar/foreman#70

Yeah, David and I talked about that approach (ignore Ruby, just work straight from Procfile/.env) late last week. As long as the way that Foreman creates the ENV for processes doesn't change, then a separate implementation can totally work. I almost went with norman as a name as well. :-)

I figure that this'll be kinda like Nack in that it'll launch the procs defined in Procfile, keep them up, shut them down, and also provide an HTTP proxy on a port that'll balance across the web.N workers like Heroku does. Between nack, supervisor, and cluster there's plenty of examples on how to robustly launch subprocesses.

We'll have to ensure that the env we use is close enough to the command-line that when the Procfile looks something like

web: java $JAVA_OPTS -jar target/dependency/jetty-runner.jar --port $PORT target/*.war

that the java in PATH and the JAVA_OPTS that the user would see on the command-line are present when launching the child procs. At that point, Pow changes look a lot like invoking Nack: proxy the request to the pool-ish thing that's acting as the single entry point for the multiple backend processes, and assume the pool-ish thing can start/stop.

Hey @josh -- I took a look at josh/norman over the weekend, and it looks like a great start. Are you interested in maintaining the canonical repo for Procfile stuff, or would you prefer to have me take your start and then go make a repo that I maintain? :-)

Contributor

josh commented Oct 3, 2011

@timshadel I want to see if I can get this thing on pair with foreman. Basically make the norman bin almost as usable. It won't need all the logging stuff which is like half the foreman codebase. Most of my concerns are around robustness. Dealing with process crashes, start up errors and proxying errors. All that stuff needs to be exposed similar to nack's apis so pow can respond to it.

@josh sounds great. You've hit on all the things that matter to me. Ping me if I can be of any help. Thx.

Contributor

josh commented Oct 3, 2011

@timshadel please do fork :octocat:.

I apologize, but I'm trying to wrap my head around the current status of Procfile and/or Foreman support in Pow.

If I understand correctly:

  • the timshadel:master fork of pow basically has foreman integrated and working
  • there may be weirdness implementing a ruby dependency in Pow, which is Node.js, so @josh started to re-implement Foreman in Node (minus much of the extraneous logging stuff) and called it noreman

My question is, @timshadel, does your current pow fork use the new noreman project or is it still pulling in the Foreman ruby dependency? Also, is there any chance you could rebase it on the lastest pow HEAD?

As I have time, I'm helping @josh on the re-implement of Foreman in Node. After talking with Foreman's maintainer, he preferred that option. Likely, the changes to Pow will be different when they incorporate josh/norman as a launching mechanism. I'll start on that as we finish up the norman implementation.

dbrock commented Dec 14, 2011

Any progress on this?

I really want to see it happen, so I just wanted to say thanks to everybody who’s working on it— sounds like you’re doing an awesome job.

I haven't been able to touch it for a month. It's still on my radar, but my window of available time closed just as we settled on a more palatable approach. I can't pick it up again until next month. You can checkout both the original commits, and josh/norman to see how the current code looks.

The basic steps are to make norman a solid launcher that reads Foreman file formats, and then create an appropriate adapter inside pow that calls norman in a way fairly similar to how it calls nack. There's more nuance to it, but that's the basic idea.

@timshadel Is this still on your radar?

I'd love to see this become reality. It makes it even easier for developers using Heroku to use pow in the dev environment.

Thank you for all the work so far.

sborsje commented Mar 26, 2012

+1! This is awesome!

Yeah, this is definitely still on my radar. Looks like there's actually a chance it may bubble to the top in a few weeks. I'll keep you posted.

Any news?

Hi, what needs to be done to get this over the line? Anything we can help out with?

dbrock commented Jun 5, 2012

I desperately want this. Anyone know what’s missing in https://github.com/josh/norman right now?

In the meantime, at least with Pow 0.4 you can do something like this:

echo 7000 > .port
ln -s $PWD/.port ~/.pow/`basename $PWD`
foreman start -p `cat .port`

Peeja commented Jul 29, 2012

Listens in…

I've finally had some time to get back to this in the last few days (more info below).

@dbrock's workaround is great. Here's a slight mod to avoid the -p every time:

# one time
echo 7000 > .port
ln -s $PWD/.port ~/.pow/`basename $PWD`
echo "port: 7000" > .foreman

# every time
foreman start

I've got a bunch more done on timshadel/norman@61078e1 and started a pass at putting timshadel/pow@4b7f591 on top of Pow 0.4.0. The main approach I've taken is to split out the Nack-specific stuff into a subclass, and create a subclass that starts and stops the Procfile-based pool instead of the Nack pool.

Bugs to work out, help accepted. :-)

Peeja commented Sep 9, 2012

👋 Help offered. What can I do?

I think the main thing is trying my norman repo above on one of your projects, and then trying my updated pow fork timshadel/pow@1e5f08e and see if you can fix some of the tests.

I'm getting a mix of tests that fail due to race conditions (event happens before test attaches the handler), and those that like broke due to my changes. I've commented out chunks of the tests to focus on a few at a time.

If you get some fixed, I can just push them up to my fork. When it's done we can squash down the changes to the set that is a clean diff to Pow 0.4.0 and ask them to pull.

Peeja commented Sep 10, 2012

Awesome. I haven't looked into Pow's code much yet; what's the best way to use a fork? I've currently got the vanilla get.pow.cx install.

Something akin to this will get you started...(but I just made up those in the browser from memory...)

git clone https://github.com/timshadel/pow.git
git checkout -b procfile procfile
cake build
cake test

This looks amazing.

idan commented Oct 9, 2012

Want. This would open the door to pow-in-front-of-WSGI.

👍 Want.

👍

👍

👍

👍

jgarber commented Apr 24, 2013

👍

dlikhten commented May 1, 2013

👍

rheaton commented May 1, 2013

👍

zilkey commented May 1, 2013

👍

👍

👍

I haven't moved on this—even my basic test fixes in #309 from almost a year ago haven't moved anywhere, so it's not worth the effort.

See where you guys can get to outside of Pow with this article by my friend @jaketrent and the hack from above to make foreman happy without any params.

@timshadel timshadel closed this Jun 5, 2013

riywo commented Jul 29, 2013

FYI: I developed anypow to use Pow in any web application and anypow can be used by foreman.

https://github.com/riywo/anypow
http://weblog.riywo.com/post/56770391548

Thanks!

hakunin commented Jan 20, 2015

Too bad this didn't make it :( any alternatives you guys are using in 2015?

dlee commented Feb 25, 2015

@hakunin I think the best alternative right now is invoker (http://invoker.codemancers.com/). The only thing it's missing that pow has is automatic app startup upon visit.

hakunin commented Feb 25, 2015

Thanks for the tip @dlee, invoker looks fantastic!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment