Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Override environment on a per-OS-process level #13

Closed
j4mie opened this Issue Jan 9, 2013 · 7 comments

Comments

Projects
None yet
2 participants

j4mie commented Jan 9, 2013

First, thanks for the work on Gaffer and apologies if this is a stupid question!

I'm trying to figure out how I would build something similar to Heroku using Gaffer for process management.

I think I need a way to inject different variables into the environment per OS process, rather than per process type.

For example:

web: gunicorn my:app -b 0.0.0.0:$PORT -w 3

If I'm running two instances of the web process type on the same machine, they'll need different values of $PORT (presumably from some global service that tracks free/used ports on a machine) or they won't both be able to start. As far as I can see from the docs, I can only set env vars at the level of the process type.

Could this be done with a plugin, perhaps?

Thanks for any help!

Owner

benoitc commented Jan 14, 2013

@j4mie in 0.4x version it can be done by having each "web" process in their own group. This is what does gaffer load command. In the new upcoming version off gaffer (sometimes tomorrow) it will be even easier since each processes templates will grouped by application (with a default app named "system").

Hope it helps. If you have more questions don't hesitate.

Owner

benoitc commented Jan 18, 2013

@j4mie btw do you mean 2 web: line in the same procfile or two different procfiles?

j4mie commented Jan 18, 2013

Hi, thanks for the reply.

I mean one web: line in a single procfile. Then I want to scale up by adding another OS process on the same machine. I need a way to inject a different value for $PORT into the second OS process.

I may have misunderstood the way Gaffer works.. I'll try to investigate further over the weekend and give a concrete example.

Cheers

j4mie commented Jan 18, 2013

OK - forgetting about Procfiles entirely, let me try to explain via the HTTP API.

So, imagine gafferd is running. I run the following with curl:

curl http://localhost:5000/processes -XPOST -d '{"name": "web", "cmd": "gunicorn", "args": ["app:app", "--bind", "127.0.0.1:$PORT"], "env": {"PORT": "9000"}, "numprocesses": 0}'

I then start one instance of this process:

curl http://localhost:5000/processes/web/_add/1 -XPOST

Now, if I try to start another one:

curl http://localhost:5000/processes/web/_add/1 -XPOST

It will fail to start because the first OS process is already bound to port 9000.

What I want (I think) is to be able to say:

curl http://localhost:5000/processes/web/_add/1 -XPOST -d '{"env": {"PORT": "9000"}}'

And then later,

curl http://localhost:5000/processes/web/_add/1 -XPOST -d '{"env": {"PORT": "9001"}}'

So the environment specified in the _add call should override the one in the process template.

Does that make sense? :)

Owner

benoitc commented Jan 24, 2013

So in current head of gaffer you can do:

$ curl http://localhost:5000/processes -XPOST -d '{"name": "app1:web", "cmd": "gunicorn", "args": ["app:app", "--bind", "127.0.0.1:$PORT"], "env": {"PORT": "9000"}, "numprocesses": 0}'
$ curl http://localhost:5000/processes -XPOST -d '{"name": "app2:web", "cmd": "gunicorn", "args": ["app:app", "--bind", "127.0.0.1:$PORT"], "env": {"PORT": "9000"}, "numprocesses": 0}'

and then:

$ curl http://localhost:5000/processes/app1:web/_add/1
$ curl http://localhost:5000/processes/app2:web/_add/1

New planned api (actually the one in the refactor branch does:

 $ curl http://localhost:5000/apps/app1 -XPOST -d '{"name": "web", "cmd": "gunicorn", "args": ["app:app", "--bind", "127.0.0.1:$PORT"], "env": {"PORT": "9000"}, "numprocesses": 0}'
 $ curl http://localhost:5000/apps/app2 -XPOST -d '{"name": "web", "cmd": "gunicorn", "args": ["app:app", "--bind", "127.0.0.1:$PORT"], "env": {"PORT": "9000"}, "numprocesses": 0}'

and then:

$ curl http://localhost:5000/apps/app1/web/scale -d '{"scale": 1}
$ curl http://localhost:5000/apps/app2/web/scale  -d '{"scale": 1}

Note: `apps``will be renamed "templates" soon.

Quite similare, but this time the application is a first grade citizen where by default the application is named system .

I guess what you want is to set a process template that can be reused to launch differerent kind of processes. I did think about that possibility too but I failed to find a clean api to then distinct the process among others and how we can have a simple API for that. Maybe we could do that:

  1. create a process template POST /apps/<appname> {.. } with an option "standalone" to override auto scaling rules reusing the default template
  2. POST /apps/<appname>/<templatename>/new { "env": {}, ... } to start a new process with this name but diffferent variable. In this case the method "new" would be used and the PID will be returned in the response without waiting the effective start. Also since this process act as standaloe, the scale method will be forbidden and max number of processes for that kind of process will be 1.

What do you think?

Owner

benoitc commented Mar 2, 2013

feature added in 13db684 & b217c6d. Now you can commit a single process using a job config.

@benoitc benoitc closed this Mar 2, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment