Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Add a way to specify env variables for processes. #122

Merged
merged 7 commits into from

3 participants

@ametaireau

This uses the python format syntax, so it's possible to convert (for instance) int arguments to hex ones.

@benoitc

looks good. maybe adding some variables before pushing it would be better? Also a quick test.

@ametaireau

Yeah, what are the other variables you're thinking about?

@benoitc

numprocesses come to my mind, userid, groupid, maybe some vars from the ini: endpoint uri , ... ?

@tarekziade
Owner

@ametaireau +1 for the test : you can change the test_cluster to quickly validate this

@benoitc I think it's simpler to be able to pass the whole ini file rather than cherry picking its value in different variable

@ametaireau
ametaireau added some commits
@ametaireau ametaireau Merge branch 'master' into process-variables 2fac5ec
@ametaireau ametaireau Add the ability to extend command parameters.
This allows to replace {varname} by its value before spawning the process.
Environment variables can be acceded by using {env.VARIABLE}.
e9c39b9
@ametaireau ametaireau merge with master 2955360
@ametaireau

This should all be good for merge now :-)

circus/process.py
((43 lines not shown))
- self.started = time.time()
+ if self.watcher is not None:
+ for option in self.watcher.optnames:
+ if option not in format_kwargs:
+ format_kwargs[option] = getattr(self.watcher, option)
+
+ cmd = self.cmd.format(**format_kwargs)
+
+ if '$WID' in cmd or (self.args and '$WID' in self.args):
+ msg = "Using $WID in the command is deprecated. You should use "\
+ "the python string format instead. In you case, this means "\
+ "replacing the $WID in your command by {wid}."
+
@tarekziade Owner

I am not sure we want {wid} neither $WID, or the one I current use for sockets: ${socket:xxx}. The first two ones because they are confusing in a commande line,

I vote for the Gnu Make standar everywhere: $(WID)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@ametaireau

This sould make it.

@tarekziade tarekziade merged commit 461c0b7 into master
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on May 19, 2012
  1. @ametaireau

    Add a way to specify env variables for processes.

    ametaireau authored
    This uses the python format syntax.
Commits on May 22, 2012
  1. @ametaireau
Commits on Jun 18, 2012
  1. @ametaireau
Commits on Jun 19, 2012
  1. @ametaireau

    Add the ability to extend command parameters.

    ametaireau authored
    This allows to replace {varname} by its value before spawning the process.
    Environment variables can be acceded by using {env.VARIABLE}.
  2. @ametaireau

    merge with master

    ametaireau authored
  3. @ametaireau

    Use the GNU Make standard to specify arguments.

    ametaireau authored
    That is, replace {var} by $(VAR)
  4. @ametaireau
This page is out of date. Refresh to see the latest.
View
115 circus/process.py
@@ -7,17 +7,20 @@
except ImportError:
# Python on Solaris compiled with Sun Studio doesn't have ctypes
ctypes = None # NOQA
+
import errno
import os
import resource
from subprocess import PIPE
import time
import shlex
+import warnings
from psutil import Popen, STATUS_ZOMBIE, STATUS_DEAD, NoSuchProcess
from circus.py3compat import bytestring, string_types
-from circus.util import get_info, to_uid, to_gid, debuglog, get_working_dir
+from circus.util import (get_info, to_uid, to_gid, debuglog, get_working_dir,
+ ObjectDict, replace_gnu_args)
from circus import logger
@@ -38,8 +41,9 @@ class Process(object):
- **wid**: the process unique identifier. This value will be used to
replace the *$WID* string in the command line if present.
- - **cmd**: the command to run. May contain *$WID*, which will be
- replaced by **wid**.
+ - **cmd**: the command to run. May contain any of the variables available
+ that are being passed to this class. They will be replaced using the
+ python format syntax.
- **args**: the arguments for the command to run. Can be a list or
a string. If **args** is a string, it's splitted using
@@ -74,32 +78,26 @@ class Process(object):
"""
def __init__(self, wid, cmd, args=None, working_dir=None, shell=False,
uid=None, gid=None, env=None, rlimits=None, executable=None,
- use_fds=False):
- self.wid = wid
- self.use_fds = use_fds
+ use_fds=False, watcher=None, spawn=True):
- if working_dir is None:
- self.working_dir = get_working_dir()
- else:
- self.working_dir = working_dir
+ self.wid = wid
+ self.cmd = cmd
+ self.args = args
+ self.working_dir = working_dir or get_working_dir()
self.shell = shell
- self.env = env
-
- if rlimits is not None:
- self.rlimits = rlimits
- else:
- self.rlimits = {}
+ self.uid = to_uid(uid) if uid else None
+ self.gid = to_gid(gid) if gid else None
+ self.env = env or {}
+ self.rlimits = rlimits or {}
+ self.executable = executable
+ self.use_fds = use_fds
+ self.watcher = watcher
- self.cmd = cmd.replace('$WID', str(self.wid))
- if uid is None:
- self.uid = None
- else:
- self.uid = to_uid(uid)
+ if spawn:
+ self.spawn()
- if gid is None:
- self.gid = None
- else:
- self.gid = to_gid(gid)
+ def spawn(self):
+ args = self.format_args()
def preexec_fn():
os.setsid()
@@ -124,29 +122,60 @@ def preexec_fn():
if self.uid:
os.setuid(self.uid)
- logger.debug('cmd: ' + bytestring(cmd))
- logger.debug('args: ' + str(args))
-
- if args is not None:
- if isinstance(args, string_types):
- args_ = shlex.split(bytestring(args))
- else:
- args_ = [bytestring(arg) for arg in args]
-
- args_ = shlex.split(bytestring(cmd)) + args_
- else:
- args_ = shlex.split(bytestring(cmd))
-
- logger.debug("process args: %s", args_)
- logger.debug('Running %r' % ' '.join(args_))
-
- self._worker = Popen(args_, cwd=self.working_dir,
+ self._worker = Popen(args, cwd=self.working_dir,
shell=self.shell, preexec_fn=preexec_fn,
env=self.env, close_fds=not self.use_fds,
- stdout=PIPE, stderr=PIPE, executable=executable)
+ stdout=PIPE, stderr=PIPE,
+ executable=self.executable)
self.started = time.time()
+ def format_args(self):
+ """ It's possible to use environment variables and some other variables
+ that are available in this context, when spawning the processes.
+ """
+ logger.debug('cmd: ' + bytestring(self.cmd))
+ logger.debug('args: ' + str(self.args))
+
+ current_env = ObjectDict(self.env.copy())
+ current_env.update(os.environ)
+
+ format_kwargs = {
+ 'wid': self.wid, 'shell': self.shell, 'args': self.args,
+ 'env': current_env, 'working_dir': self.working_dir,
+ 'uid': self.uid, 'gid': self.gid, 'rlimits': self.rlimits,
+ 'executable': self.executable, 'use_fds': self.use_fds}
+
+ if self.watcher is not None:
+ for option in self.watcher.optnames:
+ if option not in format_kwargs\
+ and hasattr(self.watcher, option):
+ format_kwargs[option] = getattr(self.watcher, option)
+
+ cmd = replace_gnu_args(self.cmd, **format_kwargs)
+
+ if '$WID' in cmd or (self.args and '$WID' in self.args):
+ msg = "Using $WID in the command is deprecated. You should use "\
+ "the python string format instead. In you case, this means "\
+ "replacing the $WID in your command by $(WID)."
+
+ warnings.warn(msg, DeprecationWarning)
+ self.cmd = cmd.replace('$WID', str(self.wid))
+
+ if self.args is not None:
+ if isinstance(self.args, string_types):
+ args = shlex.split(bytestring(replace_gnu_args(self.args,
+ **format_kwargs)))
+ else:
+ args = [bytestring(replace_gnu_args(arg, **format_kwargs))\
+ for arg in self.args]
+ args = shlex.split(bytestring(cmd)) + args
+ else:
+ args = shlex.split(bytestring(cmd))
+
+ logger.debug("process args: %s", args)
+ return args
+
@debuglog
def poll(self):
return self._worker.poll()
View
15 circus/tests/test_process.py
@@ -71,3 +71,18 @@ def test_comparison(self):
p1.stop()
p2.stop()
+
+ def test_process_parameters(self):
+
+ # all the options passed to the process should be available by the
+ # command / process
+
+ p1 = Process('1', 'make-me-a-coffee', '$(WID) --type $(ENV.TYPE)',
+ shell=False, uid=7, gid=6, spawn=False,
+ env={'TYPE': 'macchiato'})
+
+ self.assertEquals(['make-me-a-coffee', '1', '--type', 'macchiato'],
+ p1.format_args())
+
+ p2 = Process('1', 'yeah $(WID)', spawn=False)
+ self.assertEquals(['yeah', '1'], p2.format_args())
View
8 circus/tests/test_util.py
@@ -6,7 +6,7 @@
from psutil import Popen
from circus.util import (get_info, bytes2human, to_bool, parse_env,
- env_to_str, to_uid, to_gid)
+ env_to_str, to_uid, to_gid, replace_gnu_args)
class TestUtil(unittest.TestCase):
@@ -87,3 +87,9 @@ def gid_min_max():
self.assertRaises(KeyError, getpwuid, uid_min - 1)
self.assertRaises(KeyError, getgrgid, gid_max + 1)
self.assertRaises(KeyError, getgrgid, gid_min - 1)
+
+ def test_replace_gnu_args(self):
+ self.assertEquals('foobar', replace_gnu_args('$(TEST)', test='foobar'))
+ self.assertEquals('foobar', replace_gnu_args('$(TEST)', TEST='foobar'))
+ self.assertEquals('foo, foobar, baz',
+ replace_gnu_args('foo, $(TEST), baz', TEST='foobar'))
View
14 circus/util.py
@@ -5,6 +5,7 @@
import pwd
import fcntl
from functools import wraps
+import re
import sys
import shlex
@@ -339,3 +340,16 @@ def resolve_name(name):
raise ImportError(exc)
return ret
+
+
+GNU_VAR_STD = re.compile(r'\$\((?P<variable>.*?)\)')
+
+
+def replace_gnu_args(string, **options):
+ options = dict([(k.upper(), v) for (k, v) in options.items()])
+ return GNU_VAR_STD.sub(r'{\1}', string).format(**options)
+
+
+class ObjectDict(dict):
+ def __getattr__(self, item):
+ return self[item]
View
3  circus/watcher.py
@@ -319,7 +319,8 @@ def _repl(matchobj):
args=self.args, working_dir=self.working_dir,
shell=self.shell, uid=self.uid, gid=self.gid,
env=self.env, rlimits=self.rlimits,
- executable=self.executable, use_fds=self.use_sockets)
+ executable=self.executable, use_fds=self.use_sockets,
+ watcher=self)
# stream stderr/stdout if configured
if self.stdout_redirector is not None:
View
25 docs/source/configuration.rst
@@ -13,7 +13,7 @@ Example::
[watcher:myprogram]
cmd = python
- args = -u myprogram.py $WID
+ args = -u myprogram.py $(WID) $(ENV.VAR)
warmup_delay = 0
numprocesses = 5
@@ -75,7 +75,11 @@ watcher:NAME - as many sections as you want
**cmd**
The executable program to run.
**args**
- Command-line arguments to pass to the program
+ Command-line arguments to pass to the program. You can use the python
+ format syntax here to build the parameters. Environment variables are
+ available, as well as the worker id and the environment variables that
+ you passed, if any, with the "env" parameter. See
+ :ref:`formating_cmd` for more information on this.
**shell**
If True, the processes are run in the shell (default: False)
**working_dir**
@@ -178,3 +182,20 @@ socket:NAME - as many sections as you want
Once a socket is created, the *${socket:NAME}* string can be used in the
command of a watcher. Circus will replace it by the FD value.
+
+.. _formating_cmd:
+Formating the commands and arguments with dynamic variables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As you may have seen, it is possible to pass some information that are computed
+dynamically when running the processes. Among other things, you can get the
+worker id (WID) and all the options that are passed to the :class:`Process`.
+Additionally, it is possible to access the options passed to the
+:class:`Watcher` which instanciated the process.
+
+For instance, if you want to access some variables that are contained in the
+environment, you would need to do it with a setting like this::
+
+ cmd = "make-me-a-coffee --sugar $(ENV.SUGAR_AMOUNT)"
+
+This works with both `cmd` and `args`.
Something went wrong with that request. Please try again.