Executing multiple commands concurrently using Python asyncio
pip install concurrent-executor
# Or from the latest GitHub version
pip install git+https://github.com/DCsunset/concurrent-executor
All the following tools can handle signals as follows:
- Upon receiving one
SIGINT
(including keyboard interrupt) orSIGTERM
, the program will sendSIGTERM
to all spawned processes. - Upon receiving more than one of them, the program will instead send
SIGKILL
to kill all spawned processes.
cssh
is a command line tool provided by this package.
It is used to executing commands concurrently on remote servers vis SSH.
Use -H
or --hosts
to specify the hosts to run the commands on:
cssh -H host1 host2 ... host_n -- command
# pass extra ssh options
cssh -o="-q -4" -H host1 host2 ... host_n -- command
# read hosts from file
cssh -f hosts.txt -- command
# string interpolation (to include host name in command by {0})
cssh -H host1 host2 -- command --host {0}
# pipe to stdin
echo "yes" | cssh -H host1 host2 -- command
# Pipe file content to stdin
cssh -H host1 host2 -- command < input.txt
Note that --
is necessary to separate the options and the command.
For -o/--options
to work correctly, use =
to prevent it from being parsed as another option.
The standard input (stdin) of the cssh
process is piped to the stdin of every spawned processes.
For more details, see cssh -h
.
cexec
is another command line tool provided by this package.
It is used to execute arbitrary shell commands concurrently using template (string interpolation in Python).
The command itself can container placeholder in strings: (See Python string interpolation for more detail.)
# The variables are a, b, c in the template command
# This command creates 3 directories and write to a file in each directory
cexec -V a b c -- "mkdir {0} && echo 1 > {0}/out"
# Read variables from a file
cexec -f vars.txt -- "mkdir {0} && echo 1 > {0}/out"
# Run different commands directly
cexec -V "cmd1" "cmd2" "cmd3" -- "{}"
# pipe to stdin
echo "yes" | cexec -V a b c -- "cat -"
For more details, see cexec -h
.
It can also be used as a library:
import asyncio
from concurrent_executor.executor import SshExecutor
async def main():
hosts = ["host1", "host2"]
executor = SshExecutor(hosts)
# running concurrently
await executor.run("some_command --test")
# access stdout for all hosts (or stderr)
async for index, out in executor.stdout:
print(f"{host[index]}: {out}")
# wait until all finished
ret_codes = await executor.wait()
asyncio.run(main())
See more usage in concurrent_executor/cli.py
.
To set up the development environment, first clone this repo.
Then it's recommended to usevenv
:
# suppose PWD is the root dir of the repo
python -m venv venv
# activate the environment``
source venv/bin/activate
pip install -r requirements.txt
To deactivate, run deactivate
.
This project is licensed under AGPL-3.0. Copyright notice:
concurrent-executor
Copyright (C) 2023 DCsunset
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published
by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.