Pueue is a command line queue management tool for sequential and parallel execution of long running tasks. Not being bound to any terminal it is possible to check on your processes from any terminal or using the API. And the best part is that the queue will be processed by the daemon, even if you exit your ssh session.
It doesn't work for remote machines though, as the communication happens via unix sockets. You need to
ssh into your machine to communicate with the daemon.
It provides functionality for:
- Easy output inspection.
- Interaction with running processes
- Manipulation of the scheduled task order
- Running multiple tasks at once (You can decide how many concurrent tasks you want to run)
Why should I use it?
Almost everybody who lives on the commandline knows the situation, when one needs to unzip or transfer huge amounts of data in different directories.
This normally ends with about 10 open terminals/tmux sessions and an overchallenged hard drive.
Pueue is specifically designed for these situations. It executes long running tasks in their respective directories, without being bound to any terminal.
Here a few possible applications:
- Compression tasks
- Movie encoding
- Copying huge amounts of stuff
If I got your attention, feel free to give it a try!
If you think this is awesome, help me, join the development and create some PRs or suggest some improvements.
I'm always open to suggestions and already implemented a few user requested features.
There are three different ways to install pueue.
- Use an Arch Linux AUR package manager i.e Yaourt:
yaourt -S pueue-git. This will deploy the service file automatically.
- Install by using pip:
pip install pueue.
- Clone the repository and execute
python setup.py install.
How to use it:
There is a help option (-h) for all commands, but I'll list them here anyway.
pueue --daemon Starts the daemon. The daemon will try to load any queue from a previous session.
pueue --no-daemon Start the daemon in the current terminal.
pueue --stop-daemon Daemon will shut down after killing all processes.
pueue status Show the current state of the daemon and the processing state of the queue.
pueue reset Remove all commands from the queue, kill the current process and reset the queue index to 0.
pueue clear Remove all
failed commands from the queue. This will rotate logs as well.
pueue config This command allows to set different config values without editing the config file and restarting the daemon. Look at
pueue config -h for more information.
pueue add 'command' Add a command to the queue. It can be used without quotation marks, but a
-- may be necessary if you want to pass parameters (
pueue add -- ls -al). Also note that bash specific syntax like
; might cause unwanted behavior without quotation marks.
pueue edit [key] Edit the command of a specific
stashed entry in you
pueue remove [keys...] Remove the specified entries. Running processes can't be removed.
pueue stash [keys...] Stash queued entries for later processing. They won't be processed by the daemon, but can be manually enqueued again.
pueue enqueue [keys...] Enqueue stashed entries. The entries will be normally processed.
pueue switch [key_1] [key_2] Switch the entries at position
pueue start [keys...] This command has three different behaviors, depending on if and what kind of keys are given:
1. If keys of paused processes are given, the processes will be started (
SIGCONT), this happens even if the daemon is paused or if the max number of concurrent processes is reached.
2. If keys of queued or stashed processes are given, new processes will be spawned for those entries. This happens even if the daemon is paused or the max amount of processes is exceeded.
3. Otherwise the daemon will start to process the queue. This will start all paused processes (
pueue pause [keys...] --wait This command has two different behaviors, depending on if keys are given:
1. If keys are given, pause the specified processes by sending a
2. Otherwise stop processing the queue and pause all running processes. If the
--wait flag is set, the daemon will pause, but all running processes will finish on their own.
pueue restart [keys...] Enqueue the specified
failed processes again.
pueue kill [keys...] -s [signal] -a This command tries to copy the behaviour of the Linux
kill command. It will send a signal (default is
sigterm) to the specified processes.
- Available signals can be viewed with
pueue kill -hunder the
- Either an int
15, the full name
sigterm/SIGTERMor the abbreviation
term/TERMcan be used.
- Be aware that by default the signal will only be sent to the children of the shell process , i.e. if you send a
sigintright after starting
sleep 5 ; sleep 10the
sleep 5process will be stopped and the
sleep 10will spawn afterwards.
- To send the signal to the parent process as well you need to add the
- If keys are given, the signal will be send to the specified processes.
- If no keys are given, send the signal to all running processes. If the signal is
sigkillthe daemon will be paused.
pueue show --watch -k [key] Show the output of
key or the oldest running process.
show --watch will continually show the stdout output of the subprocess in a
--watch will print the stderr as well. This can be useful if the subprocess prompts for user input (This is often piped to stderr).
pueue log [keys...] Print the output and status of all finished processes or of the specified finished processes.
pueue send [input] Send a string to the subprocess's stdin. In case a process prompts for user input, you can use this to interact with the subprocess.
The stdin pipe is flushed after every
send command. To simulate a
\n you need to add a newline in your string:
pueue send 'y '
The configuration file of pueue is located in
[default] resumeAfterStart = False maxProcesses = 1 customShell = default [log] logTime = 1209600
resumeAfterStart = FalseIf you want pueue to instantly resume a queue from the last session, set this value to
maxProcesses = 1Determines how many tasks should be processed concurrently.
customShell = defaultThe path to the custom shell that should be used. Enter
In case you use any other shell, every command will be launched from an interactive session.
[customShell, '-i', '-c', command].
This is useful if you want to use aliases or environment variables.
Warning!!!: This is kind of experimental and really hard to test! Be ready to encounter bugs!!
logTime = 1209600Old logs will be deleted after the time specified in your config.
All logs can be found in
~/.shared/pueue/*.log. Logs of previous pueue sessions will be rotated and contain a timestamp in the name.
In case the daemon fails or something goes wrong, there is a separate log for the daemon at
If the daemon crashes, please send the stack trace from this log!
If you use systemd and don't install pueue with yaourt, place
Afterwards every user can start/enable their own session with:
systemctl --user start pueue.service systemctl --user enable pueue.service
make completion, which copies the completion file to
/usr/share/zsh/site-functions/ or place
_pueue in a folder, that is contained in your
FPATH environment variable.
This script will be probably added to zsh-users/zsh-completions, when it is finished.
Pueue status querying script for
If you want to see the status of the last 4 entries in your status bar, just use the
Regards to Robpol86 for providing the awesome
And thanks to thesharp for the extremely useful
Pueue already works and is used frequently. There might be some small bugs, but I didn't encounter something serious in quite a while.
Copyright © 2016 Arne Beer (@Nukesor)