Ability to submit a single command like sge_batch

[pipitone]

sge_batch has the handy feature of being able to stick a call to it in front of a command line and that command gets submitted to the queue, like so:

sge_batch echo "hello world" 

Should qbatch have this feature so that it can be a drop-in replacement?

It will likely be annoying to make argparse handle this case, but I can think of a few other ways to achieve this same feature:

1. Pipe the command to qbatch, e.g. echo 'echo "hello world"' | qbatch -. It works, but it's unwieldy.
2. Create a wrapper script that calls qbatch as above. We may have to be a little clever if we also want users to be able to pass arguments to qbatch.

[gdevenyi]

Do we want to encourage people to rely on the old sge_batch paradigm?

I'd be okay with a basic wrapper that allows qbatch to replace for this application, using the default qbatch settings (modified by environment) but not any argument passing.

[pipitone]

Do we want to encourage people to rely on the old sge_batch paradigm?

I do see how it's useful. Even without sge_batch, there are people that use fsl_sub in the same way.

I'd be okay with a basic wrapper that allows qbatch to replace for this application, using the default qbatch settings (modified by environment) but not any argument passing.

Yeah, I agree. I'll leave this for post 1.0 though.

[gdevenyi]

Do we have a name for this command? Pretty sure I can hack a bash wrapper really easily..

[pipitone]

Hmm.. What about qbatch_sub or qbsub?

Are you thinking of doing: echo "$@" | qbatch -? Almost doesn't seem worth it. :-)

[gdevenyi]

yes, that is indeed what I'm thinking, I agree, it doesn't seem like it's worth it, but if it can attract users who have that workflow, might as well offer it.

[gdevenyi]

Implement via the "--" heuristic mentioned in #90

@pipitone you had previously implemented -- for the options method, can you repurpose that argparse magic for single-job submission?

[pipitone]

There's an ambiguity here: with a single positional argument it's unclear whether the argument is a file of commands or a command itself, e.g. qbatch ./process might be running commands in the file process or running the script process.

What about a "subcommand" to disambiguate:

Usage:
    qbatch [options] <command_file>
    qbatch [options] cmd <command> [command_opts...]

e.g. qbatch ./process vs qbatch cmd ./process. With options: qbatch --ppj 8 ./process vs qbatch --ppj 8 -- cmd ./process -p

Alternatively, we could use the double dash (--) separator as indicating that the user is submitting a command to run, not a file. eg. qbatch ./process vs ./qbatch -- ./process. Two issues with that: argparse doesn't naturally handle this case so we'd have to do our own gymnastics, and it fails with command files that start with a dash, e.g. qbatch -- -process (but I think we can all agree we hate people that would try to do that).

[andrewjanke]

I'm all for using the -- version as this is common in other unix things.

I'm sure I've used this before in argparse but couldn't find it so resorted
to the internet (which is likely what I did the first time too). See the
first answer here:

http://stackoverflow.com/questions/25872515/python-argparse-treat-arguments-in-different-ways

On 29 May 2016 at 21:43, Jon Pipitone notifications@github.com wrote:

There's an ambiguity here: with a single positional argument it's unclear
whether the argument is a file of commands or a command itself, e.g. qbatch
./process might be running commands in the file process or running the
script process.

What about a "subcommand" to disambiguate:

Usage:
qbatch [options] <command_file>
qbatch [options] cmd [command_opts...]

e.g. qbatch ./process vs qbatch cmd ./process. With options: qbatch --ppj
8 ./process vs qbatch --ppj 8 -- cmd ./process -p

Alternatively, we could use the double dash (--) separator as indicating
that the user is submitting a command to run, not a file. eg. qbatch
./process vs ./qbatch -- ./process. Two issues with that: argparse
doesn't naturally handle this case so we'd have to do our own gymnastics,
and it fails with command files that start with a dash, e.g. qbatch --
-process (but I think we can all agree we hate people that would try to
do that).

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#35 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AATwit1D0iol4UHE0AaQm6iC8ORNWPduks5qGXvJgaJpZM4HqaFT
.

[andrewjanke]

Another very interesting option is docopt, I've used it a few times when
creating quick hacks, it's eerily similar to GetOpt::Tabular, but even
better.

http://docopt.org/

On 30 May 2016 at 06:25, Andrew Janke a.janke@gmail.com wrote:

I'm all for using the -- version as this is common in other unix things.

I'm sure I've used this before in argparse but couldn't find it so
resorted to the internet (which is likely what I did the first time too).
See the first answer here:

http://stackoverflow.com/questions/25872515/python-argparse-treat-arguments-in-different-ways

On 29 May 2016 at 21:43, Jon Pipitone notifications@github.com wrote:

There's an ambiguity here: with a single positional argument it's unclear
whether the argument is a file of commands or a command itself, e.g. qbatch
./process might be running commands in the file process or running the
script process.

What about a "subcommand" to disambiguate:

Usage:
qbatch [options] <command_file>
qbatch [options] cmd [command_opts...]

e.g. qbatch ./process vs qbatch cmd ./process. With options: qbatch
--ppj 8 ./process vs qbatch --ppj 8 -- cmd ./process -p

Alternatively, we could use the double dash (--) separator as indicating
that the user is submitting a command to run, not a file. eg. qbatch
./process vs ./qbatch -- ./process. Two issues with that: argparse
doesn't naturally handle this case so we'd have to do our own gymnastics,
and it fails with command files that start with a dash, e.g. qbatch --
-process (but I think we can all agree we hate people that would try to
do that).

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#35 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AATwit1D0iol4UHE0AaQm6iC8ORNWPduks5qGXvJgaJpZM4HqaFT
.

[pipitone]

Absolutely love docopt. I didn't go with it because I wanted the flexibility of argparse and to avoid the added dependency. Neither are very strong reasons but docopt wouldn't help us here I don't think ; we'd still need to handle parsing ambiguities ourselves.

In any case a +1 for --. I'll try it out and we can see what it's like in practice.

On May 29, 2016, at 4:31 PM, Andrew Janke notifications@github.com wrote:

Another very interesting option is docopt, I've used it a few times when
creating quick hacks, it's eerily similar to GetOpt::Tabular, but even
better.

http://docopt.org/

On 30 May 2016 at 06:25, Andrew Janke a.janke@gmail.com wrote:

I'm all for using the -- version as this is common in other unix things.

I'm sure I've used this before in argparse but couldn't find it so
resorted to the internet (which is likely what I did the first time too).
See the first answer here:

http://stackoverflow.com/questions/25872515/python-argparse-treat-arguments-in-different-ways

On 29 May 2016 at 21:43, Jon Pipitone notifications@github.com wrote:

There's an ambiguity here: with a single positional argument it's unclear
whether the argument is a file of commands or a command itself, e.g. qbatch
./process might be running commands in the file process or running the
script process.

What about a "subcommand" to disambiguate:

Usage:
qbatch [options] <command_file>
qbatch [options] cmd [command_opts...]

e.g. qbatch ./process vs qbatch cmd ./process. With options: qbatch
--ppj 8 ./process vs qbatch --ppj 8 -- cmd ./process -p

Alternatively, we could use the double dash (--) separator as indicating
that the user is submitting a command to run, not a file. eg. qbatch
./process vs ./qbatch -- ./process. Two issues with that: argparse
doesn't naturally handle this case so we'd have to do our own gymnastics,
and it fails with command files that start with a dash, e.g. qbatch --
-process (but I think we can all agree we hate people that would try to
do that).

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#35 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AATwit1D0iol4UHE0AaQm6iC8ORNWPduks5qGXvJgaJpZM4HqaFT
.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

[pipitone]

I've got something cooking over in https://github.com/pipitone/qbatch/tree/fix_single_command_submit

It works, but:

• I've disabled the check for command_file existence (need to call which)
• The usage/help doesn't reflect the change (not sure exactly how to do this nicely)
• Preserving quotations doesn't work yet (which is one reason the test fails)

In the meantime, you can use this bash wrapper:

#!/bin/bash
# Submits a single command to qbatch (like sge_batch)
#
# Usage: 
#     qbatchsub.sh [qbatch options] -- command [options]
# 
# For example, 
#   qbatchsub -o '-l walltime=30:00' --ppj 8 -- ./process.sh "hello world"
#
# This submits the command "./process.sh 'hello world'"

qbatch_args=()
cmd=()
dashdash_unseen="no"

while [[ $# > 0 ]]; do
    if [[ $1 = "--" ]]; then
        dashdash_unseen="yes"
    elif [[ ${dashdash_unseen} = "no" ]]; then
        qbatch_args+=("$1")
    else
        cmd+=( $(printf '%q' "$1") )
    fi
    shift;
done

echo "${cmd[@]}" | qbatch "${qbatch_args[@]}" -

[gdevenyi]

Hey @andrewjanke see implementation in #157

[andrewjanke]

I like! More reasons to dump my old perl script.

…

On Thu, 26 Apr 2018 at 11:52, Gabriel A. Devenyi ***@***.***> wrote: Hey @andrewjanke <https://github.com/andrewjanke> see implementation in #157 <#157> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATwivif27gPhPI4Iln1NLPOS1Rryc3Hks5tsShugaJpZM4HqaFT> .

[gdevenyi]

Great. can you do a quick look over and see if there's any other show stopping features you need? If so, open or poke some issues

[gdevenyi]

Fixed in #164