docopts API changes proposal

[Sylvain303]

following #5

I had a look on the API proposed in the branch develop (See man ./docopts.1). Which is quite different from the current docopts API.

Old API (current)

eval "$(python3 $libexec/docopts -A args -V "$version" -h "$help" : "$@")"

if ${args[--do-something]} ; then
  run_some_code
fi

current API in branch develop, generating temporary files for each option on the filesystem

tmpdir=$(echo "$help" | python3 $libexec/docopts "$@")                                                        
if [ -e $tmpdir/--do-something ]; then
    echo "Option given with argument $(cat $tmpdir/--do-something)."
fi

I think about a new API, more like the python's library could be quite nice to have.

Something like the following code snippet, (to be completed…)
Specifying the associative array for $args and some shell centric behavior have to be considered, too.

#!/bin/bash
# sourcing the API providing some bash functions ie: docopt()
# still using the python parser
source docopts.sh

parsed=$(docopt -A args "$help_string" "$@" "$version" "$options_first")
eval "$parsed"

and may be some shell helpers too:

help_string=$(get_help_string $0)

array_opt=( $(get_docopt array "${args[--multiple-time]})" )

Please comment and suggest. I will develop and propose some versions.

You may also be tempted to say: "bash is not capable to handle simply the complexity of the JSON parsed structure behind docopt. Consider rewriting your shell script directly in python…"

[fsaintjacques]

I really enjoy the propose API. One thing that I would love is self-contained bash file with no install-time python dependencies. This could be achieved à-la "make-dist" target where the python deps are embedded into a single bash file with HEREDOC.

[Sylvain303]

Could you paste a link to the code 'make-link' you mention, please?

Yes handling the dependencies can be tricky, but interesting.

Generally when bash programming is involving array, I switch to other languages. But for legacy code, or just is because it would be fun to have a docopt lib in every language. It can be done.

As mentioned in #5 I will create a new branch with an API_proposal.md file for writing the API to reach.

Array syntax is ugly in bash.

[fsaintjacques]

bash syntax as a whole is ugly, I don't mind iterating over arrays.

declare -a FILES=()
for ((i = 0; i < ${args[<src>,#]}; ++i)); do
  FILES+=(${args[<src>,$i]})
done

As for the make-dist, I just would that docopt.sh is self-contained, not need to pip install docopts/docopt. This way, just dropping the file in your project is sufficient. I could already to it with only the python file, but heh.

[Sylvain303]

API_proposal.md

What is make-dist?
I see what you mean, a sort of piggy-back with docopt.py embedded.

[Sylvain303]

In fact the current API (docopts 0.6.1+fix) is a quite good shell centric behavior:

Usage:                                                                                                     
  docopts [options] -h <msg> : [<argv>...]

Options:
  -h <msg>, --help=<msg>        The help message in docopt format.
                                If - is given, read the help message from
                                standard input.
                                If no argument is given, print docopts's own
                                help message and quit.
  -V <msg>, --version=<msg>     A version message.
                                If - is given, read the version message from
                                standard input.  If the help message is also
                                read from standard input, it is read first.
                                If no argument is given, print docopts's own
                                version message and quit.
  -O, --options-first           Disallow interspersing options and positional
                                arguments: all arguments starting from the
                                first one that does not begin with a dash will
                                be treated as positional arguments.
  -H, --no-help                 Don't handle --help and --version specially.
  -A <name>                     Export the arguments as a Bash 4.x associative
                                array called <name>.
  -s <str>, --separator=<str>   The string to use to separate the help message
                                from the version message when both are given
                                via standard input. [default: ----]

I was looking what was the colon (:) in the argument format? But it's simply parsed by docopt. :-) The behavior seems already providing mostly what we expect. Just add some shell wrappers and helpers.

I'm translating all examples available here into shell plus adding unit tests.

[Sylvain303]

I pushed an experimental version here which embed all code in a single file.

[fsaintjacques]

I meant make dist which is a non-official make target in make centric system used to build the distribution tar archive.

[Sylvain303]

OK, for "make dist". I commited a build.sh which do it. Seems working, and effective. I also coded a --auto. For laziness repetitive shell script. It is used like this:

#!/bin/bash
# Usage: doit [cmd] FILE...
#
# do somethingdo something

source ../docopts.sh --auto "$@"

for a in ${!args[@]} ; do
    echo "$a = ${args[$a]}"
done

I have to complete the doc, change a little the python docopts, and it seems working for now.

[fsaintjacques]

Small nitpick

https://github.com/Sylvain303/docopts/blob/shell-api-v2/docopts.sh#L30

Let the user decides if he wants python or python3.

[Sylvain303]

Yep of course, works also with legacy python version on legacy server… corrected.

[fsaintjacques]

👍

[Sylvain303]

I made a pull request, today #8 …

[fsaintjacques]

secondary nitpick this way of doing thing https://github.com/Sylvain303/docopts/blob/shell-api-v2/docopts.sh#L30. You don't need sed here. I'd create a function

function print_python_code() {
# auto generated
cat <<< __EOF__

content of docopt and docopts

__EOF__
}

I find this more shell-ish and parseable by human.

[Sylvain303]

about how I chose the

 python <(sed -n -e '/^### EMBEDDED/,$ s/^#> // p' "$docopt_sh_me") "$@"

instead of something:

function print_python_code() 

I didn't chose this way because of the stdin used by docopts.py. I presumably though about and stdin clash, if python read its code (docopts.py) from stdin and try to parse option $help $version from stdin too (-h -). I wont use nor encourage this way of doing, but I maintained this.

Embedding the code inside the shell script is cool, and challenging. Using <() bash's operator gives a file descriptor. And filtering by sed is cheap and fast, and lazy: I don't have to handle the temporary file.

I also fall in love (tombé en amour) with this operator when I discovered it, so I share it for curious code reader:

diff /etc/hosts <(ssh remote-server cat /etc/hosts)

human readable… advanced bash shell scripting is a bit ugly, and not well human readable. (See bash completion's code to become as master) docopts.sh proposed code is a wrapper, so it exposes a "friendly" api, but it's internal can do some tricky stuff. Most of the code I proposed is advanced and not friendly: docopt_get_eval_array() docopt_get_values()

The embedded python is not indented to be edited in place of course. It could be a base64 block, but I don't want extra forking / piping commands.

Saying that, I may try your suggestion because sed's oneliner are fragile. Also, some users may ask for compatibly with other shell, that supports different syntax. BSD's default sed may be not GNU sed for example.

Thanks for your comment.

[Sylvain303]

I did some tests: builder not modified and of course broken in both branches.

This one keeps the code at the end of the file docopts.sh but breaks --auto. A bash limitation that eval the code as it sources it, I suppose. So the function is not defined yet.

https://github.com/Sylvain303/docopts/blob/function-print-python-broken/docopts.sh#L90

This one works but, the code is not at the end of the file, it also disturbs code reading (big chunck of alien's code in the middle) and syntax highlighting.

https://github.com/Sylvain303/docopts/blob/function-print-python/docopts.sh#L25

Having code in the middle will also complicate the build.sh to match two delete boundaries…
Of course, this build.sh is a rudimentary proof of concept and could be a nice python script…

I'm not convinced, by the function print_python_code() …

But when the PR will be accepted, may be creating a dedicated issue, on this.

[fsaintjacques]

Your first try can be solved by wrapping

def main() {
    if [[ "$1" == "--auto" ]] ; then
        shift
        declare -A args
        eval "$(docopt_auto_parse "${BASH_SOURCE[1]}" "$@")"
    fi
}

And then, your last line will simply be a call to main and you can keep the order of function definitions you want.

[Sylvain303]

I don't think so:

declare -A args introduce a local scope for $args when used inside function.

That's why I used global instructions here for --auto behavior. I don't think it cant be solved simply. I used a grep -v 'declare -A' here, exactly for that purpose. It's a temporary fix. As a better solution would be to introduce a new switch in docopts.py --no-declare for example.

[CristianCantoro]

I wanted to ask two things:

• when the new API will become the default
• there will be backward-compatibility with the previous API?

Thanks

[Sylvain303]

@CristianCantoro I worked on a Proof of Concept of the proposed API. The go version with JSON support is disappointing in term of performance, for now. Successive call take a lot of time to parse docopts owns command line interface.

So I'm also waiting for some return or other proposal. It is really open, be creative!

And yes I will make a backward-compatibile version with the current API (-h <msg>), followed with a still compatibly + deprecation warning, followed with a dropped API version.

Most of the annoying thing I see in the current API, is using -h as an input switch to pass usage and both the help shortcut. For this to run in the code, some strange code is involved to handle wrong use case. A -u, --usage, or action-like with parse, or any thing else seems really better to me than: docopts -h "usage: prog [-v] [-h] PARAM..." : "$@"

But it may be really personal feeling.

My PoC handle both old an new API, and for this to work some tricks are involved to truncate argument for the parser. docopt lib don't allow both to work together and docopts as to pass extra arguments to the sub parser when it finish to parse its own argument. But I found a work around so it's OK for now. Just time consuming for backward compatibility, but it's the game too. ;-)

The actual go version is also stand-alone, no need to docopts.sh wrapper, which was part of my 2015 API proposal. I added some stand-alone examples: https://github.com/docopt/docopts/tree/master/examples/without_docopts_sh_lib

I'm dropping the idea to require docopts + docopts.sh to go to a more stand-alone version. I would like to make it feeling as if you where using native bash tools (sourcing code in bash it still a pain for me).

I think this issue is too old, I need to sum-up proposal into a new dedicated issue.

Regards,

[CristianCantoro]

Hi @Sylvain303, thanks for the explanation.

Let me begin by saying a couple of things:

• I remember that the biggest complain about the current API of docopts is that it is very different from the behavior of other instances of docopt in other languages, i.e. in general what docopt does is to create some sort of dictionary or structure whose name can be assigned at runtime (tipically args or arguments) and the provide a way to access to them while docopts provides directly the name of the variables. I would stick to using associative arrays, but for those you need bash 4.X and I think it is fundamental to support also previous versions of bash.
• I have written three examples in bash and python just for reference: example.sh, example_array.sh, and example.py, their outputs: one difference is that if some option is not used (see --foo) in the example above then it is set as empty while in python it is set as False.

@Sylvain303 said:

@CristianCantoro I worked on a Proof of Concept of the proposed API. The go version with JSON support is disappointing in term of performance, for now. Successive call take a lot of time to parse docopts owns command line interface.

I am not sure I am that worried about that, personally I typically read the values of your CLI options at the beginning of my scripts and then I assign them to some variables to be ready for use. Is the performance that bad that it would impact even this use case?

So I'm also waiting for some return or other proposal. It is really open, be creative!

What about having docopts produce the code to create (via eval) a with a custom name so that you can recover the arguments by calling it?

In this example the function name is arguments:

eval 'function arguments() {
    local __all=false
    local _b=false
    local __foo=
    local __help=false
    local __version=false

    local parname=$(echo "$1" | sed '\''s/-/_/g'\'')

    echo "${!parname}"
}'

and then you can use it like this:

$(arguments '--option')

Note that in this way the mangled variables are all confined withing the arguments function, so you don't pollute the global scope.

I have put an example here: example_function.sh.

This is the basic idea, it remains to be seen if it works with list of arguments. Also, the sed expression above should be fixed to just convert the initial dashes - to underscores _.

Most of the annoying thing I see in the current API, is using -h as an input switch to pass usage and both the help shortcut. For this to run in the code, some strange code is involved to handle wrong use case. A -u, --usage, or action-like with parse, or any thing else seems really better to me than: docopts -h "usage: prog [-v] [-h] PARAM..." : "$@"

But it may be really personal feeling.

I can live with it, what I have found difficult at first was how to pass effectively the usage string to docopts, in the end I settled with defining a variable using a here document (see example.sh#L5), which is quite similar to what you do in your example, defining the usage function.

I'm dropping the idea to require docopts + docopts.sh to go to a more stand-alone version. I would like to make it feeling as if you where using native bash tools (sourcing code in bash it still a pain for me).

I think it is better not to be required to source docopts.sh, you want your typical bash script to be standalone, it is not like having a module/library in a programming/scripting language that, once installed, knows where to look for the files to include/import. In this case, you would need to always have docopts.sh next to your script. I think this could be useful sometimes, but I wouldn't like to have to do it all the times.

C

[Sylvain303]

I close this issue and will republish some new proposal, as JSON test was to slow to be helpful in what I tested..
I will certainly propose to output bash function as @CristianCantoro suggested or completion.
I was reluctant to output function because of root shell script using docopts for obvious security reason.

This issue is now too old, it's time for something new. Thanks for your interest.