Opster syntax in python 3 #32

Open
oscarbenjamin opened this Issue May 21, 2012 · 3 comments

Comments

Projects
None yet
2 participants
Contributor

oscarbenjamin commented May 21, 2012

Following the merge of keyword-only argument introspection support in GH-31, I thought I'd open a new issue to discuss how opster's syntax in python3 should look. Python3 adds two new features relevant to the syntax used in opster: keyword only arguments and function annotations.

Keyword-only arguments

That opster should allow keyword arguments to be used for specifying options seems obvious to me. I also think that, when possible, opster should drop all support for any other type argument-option translation. Currently the opster syntax (when using introspection) looks like:

@opster.command()
def main(required_arg, optional_arg=None,
         option1=('o', False, 'help for --option1'),
         option2=('O', 'default', 'enter a value for option2'),
         *varargs, **globalopts)
    pass

Using keyword arguments allows it to be rewritten with varargs immediately following the other positional arguments

@opster.command()
def main(required_arg, optional_arg=None, *varargs,
         option1=('o', False, 'help for --option1'),
         option2=('O', 'default', 'enter a value for option2'),
         **globalopts)
    pass

The second form keeps the positional arguments together and shows them as they are used for the script. It means that opster can identify option arguments unambiguously in a way that can be easily explained in the docs by simply saying: Opster infers the positional arguments of the scripts from the positional arguments of main and the options of the script from the keyword-only arguments of main.

Using keyword-only arguments means that main(*args, **options) does exactly what you would hope without needing to wrap the main function as long as the options contains values for option1 and option2. It can be made to match perfectly with the expected behaviour by modifying the default arguments in place with e.g.:

def command():
    def wrapper(func):
        # Add command attribute
        func.command = make_command(func)
        # Replace defaults for kwonly arguments
        for long, (short, default, help) in func.__kwdefaults__.items():
            func.__kwdefaults__[long] = default
        # Return func with command attribute and modified defaults
        return func
    return wrapper

Only using keyword arguments is more robust, easier to document, produces clearer code, and allows opster's internal workings to be simpler. Since opster still supports python 2.6+ and most people still use python 2.x, I don't think it's possible to drop support for non-keyword-only arguments now, but I think it should be a target for the future. Specifically I think that when opster drops support for python 2.x, it should also drop support for the old argument syntax.

Another possibility is to require the keyword-argument only syntax in python 3.x now and allow the old syntax in python 2.x. This will avoid having backward compatibility problems later but at the expense of causing problems now. Lots of other people support both python 2.x and 3.x using 2to3 (like opster does) so anyone using opster like that will need to have a syntax that can work with opster under both python versions. Because of this I think it would be best to allow the old syntax under python 3.x for a while. It would, however, be good to have the new opster 3.x syntax prominently displayed in the docs, so that anyone new to opster is aware of it (and at least knows that the old syntax will not always be available).

Function annotations

The other python 3.x new feature that opster could choose to make use of is function annotations. The syntax of function annotations is

def f(a: b = c):
    pass

This results in f having an attribute f.__annotations__ with the value {'a': b}. Opster can use this to add the additional metadata to an argument representing an option without needing to change the default argument. This means that the default value of the option can be used as the default value for the argument and looks like:

@opster.command()
def main(required_arg, optional_arg=None, *varargs
         option1:('o', 'help for --option1') = False,
         option2:('O', 'enter a value for option2') = 'default',
         **globalopts)
    pass

The advantages of this are that opster can use an officially supported mechanism (annotations). It should be clear that the tuples of option data are there to provide information for opster and what the actual default value for option1 is. Also, the function main now really does work exactly as you would expect with main(*args, **options) in all situations without needing its defaults to be modified.

However, this syntax looks a little strange to me. It seems strange that the default value is so far away from the option name. I think it does make sense for the help string to be last since it could be quite long:

@opster.command()
def main(required_arg, optional_arg=None, *varargs
         option1:('o', 'help for --option1') = False,
         option2:('I', '--option2 has a really long help string that spans'
                ' several lines with lots of useless information') = False,
         option3:('O', 'enter a value for option3') = 'default',
         **globalopts)
    pass

I don't know whether I dislike the appearance of this because annotations are unfamiliar and I'm just used to seeing the old opster syntax. I guess opster's current syntax is a bit strange when you first see it (nothing else in python works the way that opster does). I do think, though, that it is a bad thing to have the important information, (long, short, default) separated by the long help string.

I thought that there could be a backward compatibility problem if opster released a version now that allows keyword-only arguments in python3 but without using annotations and opster later decided to use annotations. However, thinking about it, there is no backward compatibility problem supporting two different mechanisms for introspection is easy if the difference between them is well defined (just check for __annotations__) and if both syntaxes are simple and well defined (it is only the current syntax that is difficult to support because because it needs to workaround the lack of keyword-only arguments).

What do you think?

Owner

piranha commented May 22, 2012

Hm, in the end it probably would be nice to specify annotations, since in this case meaning of default values is not overloaded (which it is right now), so semantically that's cleaner. But I dislike the fact that default is going to be the last one here, it makes it harder to read code (in which case you don't usually need to read help).

Also, we can't restrict py3 to use only annotations - if you have application, which you want to work in both py2 and py3, you'll have to use current syntax.

In the end, maybe it's worth it to postpone implementation a bit. I'm not sure what's the situation globally, but I don't see anyone around me to work with python 3. :)

Contributor

oscarbenjamin commented May 22, 2012

No, it's not time for opster to drop support for python 2.x now. But some time in the future it will be, and opster will have the opportunity to require a single cleaner syntax. And, at that time anyone still wanting to support opster on 2.x and 3.x can just use an older version of opster, since the current versions won't support python 2.x anyway.

If there is a new syntax then opster can support both, perhaps by checking for keyword arguments and/or annotations. Anyone using those features must be using only python 3.x and should really be using opster's new, future syntax (whatever that is).

What I'm saying is that it would be good to make a decision on future syntax now because noone is using python 3.x yet.
So now, while noone's using it, is a good time to decide on a future syntax, implement/test it and display it prominently in the docs, so that anyone who starts something new in python 3.x will use it, and some of the people supporting both python 2.x and 3.x will be aware of the fact that the syntax will change at some point (they can always just embed an older version of opster if they need it to be eternally stable).

Also, I though of another possibility using annotations for positional arguments

@command()
def main(numtimes: (int, 'number of times'),
         inputfile: (file, 'file to read data from') = sys.stdin,
         *,
         format: ('f', 'output format') = 'binary',
         quiet:('q', 'Supress all output') = False):
     pass

Then opster could show help for the positional arguments and validate their types etc. This is posible with keyword-only arguments since opster has an independent way of distinguishing positional arguments and option arguments.

Also, since the main issue seems to be presentation, how about:

@opster.command()
def main(required_arg, optional_arg=None, *varargs,
         option1:('o', False, 'help for --option1') = False,
         option2:('O', 'default_cl', 'enter a value for option2') = 'default_py',
         option3:('q', 4, 'this will be a required kwarg when calling main()'),
         **globalopts)
    pass

This way it looks pretty much like before and you have the option to provide a different default value when main is called just as main(). With this form, if a default value for an option is not also provided at the end then keyword argument to main becomes a required keyword argument when main is called on its own. Alternatively, if desired, the functions kwdefaults can be modified in place so that they have the same default values as the options.

Contributor

oscarbenjamin commented May 22, 2012

Another syntax that has worked in all versions of opster although it's not really documented any more is

options = [('o', 'option1', False, 'help for option1')]

@opster.command(options=options)
def main(**opts):
    if opts['option1']:
        pass

The opster docs could describe that as a future-proof python 2.x and 3.x syntax.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment