Implement automatic tool generation based on the source code of the tool

[Kulivox]

Hello,
A few months ago I had to implement a bunch of tool wrappers for an open source toolkit and realised how much time could be saved if there was an option of automatic generation of parts of the wrapper in tool_init. That lead me to develop PyGalGen, my take on the functionality of tool_init. I have shown the tool to @martenson, and he has agreed that migration of the part of the functionality that generates the wrapper file is worthwhile.

This pull request contains the prototype of the functionality. The generator is capable of extracting inputs and generating a command section for tools written in Python3 that use the ArgumentParser module.
There is still some work to be done, mainly in testing and documentation.

Current functionality:
Parser information can be extracted if the following conditions are met:

• tool uses ArgumentParser
• the parser init is in a single function

TODO:

• add documentation

Example input: url
Example output: https://pastebin.com/v9z4ZMAD

Suggestions are welcome!

[bernt-matthias]

Hi @Kulivox are you aware of https://github.com/hexylena/argparse2tool

[Kulivox]

Hi @Kulivox are you aware of https://github.com/hexylena/argparse2tool

Hi, no, I didn't know such project already exists
Good to know I guess :D

[hexylena]

@bernt-matthias I met with @Kulivox @biomonika and discussed a bit how to proceed, what to do with argparse2tool. They discussed their methods, it seems a bit safer/cleaner than my fake argparse method and I'm happy to support this being in planemo, instead of a2t, and instead of attempting to merge the two as I think the two of us are the primary users. So we'll let them co-exist rather than spending the time to try and merge.

It would be nice to have a2t's functionality with both cwl + galaxy export, but I think that's not a priority, we can always direct people to use a2t instead if they need that functionality, just means I'll probably do less maintenance of it. @Kulivox's implement is more complete it seems, a2t is missing argparse command groups currently.

And TIL planemo has a text tool template inside, maybe that's a use case for our galaxyxml (but maybe that's also not worth the return on investment?)

@Kulivox would you be open to handling ('output', type=argparse.FileType('w')) as a tool output? I think that's the only major difference I see, and I'm pretty sure I'm the only person alive who uses it, but, I'd still like it. We also have some existing tool test cases, if you wanted to use those as a basis for the tests here.

[Kulivox]

@hexylena Sure, I'll add the type mapping in, and thanks; I think I'll also use the tool examples you provided to create tests

[Kulivox]

I'll be doing a final pass over the functionality of autpygen during the next few days, but I don't think I'll be changing much, maybe adding support for some very specific argument types and some cleanup of the output. The PR is IMHO ready for review, I have updated automatically generated docs, but I won't be adding custom docs until the functionality is finalised. Would it be possible for you @bernt-matthias to take a look at this PR? Or should I contact someone else?

[bernt-matthias]

@Kulivox will you be at the EGD in Freiburg next week?

[Kulivox]

@bernt-matthias No, unfortunately

[bernt-matthias]

Here some first comments. So far I have only checked the test case.

Wondering if https://github.com/hexylena/galaxyxml/ would be of any use. But I guess you already discussed this with @hexylena .

[bernt-matthias]

Hey @Kulivox would you be available for a VC on Thursday or Friday? I guess the easiest for a review would be if you could walk us through your code.

Maybe @hexylena and @nsoranzo would like to join?

[hexylena]

sounds good.

…

On 04-10-2022 11:21, M Bernt wrote: Hey @Kulivox <https://github.com/Kulivox> would you be available for a VC on Thursday or Friday? I guess the easiest for a review would be if you could walk us through your code. Maybe @hexylena <https://github.com/hexylena> and @nsoranzo <https://github.com/nsoranzo> would like to join? — Reply to this email directly, view it on GitHub <#1263 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADP7O5EPPB7HU5RY55TZ7LWBPZKRANCNFSM57IE7DKQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Kulivox]

Hi, I should be available on Friday after 4 pm CET. Does that work for you too? @bernt-matthias @hexylena

[bernt-matthias]

Hi @Kulivox I will be on the train from approx 2pm. So unfortunately, 4pm won't work for me.

[Kulivox]

Ah, that is unfortunate. I'll be at work from 7-15 on Friday, so I can't do the call earlier. What about some later date? I should be available anytime during the weekend and after 4pm on weekdays @bernt-matthias

[bernt-matthias]

Hi @Kulivox Due to vacation I will only have time from Nov 7 :(. But maybe @hexylena and @nsoranzo could help earlier...?

[Kulivox]

Hi, @hexylena @nsoranzo, would it be possible to arrange a VC where we take a look at the PR as @bernt-matthias suggested? I should be available any day apart from Tuesday.

[biomonika]

Hi @bernt-matthias , I hope you have had a great time away. If you're up for it, could we please continue the review? Thanks a lot for your help!

[bernt-matthias]

Hi @Kulivox and @hexylena could you suggest a date / dates during next week? Currently I'm available except for Wednesday Morning (Berlin time).

[Kulivox]

@bernt-matthias I am free on Monday afternoon (16:30 + CET), Tuesday evening (18:00+) and then on any day from 7-9 AM CET and from 16:15-23:59 CET

[bernt-matthias]

Monday afternoon (16:30 + CET) would be cool for me.

[hexylena]

I can make that as well

…

On Tue, 8 Nov 2022, 17:18 M Bernt, ***@***.***> wrote: Monday afternoon (16:30 + CET) would be cool for me. — Reply to this email directly, view it on GitHub <#1263 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADP7O4UDZMRBXRZJOVE5KDWHJ4IXANCNFSM57IE7DKQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Kulivox]

@hexylena @bernt-matthias Alright then, I'll set up a meeting starting at 17:00 because I might not be ready at 16:30 (train journey home)

[bernt-matthias]

Wonderful.

[bernt-matthias]

Currently testing another tool. If I find some things that might be improved I will add them here:

If there is a command line boolean flag: --flag it would be great to define it with truevalue="--flag" falsevalue="--flag" checked="true/false" (checked depending on the default) .. the value attribute should be omitted.

Use it in the command just as $flag.

Or wouldn't this work for:

    parser.add_argument('--flag', metavar='BOOL', type=bool, default=true)

?

[bernt-matthias]

I guess we should merge this soon. It's really good and usable in my opinion. We can still improve it and fix bugs later on... or what do you think @mvdbeek @nsoranzo and @hexylena ?

[bernt-matthias]

This is good to go in my opinion. Not sure about docs: Obviously a good idea, but I don't know where it would fit best and what the extent/form should be.

[mvdbeek]

Yes, at least a small example would be nice, maybe in writing_standalone.rst ? This is nicely isolated and looking good.

[mvdbeek]

Thanks for your work @Kulivox! I've pushed a new planemo version to https://pypi.org/project/planemo/0.75.11/