New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add commands to create training materials #861

Merged
merged 27 commits into from Sep 11, 2018

Conversation

Projects
None yet
4 participants
@bebatut
Copy link
Member

bebatut commented Aug 8, 2018

Hi,

This PR adds new commands to planemo (as discussed during GCCBOSC: galaxyproject/training-material#941):

  1. training_init: a command to create new training material for the GTN material given the template there:
    1. It creates a topic if not present (with metadata filled)
    2. It creates a tutorial if not present (with all needed files)
    3. It generates the skeleton of a tutorial if a workflow is given and add also the workflow
    4. It creates the data_library.yaml file by parsing the Zenodo API and add the link in the tutorial
    5. It updates the metadata if needed
  2. training_generate_tuto_from_wf to generate the tutorial.md of an existing tutorial from a workflow
  3. training_fill_data_library to fill the data_library.yaml file by parsing the Zenodo API of an existing tutorial

I tested it by generating several tutorial skeletons. @abretaud tested it too.

I added tests to cover most of the functions (I have some doubts about the tests of the commands).
I am not sure about the code quality, particularly the training.py script if not so straightforward.
I hope I could get your help there 馃槃

ping @shiltemann, @jmchilton

B茅r茅nice

@abretaud

This comment has been minimized.

Copy link
Contributor

abretaud commented Aug 9, 2018

Oops, I get this exception now:

Traceback (most recent call last):
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/bin/planemo", line 11, in <module>
    load_entry_point('planemo==0.55.0.dev0', 'console_scripts', 'planemo')()
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/cli.py", line 195, in handle_blended_options
    return f(*args, **kwds)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/commands/cmd_training_init.py", line 19, in cli
    training.init(ctx, kwds)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 855, in init
    create_tutorial(kwds, tuto_dir, topic_dir, tuto_template_dir, ctx)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 824, in create_tutorial
    create_tutorial_from_workflow(kwds, z_file_links, tuto_dir, ctx)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 770, in create_tutorial_from_workflow
    body += get_handson_box(step, wf['steps'], tools)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 747, in get_handson_box
    paramlist = get_param_desc(step_params, step_inputs, tp_desc, 1, steps, should_be_there=True)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 731, in get_param_desc
    paramlist += format_param_desc(step_param, step_inputs, tp_d, level, wf_steps)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 704, in format_param_desc
    paramlist += format_section_param_desc(step_params, step_inputs, tp_desc, level, wf_steps)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 613, in format_section_param_desc
    sub_param_desc = get_param_desc(params, inputs, get_tool_input(tp_desc), level+1, wf_steps)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 731, in get_param_desc
    paramlist += format_param_desc(step_param, step_inputs, tp_d, level, wf_steps)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 706, in format_param_desc
    paramlist += format_conditional_param_desc(step_params, step_inputs, tp_desc, level, wf_steps)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 628, in format_conditional_param_desc
    params[test_param['name']],
TypeError: string indices must be integers
@bebatut

This comment has been minimized.

Copy link
Member Author

bebatut commented Aug 9, 2018

馃槦
Can you give me your workflow?

@bebatut

This comment has been minimized.

Copy link
Member Author

bebatut commented Aug 16, 2018

@abretaud it should be fixed. can you test? thanks

@abretaud

This comment has been minimized.

Copy link
Contributor

abretaud commented Aug 20, 2018

@bebatut I confirm! thanks

@abretaud

This comment has been minimized.

Copy link
Contributor

abretaud commented Aug 20, 2018

I've just re-checked the tutorial.md generated with the latest code on the maker workflow:

  • some select boxes are still absent from the tutorial: e.g. "Organism type" for Maker tool
  • the same for a multiple select box: "Select fasta outputs" for gffread
    The rest looks good :)
@jmchilton

This comment has been minimized.

Copy link
Member

jmchilton commented Aug 21, 2018

Some of the tests just don't pass for me:

    n = 'save_db'
    tp_desc = tools[step['name']][n]
    step_param = training.get_lower_params(step_params, n)
    print(tp_desc)
    print(n)
    print(step_params)
    print(step_param)
    paramlist = training.format_param_desc(
        step_param,
        step_inputs,
        tp_desc,
        0,
        wf['steps'],
        force_default=False)
    assert '`Yes`' in paramlist, paramlist

That parameter appears to be false not true for instance. Does that work for you locally - does it require Galaxy modifications?

> ### {{ '{%' }} icon hands_on {{ '%}' }} Hands-on: Data upload
>
> 1. Import the following files from [Zenodo]({{ zenodo_link }}) or from a data
> library named `TODO` if available (ask your instructor)

This comment has been minimized.

@abretaud

abretaud Aug 23, 2018

Contributor

This should probably be the same as L147?

@@ -0,0 +1,7 @@
---

This comment has been minimized.

@abretaud

abretaud Aug 23, 2018

Contributor

Is this file still needed? It looks its format changed

> > ### {{ '{%' }} icon tip {{ '%}' }} Tip: Importing data from a data library
> >
> > * Go into "Shared data" (top panel) then "Data libraries"
> > * Click on "Training data" and then "{{ topic_title }}"

This comment has been minimized.

@abretaud

abretaud Aug 23, 2018

Contributor

Is it only topic_title or topic_title/tutorial_title?

wf, tools = get_wf_tools_from_running_galaxy(kwds)
else:
wf, tools = get_wf_tool_from_local_galaxy(kwds, kwds["workflow"], ctx)
save_to_yaml(tools, 'tools.yaml')

This comment has been minimized.

@abretaud

abretaud Aug 23, 2018

Contributor

Hum, don't know if it's intended but now I get a tools.yaml at the root of the gtn dir (but it doesn't look the same as the tutorial tools.yaml)

@bebatut bebatut force-pushed the training_init branch from 55f6d47 to 1d902b9 Aug 29, 2018

@bebatut

This comment has been minimized.

Copy link
Member Author

bebatut commented Aug 29, 2018

I updated to fit with the new metadata definition

@jmchilton The test should be fixed now
@abretaud I fixed some of your comments. I need to test again your workflow to investigate the issue with the boxes... Next week

@bebatut

This comment has been minimized.

Copy link
Member Author

bebatut commented Sep 7, 2018

@abretaud I checked your comments. If the value of a parameter is set to the default value in the tool description, the corresponding parameter description does not appear (to avoid all parameters to be described, specially for big tool). It explain why "Organism type" for Maker tool does not appear
Does it make sense?

@abretaud

This comment has been minimized.

Copy link
Contributor

abretaud commented Sep 7, 2018

@bebatut ah yes, it makes sense, I thought it was not the default value

@bebatut

This comment has been minimized.

Copy link
Member Author

bebatut commented Sep 7, 2018

I restructured the code and split it into classes to try to make it easier to maintain.
I extracted the doc for it.
I think it is ready for review. ping @jmchilton, @shiltemann, @erasche

@mblue9

This comment has been minimized.

Copy link

mblue9 commented Sep 10, 2018

Hi @bebatut !

I'm trying training_generate_from_wf out at the moment and getting an error TypeError: string indices must be integers. This is the workflow I'm using (I just changed the file ending to .txt from .ga to add it here)
Galaxy-Workflow-limma-voom_tutorial_fastqs_to_counts.txt

Btw I'm using python 2.7 with this but I see python 3.6 in @abretaud's error above, should I be using python 3 or it doesn't matter?

@bebatut

This comment has been minimized.

Copy link
Member Author

bebatut commented Sep 10, 2018

@mblue9 I could not reproduce your error (with Python 2.7 or 3.6) 馃槦
Did you use the latest version from this PR? What command did you use?

@jmchilton

This comment has been minimized.

Copy link
Member

jmchilton commented Sep 10, 2018

That error happens in the Travis tests as well https://travis-ci.org/galaxyproject/planemo/jobs/425797918.

======================================================================
ERROR: Test :func:`planemo.training.tutorial.tutorial.create_tutorial`.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/galaxyproject/planemo/.tox/py27/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/build/galaxyproject/planemo/tests/test_training_tutorial.py", line 432, in test_tutorial_create_tutorial
    tuto.create_tutorial(CTX)
  File "/home/travis/build/galaxyproject/planemo/planemo/training/tutorial.py", line 463, in create_tutorial
    self.create_hands_on_tutorial(ctx)
  File "/home/travis/build/galaxyproject/planemo/planemo/training/tutorial.py", line 444, in create_hands_on_tutorial
    self.body = get_hands_on_boxes_from_local_galaxy(self.training.kwds, self.init_wf_fp, ctx)
  File "/home/travis/build/galaxyproject/planemo/planemo/training/tutorial.py", line 592, in get_hands_on_boxes_from_local_galaxy
    tuto_body = format_wf_steps(wf, config.gi)
  File "/home/travis/build/galaxyproject/planemo/planemo/training/tutorial.py", line 574, in format_wf_steps
    tool_inp = ToolInput(inp, wf_param_values, steps, 1, should_be_there=True)
  File "/home/travis/build/galaxyproject/planemo/planemo/training/tool_input.py", line 48, in __init__
    self.wf_param_values = self.wf_param_values[self.name]
TypeError: string indices must be integers
-------------------- >> begin captured stdout << ----------
@mblue9

This comment has been minimized.

Copy link

mblue9 commented Sep 10, 2018

@bebatut and @jmchilton thanks for taking a look! Yes I used the version from this PR and the command I used was: planemo training_generate_from_wf --topic_name transcriptomics --tutorial_name "limma-voom_fastqs_to_counts" --workflow ~/training/pmac/galaxy/limma-voom/limma-voom_fastqs_to_counts/Galaxy-Workflow-limma-voom_tutorial_fastqs_to_counts.ga --galaxy_root ~/galaxy/galaxy/

(The training_init command seemed to work)

@bebatut does it work for you with my workflow? If so do you think you could send me the tutorial.md and I could keep going with that? as I'm running this tutorial in a couple of weeks. Otherwise I'll just manually create it.

@bebatut

This comment has been minimized.

Copy link
Member Author

bebatut commented Sep 11, 2018

@mblue9 It works for me with your workflow: the tutorial.md (with changed extension)
Does it work if you do not specify the --galaxy_root? And with the training-init with the workflow?

@bebatut

This comment has been minimized.

Copy link
Member Author

bebatut commented Sep 11, 2018

@jmchilton All the tests implying to start a Galaxy instance (though the call of get_hands_on_boxes_from_local_galaxy or in which I started a Galaxy instance) and then getting the workflow and the tools from this Galaxy instancne seems to fail on Travis (but locally for me):

Test :func:`planemo.training.generate_tuto_from_wf`. ... ERROR
Test :func:`planemo.training.tutorial.format_wf_steps`. ... ERROR
Test :func:`planemo.training.tutorial.get_hands_on_boxes_from_local_galaxy`. ... ERROR
Test :func:`planemo.training.tutorial.get_hands_on_boxes_from_running_galaxy`. ... ERROR
Test :func:`planemo.training.tutorial.get_hands_on_boxes_from_local_galaxy`. ... ERROR
Test :func:`planemo.training.tutorial.get_hands_on_boxes_from_running_galaxy`. ... ERROR

Are you aware of any similar issue? I will investigate how the workflow and the extracted tools from these Galaxy instance look like (empty or not).

@mblue9

This comment has been minimized.

Copy link

mblue9 commented Sep 11, 2018

@bebatut thanks for the file!!! That auto-generation is so cool 馃槑 馃檶and going to save me a chunk of time (and tedium) 馃槃!!!

I tried training_generate_from_wf without --galaxy_root and I get the same error. I had run training-init without --galaxy_root.

@bebatut bebatut force-pushed the training_init branch 2 times, most recently from c385d2d to ff390ff Sep 11, 2018

@bebatut bebatut force-pushed the training_init branch from ff390ff to 45cd23a Sep 11, 2018

@jmchilton

This comment has been minimized.

Copy link
Member

jmchilton commented Sep 11, 2018

Fixed some linting issues and I added test decorators so the tests requiring Galaxy don't slow the "quick" version of the test suite. All the tests will still run with the full suite for both Python 2 and Python 3.

I couldn't love this anymore - it is absolutely amazing.

@jmchilton jmchilton merged commit fef52a2 into master Sep 11, 2018

0 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
continuous-integration/travis-ci/push The Travis CI build is in progress
Details

jmchilton added a commit that referenced this pull request Sep 11, 2018

Make @bebatut an official committer for Planemo.
The wide scope and stellar quality of the new training material commands, library functionality, and tests clearly warrant this. See #861. This is both to say thanks for the great contribution and done in hopes of encouraging continued use of Planemo for this awesome functionality.

@martenson martenson deleted the training_init branch Nov 7, 2018

@martenson martenson restored the training_init branch Nov 7, 2018

@martenson martenson deleted the training_init branch Nov 7, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment