New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

shed_upload silent if missing .shed.yml include files #158

Closed
peterjc opened this Issue Apr 30, 2015 · 1 comment

Comments

Projects
None yet
2 participants
@peterjc
Copy link
Contributor

peterjc commented Apr 30, 2015

My reading of the code suggests that the default include value is ** (all files next to the .shed.yml file). I have tried adding an explicit listing of files, e.g.

$ more ~/repositories/pico_galaxy/tools/venn_list/.shed.yml
name: venn_list
owner: peterjc
homepage_url: https://github.com/peterjc/pico_galaxy/tools/venn_list
remote_repository_url: https://github.com/peterjc/pico_galaxy/tools/venn_list
description: Draw Venn Diagram (PDF) from lists, FASTA files, etc
long_description: |
  Draw Venn Diagrams for 1, 2 or 3 sets of identifiers as a PDF file.

  Can parse FASTA, FASTQ, SFF or tabular files (taking column one) for identifiers.
  This can be combined with the extensive tabular file filtering and manipulation
  tools within Galaxy to prepare the input data.

  Uses the R/Bioconductor package limma to draw the PDF file, called from Python with
  rpy.

  Uses Biopython to parse SFF files.
categories:
- Graphics
- Sequence Analysis
- Visualization
type: unrestricted
include:
- README.rst
- venn_list_with_typo.py
- venn_list.xml
- tool_dependencies.xml

This fails to complain about missing venn_list_with_typo.py:

$ planemo shed_upload --tar_only  ~/repositories/pico_galaxy/tools/venn_list 
cp /tmp/tmpx0zfjB shed_upload.tar.gz
$ tar -tzf /tmp/tmpx0zfjB
tool_dependencies.xml
README.rst
venn_list.xml

On the positive side, it did not include venn_list.py because it wasn't asked to (deliberate mismatch with venn_list_with_typo.py).

I want this example to fail, complaining that venn_list_with_typo.py was listed in the includes, but not found. Ideally the error would list all named but missing files, but just the first missing file would be acceptable.

As an aside, would checking the include files exist also fall under planemo lint functionality for validating the .shed.yml?

@jmchilton jmchilton added the bug label May 1, 2015

jmchilton added a commit to jmchilton/planemo that referenced this issue May 1, 2015

Fail if missing inclusions encountered.
 - Shed actions (e.g. upload) will fail for that repository.
 - Shed linting will warn about this.

Fixes galaxyproject#158 reported by @peterjc.

@jmchilton jmchilton closed this in 4cb024b May 1, 2015

@peterjc

This comment has been minimized.

Copy link
Contributor Author

peterjc commented May 1, 2015

Not sure it is meant to be this noisy, but the example does fail now :)

$ planemo shed_upload --tar_only  ~/repositories/pico_galaxy/tools/venn_list
Traceback (most recent call last):
  File "/mnt/galaxy/repositories/planemo/.venv/bin/planemo", line 9, in <module>
    load_entry_point('planemo==0.9.0.dev0', 'console_scripts', 'planemo')()
  File "/mnt/galaxy/repositories/planemo/.venv/lib/python2.6/site-packages/click-4.0-py2.6.egg/click/core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "/mnt/galaxy/repositories/planemo/.venv/lib/python2.6/site-packages/click-4.0-py2.6.egg/click/core.py", line 644, in main
    rv = self.invoke(ctx)
  File "/mnt/galaxy/repositories/planemo/.venv/lib/python2.6/site-packages/click-4.0-py2.6.egg/click/core.py", line 991, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/galaxy/repositories/planemo/.venv/lib/python2.6/site-packages/click-4.0-py2.6.egg/click/core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/mnt/galaxy/repositories/planemo/.venv/lib/python2.6/site-packages/click-4.0-py2.6.egg/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/mnt/galaxy/repositories/planemo/.venv/lib/python2.6/site-packages/click-4.0-py2.6.egg/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/mnt/galaxy/repositories/planemo/.venv/lib/python2.6/site-packages/click-4.0-py2.6.egg/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/mnt/galaxy/repositories/planemo/.venv/lib/python2.6/site-packages/planemo-0.9.0.dev0-py2.6.egg/planemo/commands/cmd_shed_upload.py", line 59, in cli
    exit_code = shed.for_each_repository(upload, path, **kwds)
  File "/mnt/galaxy/repositories/planemo/.venv/lib/python2.6/site-packages/planemo-0.9.0.dev0-py2.6.egg/planemo/shed/__init__.py", line 490, in for_each_repository
    for realized_repository in effective_repositories:
  File "/mnt/galaxy/repositories/planemo/.venv/lib/python2.6/site-packages/planemo-0.9.0.dev0-py2.6.egg/planemo/shed/__init__.py", line 536, in realize_effective_repositories
    kwds.get("fail_on_missing", True)
  File "/mnt/galaxy/repositories/planemo/.venv/lib/python2.6/site-packages/planemo-0.9.0.dev0-py2.6.egg/planemo/shed/__init__.py", line 699, in realizations
    yield self._realize_to(directory, name, multiple, fail_on_missing)
  File "/mnt/galaxy/repositories/planemo/.venv/lib/python2.6/site-packages/planemo-0.9.0.dev0-py2.6.egg/planemo/shed/__init__.py", line 712, in _realize_to
    raise Exception(msg)
Exception: Failed to include files for ['venn_list_with_typo.py']

jmchilton added a commit to jmchilton/planemo that referenced this issue May 1, 2015

Exception handling improvements.
For shed operations - catch exceptions, log them, but continue to completion and then return a 254 exit code at the end indicating there were problems with one or more repositories. This behavior has can be disabled with any of the shed operations using the new --fail_fast option.

Clean up noisy exception reported by @peterjc @ galaxyproject#158 (comment) (unless --fail_fast is used - then you get the full stack trace).

Tests.

Fixes galaxyproject#114.

jmchilton added a commit to jmchilton/planemo that referenced this issue May 1, 2015

Exception handling improvements.
For shed operations - catch exceptions, log them, but continue to completion and then return a 254 exit code at the end indicating there were problems with one or more repositories. This behavior has can be disabled with any of the shed operations using the new --fail_fast option.

Clean up noisy exception reported by @peterjc @ galaxyproject#158 (comment) (unless --fail_fast is used - then you get the full stack trace).

Tests.

Fixes galaxyproject#114.

jmchilton added a commit to jmchilton/planemo that referenced this issue May 1, 2015

Exception handling improvements.
For shed operations - catch exceptions, log them, but continue to completion and then return a 254 exit code at the end indicating there were problems with one or more repositories. This behavior has can be disabled with any of the shed operations using the new --fail_fast option.

Clean up noisy exception reported by @peterjc @ galaxyproject#158 (comment) (unless --fail_fast is used - then you get the full stack trace).

Tests.

Fixes galaxyproject#114.

jmchilton added a commit to jmchilton/planemo that referenced this issue Oct 20, 2015

Fail if missing inclusions encountered.
 - Shed actions (e.g. upload) will fail for that repository.
 - Shed linting will warn about this.

Fixes galaxyproject#158 reported by @peterjc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment