New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use XSD to lint tools and repositories. #130

Merged
merged 2 commits into from Apr 20, 2015

Conversation

Projects
None yet
3 participants
@jmchilton
Copy link
Member

jmchilton commented Apr 17, 2015

  • Implement validation abstraction that will use either lxml (Python lib) or xmllint (command line app) dependending on what is available - with test cases.
  • Add a --xsd flag to the lint command (for tools) that lints against the experimental XSD from https://github.com/JeanFred/Galaxy-XSD.
  • Implement repository_dependencies.xsd to describe Tool Shed repository_dependencies.xml files (fairly complete).
  • Implement tool_dependencies.xsd to describe Tool Shed tool_dependencies.xml files.
    • Validates attributes and elements down to the action elements and then largely gives up (sticking any and anyAttribute tags on that element).
    • Registers everything in tools-devteam as valid, and detects one invalid XML file in tools-iuc.
  • Implement new shed_lint command that:
    • Validates discovered tool_dependencies.xml against schema
    • Validates discovered repository_dependencies.xml against schema
    • Bare minimum to lint .shed.yml files.
    • Optionally also lints tools in repositories with the --tools argument. Like shed_upload uses presence of .shed.yml to detect repositories, this should be generalized.
    • Can recursively lint many repositories at one time with -r.
  • Refactoring of existing stuff to support this and make room for XSD validation of tool XML files and generalizing applying actions over many repositories and many tools.
Use XSD to lint tools and repositories.
 - Implement validation abstraction that will use either lxml (Python lib) or xmllint (command line app) dependending on what is available - with test cases.
 - Add a ``--xsd`` flag to the lint command that lints against the experimental XSD from https://github.com/JeanFred/Galaxy-XSD.
 - Implement ``repository_dependencies.xsd`` to describe Tool Shed ``repository_dependencies.xml`` files (fairly complete).
 - Implement ``tool_dependencies.xsd`` to describe Tool Shed ``tool_dependencies.xml`` files.
   - Validates attributes and elements down to the ``action`` elements and then largely gives up (sticking ``any`` and ``anyAttribute`` tags on that element).
   - Registers everything in tools-devteam as valid, and detects one invalid XML file in tools-iuc.
 - Implement new ``shed_lint`` command that:
   - Validates tool_dependencies.xml against schema
   - Validates repository_dependencies.xml against schema
   - Bare minimum to lint .shed.yml files.
   - Optionally also lints tools in repsitories with the ``--tools`` argument.
   - Can recursively lint many repositories at ont time ``-r``.
 - Refactoring of existing stuff to support this and make room for XSD validation of tool XML files and generalizing applying actions over many repositories and many tools.

@jmchilton jmchilton referenced this pull request Apr 17, 2015

Closed

XSD Validation of Tools #127

@@ -192,6 +194,24 @@ def build_tarball(tool_path):
return temp_path


def walk_repositories(path):

This comment has been minimized.

@erasche

erasche Apr 17, 2015

Member

oh, nice, will reuse this in recursive shed uploads.

This comment has been minimized.

@jmchilton

jmchilton Apr 17, 2015

Author Member

That is where I got it, it has already be refactored out of shed_upload.

This comment has been minimized.

@erasche

erasche Apr 17, 2015

Member

Oh, excellent, thanks!

fre. 17. apr. 2015 kl. 09.51 skrev John Chilton notifications@github.com:

In planemo/shed.py
#130 (comment):

@@ -192,6 +194,24 @@ def build_tarball(tool_path):
return temp_path

+def walk_repositories(path):

That is where I got it, it has already be refactored out of shed_upload.


Reply to this email directly or view it on GitHub
https://github.com/galaxyproject/planemo/pull/130/files#r28600556.

@martenson

This comment has been minimized.

Copy link
Member

martenson commented Apr 17, 2015

This is looking great. Very useful.

lint_ctx.warn("Failed to parse .shed.yml file [%s]" % str(e))

warned = False
for required_key in ["owner", "name"]:

This comment has been minimized.

@erasche

erasche Apr 17, 2015

Member

any reason for not using the expanded key set? You might add description, as that's the only other thing that's strictly required for a create-repo to succeed.

This comment has been minimized.

@jmchilton

jmchilton Apr 17, 2015

Author Member

It is required for creation, but not for vanilla upload - even these two fields are not strictly required... hmm.... I am not against requiring everything in .shed.yml but this was a conservative first attempt.

@jmchilton jmchilton referenced this pull request Apr 17, 2015

Open

Implement shed_lint. #107

10 of 11 tasks complete
@@ -0,0 +1,15 @@
<?xml version="1.0" encoding="UTF-8"?>

This comment has been minimized.

@erasche

erasche Apr 17, 2015

Member

not using submodules?

This comment has been minimized.

@erasche

erasche Apr 17, 2015

Member

I know how much fun copy and pasting scripted import of files from other projects is ;)

This comment has been minimized.

@jmchilton

jmchilton Apr 17, 2015

Author Member

Hmm... maybe... pypa/pip#577 ... yeah I should do that.

This comment has been minimized.

@jmchilton

jmchilton Apr 20, 2015

Author Member

Done - thanks for the suggestion.

@erasche

This comment has been minimized.

Copy link
Member

erasche commented Apr 17, 2015

+1 from me, can't wait to add this to the jenkins testing bot! :D

jmchilton added a commit that referenced this pull request Apr 20, 2015

Merge pull request #130 from galaxyproject/xsd
Use XSD to lint tools and repositories.

@jmchilton jmchilton merged commit ee014ea into master Apr 20, 2015

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@bgruening bgruening deleted the xsd branch May 1, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment