New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve tool validation and metadata generation #5119

Merged
merged 23 commits into from Dec 7, 2017

Conversation

Projects
None yet
3 participants
@mvdbeek
Member

mvdbeek commented Dec 2, 2017

(Builds on #5099)
This introduces a ValidationConext object, that should contain just enough information to load tools and data tables and can be used in place of app. This avoids manipulation of app-wide settings during tool installations, metadata setting and browsing of toolshed repos.

mvdbeek added some commits Nov 26, 2017

Fix copying of loc.sample files
`filename` should be joined with the tool_path, as evidenced by other uses of
copy_sample_file.
Do not write shed_tool_data_table_config in place
This should prevent errors such as:
```
galaxy.util ERROR 2017-11-17 09:20:12,842 Error parsing file /data/users/mvandenb/gx/config/shed_tool_data_table_conf.xml
Traceback (most recent call last):
  File "lib/galaxy/util/__init__.py", line 217, in parse_xml
    root = tree.parse(fname, parser=ElementTree.XMLParser(target=DoctypeSafeCallbackTarget()))
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 657, in parse
    self._root = parser.close()
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1654, in close
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
    raise err
ParseError: no element found: line 1, column 0
```
reported in #5031.
Fix installation of data manager revision updates
Without this updates of data managers fails with
```
AttributeError: 'NoneType' object has no attribute 'get'
  File "galaxy/web/framework/middleware/sentry.py", line 40, in __call__
    iterable = self.application(environ, start_response)
  File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/paste/recursive.py", line 85, in __call__
    return self.application(environ, start_response)
  File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/paste/httpexceptions.py", line 640, in __call__
    return self.application(environ, start_response)
  File "galaxy/web/framework/base.py", line 136, in __call__
    return self.handle_request(environ, start_response)
  File "galaxy/web/framework/base.py", line 215, in handle_request
    body = method(trans, **kwargs)
  File "galaxy/web/framework/decorators.py", line 98, in decorator
    return func(self, trans, *args, **kwargs)
  File "galaxy/webapps/galaxy/controllers/admin_toolshed.py", line 1849, in update_to_changeset_revision
    persist=True)
  File "tool_shed/galaxy_install/metadata/installed_repository_metadata_manager.py", line 30, in __init__
    metadata_dict=metadata_dict, user=None)
  File "tool_shed/metadata/metadata_generator.py", line 63, in __init__
    self.metadata_dict = {'shed_config_filename': self.shed_config_dict.get('config_filename', None)}
```
Use a temporary ToolDataTableManager when generating metadata
Otherwise loaded data tables bleed into the app-wide data tables. This can be
seen with lots of data tables pointing to (non-existing, at that point)
temporary files after a metadata reset in the galaxy admin panel,
but it can also cause failure to install tools with entirely new data tables.
from .bunch import Bunch
class MiniApp(object):

This comment has been minimized.

@jmchilton

jmchilton Dec 2, 2017

Member

Entirely personal preference - but what about the path and name galaxy.tools.repositories.ValidationContext instead of galaxy.util.miniapp.MiniApp?

Edit: For more justification - it seems very tool specific - so I don't think it belongs in galaxy.util. I imagine there are other places we will want to decompose app - indeed there are other places we have already decomposed trans so we should make it clear what the purpose of a particular decomposition is I guess -for instance with the WorkContext alternative to trans.

This comment has been minimized.

@mvdbeek

mvdbeek Dec 2, 2017

Member

Sure, I can do that!

@mvdbeek mvdbeek changed the title from WIP: Improve tool validation and metadata generation to Improve tool validation and metadata generation Dec 3, 2017

@mvdbeek mvdbeek added status/review and removed status/WIP labels Dec 3, 2017

@galaxybot galaxybot added this to the 18.01 milestone Dec 3, 2017

@jmchilton

This comment has been minimized.

Member

jmchilton commented Dec 7, 2017

Amazing, amazing, amazing... amazing! Keep up the amazing work!

@jmchilton jmchilton merged commit 30e2736 into galaxyproject:dev Dec 7, 2017

7 checks passed

api test Build finished. 332 tests run, 4 skipped, 0 failed.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
framework test Build finished. 165 tests run, 0 skipped, 0 failed.
Details
integration test Build finished. 59 tests run, 0 skipped, 0 failed.
Details
lgtm analysis: JavaScript No alert changes
Details
selenium test Build finished. 117 tests run, 2 skipped, 0 failed.
Details
toolshed test Build finished. 577 tests run, 0 skipped, 0 failed.
Details

@mvdbeek mvdbeek deleted the mvdbeek:dev_toolshed_install_improvements_2 branch Jun 12, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment