Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speedup startup and toolbox operations #3909

Merged
merged 53 commits into from
May 2, 2017
Merged

Speedup startup and toolbox operations #3909

merged 53 commits into from
May 2, 2017

Commits on May 1, 2017

  1. Use floating point to log toolox reload times

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    41cbe24 View commit details
    Browse the repository at this point in the history

  2. Add timing information for Data Manager reload

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    4d1cae4 View commit details
    Browse the repository at this point in the history

  3. Add a short-lived (1s) ToolShedRepositoryCache 

    This will speed up repeated ToolShedRepositories lookups, like when
building the Toolbox for the first time, or when loading a
shed_tool_conf.xml file with many deactivated tools.
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    c06b4c9 View commit details
    Browse the repository at this point in the history

  4. Speed up tool-data table reloads by doing a os.walk and caching the r… 

    …esult for 1s instead of repeatedly calling os.path.exists.
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    43c9c28 View commit details
    Browse the repository at this point in the history

  5. Only index changed tools when re-indexing the toolbox 

    This speeds up repeated toolbox reloads enourmously (from 1 second on my
test-instance to 0.1 seconds).
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    34ed9b4 View commit details
    Browse the repository at this point in the history

  6. Always use cache to fetch tool_version objects

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    c6429bf View commit details
    Browse the repository at this point in the history

  7. Use cElementTree for light xml parsing speedup

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    96cb874 View commit details
    Browse the repository at this point in the history

  8. Update ToolVersionCache when adding a new ToolShedLineage

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    2906f0c View commit details
    Browse the repository at this point in the history

  9. Move tool_cache.reset_status() calls to reload toolbox/data managers,… 

    … so that we can use information about whether tools are new during reloads
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    6c22726 View commit details
    Browse the repository at this point in the history

  10. Make sure re-indexing the toolbox occurs on the new toolbox

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    431a482 View commit details
    Browse the repository at this point in the history

  11. Remove replace_existing option for data managers 

    If a data manager is defined twice for the same data manager id in the
shed_tool_conf.xml the first data manager will be loaded. This is not a
problem when updating a data manager through the toolshed, because the
outdated data manager will be removed from the tool cache and a new
toolbox is being built.
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    ea9ecdf View commit details
    Browse the repository at this point in the history

  12. Reset the tool cache once all tools are loaded after starting up galaxy

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    db910f4 View commit details
    Browse the repository at this point in the history

  13. Reduce logging during toolbox reloads 

    We don't log loading of tools that have not changed when reloading the
toolbox.
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    9aa5556 View commit details
    Browse the repository at this point in the history

  14. Fix unittests that access app.tool_cache

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    475b886 View commit details
    Browse the repository at this point in the history

  15. Re-use ExecutionTimer abstractions for consistent log patterns.

    @jmchilton @mvdbeek
    jmchilton authored and mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    beae93b View commit details
    Browse the repository at this point in the history

  16. Log startup time

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    a83ef0b View commit details
    Browse the repository at this point in the history

  17. Fix cElementTree imports

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    3ff2a78 View commit details
    Browse the repository at this point in the history

  18. Fix import order in lib/galaxy/tools/data/__init__.py

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    c905c42 View commit details
    Browse the repository at this point in the history

  19. Add possibility explicitly rebuild ToolShedRepositoryCache

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    b72994f View commit details
    Browse the repository at this point in the history

  20. Fix path checking for files that are not in the tool_data_path directory

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    84c1559 View commit details
    Browse the repository at this point in the history

  21. Use generator instead of list comprehenesion followed converting it t… 

    …o a set. (Thanks @nsoranzo)
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    0504dcd View commit details
    Browse the repository at this point in the history

  22. Don't return toolbox (thx @nsoranzo)

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    cbed78f View commit details
    Browse the repository at this point in the history

  23. Use ExecutionTimer abstraction to report startup time

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    4dd49c9 View commit details
    Browse the repository at this point in the history

  24. Remove unused time import

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    dc16175 View commit details
    Browse the repository at this point in the history

  25. Be more defesive when accessing app.tool_cache

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    4c44338 View commit details
    Browse the repository at this point in the history

  26. Rebuild repo cache when deciding if repository should be created.

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    5069815 View commit details
    Browse the repository at this point in the history

  27. Avoid having tools in tool_cache marked as new when they have rapidly… 

    … shifted from new to expired.
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    d15813c View commit details
    Browse the repository at this point in the history

  28. Don't crash if a tool has not been added to the cache properly

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    586f494 View commit details
    Browse the repository at this point in the history

  29. Refresh tool_data_path_files if we know that a new table has been wri… 

    …tten
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    3d6bc1d View commit details
    Browse the repository at this point in the history

  30. Reference old_toolbox earlier, since reloading can happen in a differ… 

    …ent thread, which will lead to a hang when waiting for another toolbox reload
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    07da123 View commit details
    Browse the repository at this point in the history

  31. Always refresh tool_shed_repository_cache before fetching repositorie… 

    …s by ids (since that is only called from admin_toolshed.py and installed_repository_metadata_manager.py)
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    677d986 View commit details
    Browse the repository at this point in the history

  32. Fix import order in lib/galaxy/config.py

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    f940c79 View commit details
    Browse the repository at this point in the history

  33. Allow looking up repositories by repository_id via ToolShedRepository… 

    …Cache
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    89f57f2 View commit details
    Browse the repository at this point in the history

  34. Refresh TSR cache when needed

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    456d1b4 View commit details
    Browse the repository at this point in the history

  35. Restore deactivate/activate functionality

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    8411b02 View commit details
    Browse the repository at this point in the history

  36. Revert "Fix cElementTree imports" 

    This reverts commit 6d365f0.
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    15e2c9b View commit details
    Browse the repository at this point in the history

  37. Revert "Use cElementTree for light xml parsing speedup" 

    This reverts commit 556469f.
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    7e61f8c View commit details
    Browse the repository at this point in the history

  38. Use ExecutionTimer

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    141f152 View commit details
    Browse the repository at this point in the history

  39. Use tool_shed_repository_cache in populate_tool_shed_info

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    73c7c36 View commit details
    Browse the repository at this point in the history

  40. Replace getchildren(), which has been deprecated (https://docs.python… 

    ….org/2/library/xml.etree.elementtree.html#xml.etree.ElementTree.Element.getchildren)
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    869d7bf View commit details
    Browse the repository at this point in the history

  41. Write out the datatype registry only once

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    3079d0c View commit details
    Browse the repository at this point in the history

  42. Build toolbox search in a local control task

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    f88304b View commit details
    Browse the repository at this point in the history

  43. Don't fail tests without internet connectivity

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    5e7da2c View commit details
    Browse the repository at this point in the history

  44. Write out datatype registry when loading set_metadata tool

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    5a26896 View commit details
    Browse the repository at this point in the history

  45. Keep cache thread-local

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    f199d5b View commit details
    Browse the repository at this point in the history

  46. Use current session

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    b4e4f3c View commit details
    Browse the repository at this point in the history

  47. Fix thread local cache use

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    ff1b66a View commit details
    Browse the repository at this point in the history

  48. Send control task as postfork function

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    84351de View commit details
    Browse the repository at this point in the history

  49. Work around InstanceDetached errors

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    b666649 View commit details
    Browse the repository at this point in the history

  50. Skip time-based reloading, rely on detecting DetachedInstanceErrors t… 

    …o trigger cache rebuild
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    86684df View commit details
    Browse the repository at this point in the history

  51. Attempt to speedup proprietary datatype loading

    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    cdad5be View commit details
    Browse the repository at this point in the history

  52. Remove operations from custom_datatype_manager 

    Updating the upload file formats is handled when loading the datatype
into the registry, and reloading the upload tool seems unnecessary and
insufficient, since this would only affect the current web handler. In
any case removed and added datatypes are immediately propagated to the
upload tool.
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    2587ab5 View commit details
    Browse the repository at this point in the history

  53. Build search index at startup, rebuild repository cache during toolbo… 

    …x or datamanager reload
    @mvdbeek
    mvdbeek committed May 1, 2017
    Configuration menu
    Copy the full SHA
    af156a8 View commit details
    Browse the repository at this point in the history