Merge pull request #1065 from simonbray/autoupdate-sb

Add autoupdate command

docs/autoupdate.rst

@@ -0,0 +1,40 @@
+=============================
+Autoupdating tools
+=============================
+
+Galaxy tools use underlying command-line software which are specified using ``<requirement>`` tags in the XML wrapper. If developers continue to release new versions of the software, the creator/maintainer needs to ensure that the tool requirements are bumped, otherwise the tool will become updated.
+
+Planemo provides a ``autoupdate`` subcommand which can be used to perform this task automatically. Basic usage is as follows:
+
+::
+
+    planemo autoupdate tool_wrapper.xml
+
+There are various flags which can be applied; these are some of the most important:
+  - ``--recursive``, which performs the autoupdate recursively on subdirectories
+  - ``--dry-run``, which checks if tool requirements are up-to-date without making the necessary change automatically
+  - ``--test``, which runs tests on all autoupdated tools. If this flag is used, all options available for ``planemo test`` are also available.
+  - ``--update_test_data`` (if ``--test`` is also selected) which will update test data for failed tests
+  - ``--skiplist``, pointing to a list of tool wrappers for which updates should be skipped
+  - ``--skip_requirements`` with a comma-separated list of packages not to update.  ``python``, ``perl``, ``r-base`` are skipped by default.
+
+One of the most efficient ways to use it is by implementing a CI cron job which runs the command on an entire GitHub repository of tool wrappers.
+
+Formatting tools
+=============================
+
+``autoupdate`` assumes that XML tools are formatted in a certain way - in accordance with the `IUC best practices`_. In particular:
+
+  - the tool ``version`` attribute must be set to ``@TOOL_VERSION@+galaxy0`` or ``@TOOL_VERSION@+galaxy@VERSION_SUFFIX@``
+  - A token ``@TOOL_VERSION@`` should be created which corresponds to the version number of the main requirement.
+  - Optionally, a token ``@VERSION_SUFFIX@`` should be created, which should be an integer representing the number of times the XML wrapper has been updated since ``@TOOL_VERSION@`` was updated.
+
+
+Implementing an autoupdate CI job
+=================================
+
+ A useful way to use the autoupdate command is to implement it as a CI job, so that tools in a repo can be updated on a regular basis (e.g. weekly). An example implementation is the `planemo-autoupdate`_ GitHub bot.
+
+
+.. _IUC best practices: https://galaxy-iuc-standards.readthedocs.io/en/latest/best_practices/tool_xml.html
+.. _planemo-autoupdate: https://github.com/planemo-autoupdate/autoupdate

docs/commands.rst

@@ -7,6 +7,7 @@ implemented as a subcommand of the ``planemo`` executable. This section of the
 documentation describes these commands.
 
 
+.. include:: commands/autoupdate.rst
 .. include:: commands/ci_find_repos.rst
 .. include:: commands/ci_find_tools.rst
 .. include:: commands/clone.rst

docs/commands/autoupdate.rst

@@ -0,0 +1,221 @@
+
+``autoupdate`` command
+======================================
+
+This section is auto-generated from the help text for the planemo command
+``autoupdate``. This help message can be generated with ``planemo autoupdate
+--help``.
+
+**Usage**::
+
+    planemo autoupdate [OPTIONS] TOOL_PATH
+
+**Help**
+
+Auto-update tool requirements by checking against Conda and updating if newer versions are available.
+**Options**::
+
+
+      --dry-run                       Perform a dry run autoupdate without modifying
+                                      the XML files.
+    
+      -r, --recursive                 Recursively perform command for
+                                      subdirectories.
+    
+      --test                          Test updated XML files.
+      --skiplist TEXT                 Skiplist file, containing a list of tools for
+                                      which autoupdate should be skipped.
+    
+      --skip_requirements TEXT        Comma-separated list of requirements which
+                                      should be not be updated. Default is
+                                      python,r-base,perl.
+    
+      --update_test_data              Update test-data directory with job outputs
+                                      (normally written to directory
+                                      --job_output_files if specified.)
+    
+      --paste_test_data_paths / --no_paste_test_data_paths
+                                      By default Planemo will use or not use
+                                      Galaxy's path paste option to load test data
+                                      into a history based on the engine type it is
+                                      targeting. This can override the logic to
+                                      explicitly enable or disable path pasting.
+    
+      --test_output PATH              Output test report (HTML - for humans)
+                                      defaults to tool_test_output.html.
+    
+      --test_output_text PATH         Output test report (Basic text - for display
+                                      in CI)
+    
+      --test_output_markdown PATH     Output test report (Markdown style - for
+                                      humans & computers)
+    
+      --test_output_xunit PATH        Output test report (xunit style - for CI
+                                      systems
+    
+      --test_output_junit PATH        Output test report (jUnit style - for CI
+                                      systems
+    
+      --test_output_allure DIRECTORY  Output test allure2 framework resutls
+      --test_output_json PATH         Output test report (planemo json) defaults to
+                                      tool_test_output.json.
+    
+      --job_output_files DIRECTORY    Write job outputs to specified directory.
+      --summary [none|minimal|compact]
+                                      Summary style printed to planemo's standard
+                                      output (see output reports for more complete
+                                      summary). Set to 'none' to disable completely.
+    
+      --galaxy_root DIRECTORY         Root of development galaxy directory to
+                                      execute command with.
+    
+      --galaxy_python_version [3|3.6|3.7|3.8|3.9]
+                                      Python version to start Galaxy under
+      --extra_tools PATH              Extra tool sources to include in Galaxy's tool
+                                      panel (file or directory). These will not be
+                                      linted/tested/etc... but they will be
+                                      available to workflows and for interactive
+                                      use.
+    
+      --install_galaxy                Download and configure a disposable copy of
+                                      Galaxy from github.
+    
+      --galaxy_branch TEXT            Branch of Galaxy to target (defaults to
+                                      master) if a Galaxy root isn't specified.
+    
+      --galaxy_source TEXT            Git source of Galaxy to target (defaults to
+                                      the official galaxyproject github source if a
+                                      Galaxy root isn't specified.
+    
+      --skip_venv                     Do not create or source a virtualenv
+                                      environment for Galaxy, this should be used to
+                                      preserve an externally configured virtual
+                                      environment or conda environment.
+    
+      --no_cache_galaxy               Skip caching of Galaxy source and dependencies
+                                      obtained with --install_galaxy. Not caching
+                                      this results in faster downloads (no git) - so
+                                      is better on throw away instances such with
+                                      TravisCI.
+    
+      --no_cleanup                    Do not cleanup temp files created for and by
+                                      Galaxy.
+    
+      --galaxy_email TEXT             E-mail address to use when launching single-
+                                      user Galaxy server.
+    
+      --docker / --no_docker          Run Galaxy tools in Docker if enabled.
+      --docker_cmd TEXT               Command used to launch docker (defaults to
+                                      docker).
+    
+      --docker_sudo / --no_docker_sudo
+                                      Flag to use sudo when running docker.
+      --docker_host TEXT              Docker host to target when executing docker
+                                      commands (defaults to localhost).
+    
+      --docker_sudo_cmd TEXT          sudo command to use when --docker_sudo is
+                                      enabled (defaults to sudo).
+    
+      --mulled_containers, --biocontainers
+                                      Test tools against mulled containers (forces
+                                      --docker).
+    
+      --job_config_file FILE          Job configuration file for Galaxy to target.
+      --tool_dependency_dir DIRECTORY
+                                      Tool dependency dir for Galaxy to target.
+      --test_data DIRECTORY           test-data directory to for specified tool(s).
+      --tool_data_table PATH          tool_data_table_conf.xml file to for specified
+                                      tool(s).
+    
+      --dependency_resolvers_config_file FILE
+                                      Dependency resolver configuration for Galaxy
+                                      to target.
+    
+      --brew_dependency_resolution    Configure Galaxy to use plain brew dependency
+                                      resolution.
+    
+      --shed_dependency_resolution    Configure Galaxy to use brewed Tool Shed
+                                      dependency resolution.
+    
+      --no_dependency_resolution      Configure Galaxy with no dependency resolvers.
+      --conda_prefix DIRECTORY        Conda prefix to use for conda dependency
+                                      commands.
+    
+      --conda_exec FILE               Location of conda executable.
+      --conda_debug                   Enable more verbose conda logging.
+      --conda_channels, --conda_ensure_channels TEXT
+                                      Ensure conda is configured with specified
+                                      comma separated list of channels.
+    
+      --conda_use_local               Use locally built packages while building
+                                      Conda environments.
+    
+      --conda_dependency_resolution   Configure Galaxy to use only conda for
+                                      dependency resolution.
+    
+      --conda_copy_dependencies       Conda dependency resolution for Galaxy will
+                                      copy dependencies instead of attempting to
+                                      link them.
+    
+      --conda_auto_install / --no_conda_auto_install
+                                      Conda dependency resolution for Galaxy will
+                                      attempt to install requested but missing
+                                      packages.
+    
+      --conda_auto_init / --no_conda_auto_init
+                                      Conda dependency resolution for Galaxy will
+                                      auto install conda itself using miniconda if
+                                      not availabe on conda_prefix.
+    
+      --profile TEXT                  Name of profile (created with the
+                                      profile_create command) to use with this
+                                      command.
+    
+      --postgres                      Use postgres database type.
+      --database_type [postgres|postgres_docker|sqlite|auto]
+                                      Type of database to use for profile - 'auto',
+                                      'sqlite', 'postgres', and 'postgres_docker'
+                                      are available options. Use postgres to use an
+                                      existing postgres server you user can access
+                                      without a password via the psql command. Use
+                                      postgres_docker to have Planemo manage a
+                                      docker container running postgres. Data with
+                                      postgres_docker is not yet persisted past when
+                                      you restart the docker container launched by
+                                      Planemo so be careful with this option.
+    
+      --postgres_psql_path TEXT       Name or or path to postgres client binary
+                                      (psql).
+    
+      --postgres_database_user TEXT   Postgres username for managed development
+                                      databases.
+    
+      --postgres_database_host TEXT   Postgres host name for managed development
+                                      databases.
+    
+      --postgres_database_port TEXT   Postgres port for managed development
+                                      databases.
+    
+      --file_path DIRECTORY           Location for files created by Galaxy (e.g.
+                                      database/files).
+    
+      --database_connection TEXT      Database connection string to use for Galaxy.
+      --shed_tool_conf TEXT           Location of shed tools conf file for Galaxy.
+      --shed_tool_path TEXT           Location of shed tools directory for Galaxy.
+      --galaxy_single_user / --no_galaxy_single_user
+                                      By default Planemo will configure Galaxy to
+                                      run in single-user mode where there is just
+                                      one user and this user is automatically logged
+                                      it. Use --no_galaxy_single_user to prevent
+                                      Galaxy from running this way.
+    
+      --report_level [all|warn|error]
+      --report_xunit PATH             Output an XUnit report, useful for CI testing
+      --fail_level [warn|error]
+      -s, --skip TEXT                 Comma-separated list of lint tests to skip
+                                      (e.g. passing --skip 'citations,xml_order'
+                                      would skip linting of citations and best-
+                                      practice XML ordering.
+    
+      --help                          Show this message and exit.
+    

docs/planemo.commands.rst

@@ -4,6 +4,14 @@ planemo.commands package
 Submodules
 ----------
 
+planemo.commands.cmd\_autoupdate module
+---------------------------------------
+
+.. automodule:: planemo.commands.cmd_autoupdate
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 planemo.commands.cmd\_ci\_find\_repos module
 --------------------------------------------
 

docs/planemo.engine.rst

@@ -4,6 +4,14 @@ planemo.engine package
 Submodules
 ----------
 
+planemo.engine.autoupdate module
+--------------------------------
+
+.. automodule:: planemo.engine.autoupdate
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 planemo.engine.cwltool module
 -----------------------------
 

docs/planemo.rst

@@ -24,6 +24,14 @@ Subpackages
 Submodules
 ----------
 
+planemo.autoupdate module
+-------------------------
+
+.. automodule:: planemo.autoupdate
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 planemo.bioblend module
 -----------------------