Merging with upstream #1
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Update docs to hide Mesos * address review comments * remove invisible characters? * replace mesos in more places * Document Kubernetes-managed autoscaling, with in-workflow Mesos autoscaling as deprected * Reword some documentation and messages * Chase out more Mesoses * Don't insist on processes actually running promptly in parallel * Ask for a compatible set of Sphinx packages * Keep back astroid We can't use astroid 3 until sphinx-autoapi releases a fix for readthedocs/sphinx-autoapi#392 --------- Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
…4600) This will fix DataBiosphere#4599 by making the mock leader thread safe.
…4589) * monkeypatch coerce for workflow related nodes * Fix task inputs string coerce * Disable kubernetes * Comment out cwl kubernetes * Maybe markers are wrong and comment out cactus-on-kubernetes * Add docstrings to changed functions + change input list to dict * Deal with nonetype --------- Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
* Reorganize tests and move integration tests to scheduled pipeline runs * Also handle tags
* Centralize defaults * Add requirements * Grab logLevel grabbed logLevel used to be the default in Config(), so grab effective logLevel that is set * Satisfy mypy mypy might still complain about missing stubs for configargparser though * Fix wrong default * add config tool * temp fix config sets defaults but so does argparse, runs twice in workflows but deals with tests * Fix create_config for tests instead * Fix setting of config defaults * Go back to previous method, create defaults at init * Fix default cli options set * Centralize, config util, and read properly * Fix type hinting to support 3.9 * mypy * Fix cwl edge case * Fix tests * fix typos, always generate config, fix some tests * Remove subprocess as maybe tests are flaky on CI with it? * just run quick_test_offline * make CI print stuff * Harden default config creation against races * Cleanup and argument renaming * Fix bad yaml and toil status bug * Fix mypy * Change behavior of --stats and --clean * Change test behavior as options namespace and config now have the same behavior * Put forgotten line ouch * Batchsystem, requirements, fixes for tests * Mypy conformance * Mypy conformance * Fix retryCount argument and kubernetesPodTimeout type * Only run batchsystem and slurm_test tests on CI * Whoops, this implementation never worked * Add pyyaml to requirements for slurm to pass * Add rest of gitlab CI back and run all tests * Update stub file to be compatible with updated mypy * Fix environment CLI option * Update provisioner test to use configargparse * Code cleanup and add jobstore_as_flag to DefaultArgumentParser etc * Fix toil config test * Add suggestions * Deprecate options, add underscore CLI options only for newly deprecated options * Update docs/argparse help and fix bug with deprecated options also make most generic arg as default for runLocalJobsOnWorkers * Add config file section to docs * Remove upper bound for ruamel requirements * Remove redundancies and improve disableCaching's destination name * Update src/toil/batchSystems/kubernetes.py Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> * Remove redundant line in status util * Remove comments in configargparse stub * Workaround to get action=append instead of nargs and get proper backwards compatibility Fix wrong name for link_imports and move_exports, remove new unused functions * Import SYS_MAX_SIZE from common rather than duplicating it * Mypy and syntax errors * Move config options back to the old naming syntax * Change names for link_imports and move_exports to camelCase options * Fix formatting * Bring back old --restart and --clean functionality where they collide and raise an error * Make debug less spammy and remove unused types * Disable kubernetes temporarily * Revert changes to --restart and --clean collision * Typo in tests * Change some comments and add member fields to config * Fix pickling error when jobstate file doesnt exist and fix threading error when lock file exists then disappears (DataBiosphere#4575) Co-authored-by: Brandon Walker <walkerbd@dali1.dali.hpc.ncats.nih.gov> Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> * Reduce the number of assert statements (DataBiosphere#4590) * Change all asserts to raising errors for central toil files Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> * Fix mypy and update docs to match options in common * Update src/toil/common.py Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> --------- Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Co-authored-by: Brandon Walker <43654521+misterbrandonwalker@users.noreply.github.com> Co-authored-by: Brandon Walker <walkerbd@dali1.dali.hpc.ncats.nih.gov>
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
* Ensure sibling files stay sibling files when downloaded * Fix incorrect argument order * Fix directory collisions with sibling files
* Make sure the batch logs dir exists if it is set * Test Slurm with nonexistent --batchLogsDir
…phere#4637) * Report errors in WDL using MiniWDL's error location printer * Decorate actual tasks with fancy WDL error reporting * Slap WDL error reporting on main * Remove banned ignore comment
* Remove python 3.7 and add python 3.11 and make python3.11 the main python package * Move main python package back to 3.9 * Incude python3.11 in docker * Test 3.11 in CI * Add python3.11 to CI dockerfile * Add 3.11 to setup.py and debugging statements * Python 3.7 backwards compatibility * Update to py 3.12 and run 3.12 on gitlab CI * Comment out fstring and try importlib * Debug lint * Ensure mypy is using python3.12 * Print python version beofre mypy * Fix virtualenv, pip for python3.12 * Get rid of mesos tests/builds * 3.12 * Revert debug change * Go back to 3.11 and update docker package to make requests work again * use an available htcondor package closest to 3.10 version * update htcondor for all * get pip for all python versions * get virtualenv for all python versions * needs specific ordering * Separate mesos tests * remove 3.7 from CI image * Remove debug statement from makefile
* Parse config file separately from rest of args * Mypy * update configargparse stub * Dont try to eat cwl arguments * Use simpler workaround * Revert to just CWL * Change REMAINDER to "*", add help statements and test command line inputs * Remove extradockergroup name * Declare type * Add proper relative path to cwl file * Remove unnecessary test --------- Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
…DataBiosphere#4659) Updates the requirements on [ruamel-yaml]() to permit the latest version. --- updated-dependencies: - dependency-name: ruamel-yaml dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Properly build 3.11, fix dependencies and move aws stubs/mock into dev * only keep htcondor installs in appliance builds * Remove unused import * Fix extras_require syntax
…gs happen to our mutex file (DataBiosphere#4656)
* Bump mypy from 1.5.1 to 1.6.1 Bumps [mypy](https://github.com/python/mypy) from 1.5.1 to 1.6.1. - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](python/mypy@v1.5.1...v1.6.1) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * type fix --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Michael R. Crusoe <michael.crusoe@gmail.com>
This should fix DataBiosphere#4648 by turning on the batch system tests again. The Mesos-specific ones are already moved elsewhere. Co-authored-by: Lon Blauvelt <lblauvel@ucsc.edu>
Bumps [miniwdl](https://github.com/chanzuckerberg/miniwdl) from 1.10.0 to 1.11.1. - [Release notes](https://github.com/chanzuckerberg/miniwdl/releases) - [Commits](chanzuckerberg/miniwdl@v1.10.0...v1.11.1) --- updated-dependencies: - dependency-name: miniwdl dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Implement new batch system finding API and plugin scan * Satisfy MyPy * Implement deprecation for the old constants * Get plugin loader to actually load, and drop TES * Remove TES Kubernetes setup we don't use * Stop asking for needs_tes --------- Co-authored-by: Lon Blauvelt <lblauvel@ucsc.edu>
* skip unwanted networkx version * Limit to released major versions of networkx --------- Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
* Add a bunch of value resolving logging * Quiet debugging a bit * Move default setting for workflows so it works on subworkflows * Remember to keep making a ToilFsAccess on the leader * Satisfy MyPy * Stop giving CWL containers directories full of broken symlinks * Update test to expect no symlinks * Move CWL integration tests for bioconda/biocontainers to integration test runs * Wrap mkdtemp to fix DataBiosphere#4644 * Sort imports in example scripts * Use absolute-ized paths for work and coordination directories
…here#4685) Bumps [cwltool](https://github.com/common-workflow-language/cwltool) from 3.1.20231020140205 to 3.1.20231114134824. - [Release notes](https://github.com/common-workflow-language/cwltool/releases) - [Commits](common-workflow-language/cwltool@3.1.20231020140205...3.1.20231114134824) --- updated-dependencies: - dependency-name: cwltool dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump mypy from 1.6.1 to 1.7.0 Bumps [mypy](https://github.com/python/mypy) from 1.6.1 to 1.7.0. - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](python/mypy@v1.6.1...v1.7.0) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * mypy 1.7.0 type updates * format modified files * remove unused imports --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Michael R. Crusoe <michael.crusoe@gmail.com>
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
* Reenable kubernetes tests that don't require a local cluster, eg CWL on ARM and Cactus integration on kubernetes * Disable CWL kubernetes * enable cactus tests * Add to scheduled integration tests * Add forgotten file * Remove print statements * Remove unnecessary env var and move file * Run test when updated Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> * update gitlab * Fix typo in path * Add virtualenv and prepare build to gitlab CI to run tests properly * add gitlab setup scripts * add gitlab setup scripts --------- Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Bumps [mypy](https://github.com/python/mypy) from 1.7.0 to 1.7.1. - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](python/mypy@v1.7.0...v1.7.1) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… the workflow (DataBiosphere#4873) * Debug envvar * add error to message * Add logic for unexpected background thread failure * Set block back to true * Don't duplicate thread exception message and print at end * Revert "Debug envvar" This reverts commit 1339285. * Apply suggestions from code review Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Bumps [mypy](https://github.com/python/mypy) from 1.9.0 to 1.10.0. - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](python/mypy@1.9.0...v1.10.0) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…s started (DataBiosphere#4881) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Update wdltoil_test.py * Fix typo * Fix version for integration tests --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Take out boto2 from awsProvisioner.py * Add mypy stub file for s3 * Lazy import aws to avoid dependency if extra is not installed yet * Also lazy import in tests * Separate out wdl kubernetes test to avoid missing dependency * Add unittest main * Fix wdl CI to run separated tests * Fix typo in lookup * Update moto and remove leftover line in node.py * Remove all instances of boto * Fix issues with boto return types and grab attributes before deleting * Remove some unnecessary abstraction * Fix improperly types in ec2.py * Ensure UUID is a string for boto3 * No more boto * Remove comments * Move attribute initialization * Properly delete all attributes of the item * Move out pager and use pager for select to get around output limits * Turn getter into method * Remove comment in setup.py * Remove commented dead import * Remove stray boto import * Apply suggestions from code review Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> * Rename, rearrange some code * Revert not passing Value's to attributes when deleting attributes in SDB * Fix missed changed var names * Change ordering of jobstorexists exception to fix improper output on exception --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
* file incorrect file extensions. * fix typos --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Since DataBiosphere#4811 made the batch systems take the command as an argument, we now have to bump the major version to signal incompatibility with any old batch system plugins. Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Detect missing files at the offending step and announce the problem conspicuously * Log the offending expression * Resolve symlinks against container mounts during file virtualization * Try and forward along original virtualized filenames --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* pytz is not needed in Python 3.9+, or with the zoneinfo backport * make diff_mypy: quieter and target the correct branch * Linting. * Satisfy MyPy more (new MyPy?) --------- Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: DailyDreaming <lblauvel@ucsc.edu>
…sphere#4887) This should fix DataBiosphere#4886 by not suggesting to the user that "infinity" is an option value that can be used. It also explains the option intervals in words instead of interval notation, which people might not be expecting. Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Fix WDL option spelling and tolerate Cromwell-isms * Linting. * Satisfy MyPy more (new MyPy?) --------- Co-authored-by: DailyDreaming <lblauvel@ucsc.edu> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Remove wrapped CWL doc example. * Patch missing links. * Remove AWS dependant import/test from cwlTest.py. * Missing @slow. * Missing import. * Make SimpleDB retry on EndpointConnectionError * Linting. * Satisfy MyPy more (new MyPy?) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
* Add retries to flaky test * get rid of extra import --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Fix 3.8 import and remove dead comment in requirements.txt --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Lon Blauvelt <lblauvel@ucsc.edu>
* Add Python 3.12 to CI * Update sphinx-autoapi and astroid to deal with crash pylint-dev/pylint#8782 * Remove dead comment * Add rules to 3.11 build * update htcondor * Update use of HTcondor in appliance build * Ensure tests are instanced and don't jumble relative paths + debug logging * oops, update utilsTest too * is this a pytest issue? * Add some more log messages * Fix time.sleep * Remove the debug statement in docker
Bumps [flask-cors](https://github.com/corydolphin/flask-cors) from 4.0.0 to 4.0.1. - [Release notes](https://github.com/corydolphin/flask-cors/releases) - [Changelog](https://github.com/corydolphin/flask-cors/blob/main/CHANGELOG.md) - [Commits](corydolphin/flask-cors@4.0.0...4.0.1) --- updated-dependencies: - dependency-name: flask-cors dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…at (DataBiosphere#4912) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This reverts commit 460846d.
Bumps [miniwdl](https://github.com/chanzuckerberg/miniwdl) from 1.11.1 to 1.12.0. - [Release notes](https://github.com/chanzuckerberg/miniwdl/releases) - [Commits](chanzuckerberg/miniwdl@v1.11.1...v1.12.0) --- updated-dependencies: - dependency-name: miniwdl dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Add Python 3.12 to CI * Update sphinx-autoapi and astroid to deal with crash pylint-dev/pylint#8782 * Remove dead comment * Add rules to 3.11 build * update htcondor * Update use of HTcondor in appliance build * Ensure tests are instanced and don't jumble relative paths + debug logging * oops, update utilsTest too * is this a pytest issue? * Add some more log messages * Fix time.sleep * Remove the debug statement in docker * remove logger print statements in utilsTest.py and pin pytest * Up the timeout on some tests (possiby a timing issue) * Up the timeout on more tests * Up the pytest version again
…4926) Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
…4925) It looks like if you give away your debugger PIN, people can use your Werkzeug debugger. This is somehow a security issue and was apparently never fixed on Werkzeug 2. Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changelog Entry
To be copied to the draft changelog by merger:
Reviewer Checklist
issues/XXXX-fix-the-thingin the Toil repo, or from an external repo.camelCasethat want to be insnake_case.docs/running/{cliOptions,cwl,wdl}.rstMerger Checklist