New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Azbatch executor #1953
feat: Azbatch executor #1953
Commits on Oct 22, 2022
Commits on Oct 25, 2022
Commits on Oct 27, 2022
Commits on Nov 2, 2022
Commits on Nov 3, 2022
-
-
fix: fix false rerun triggering downstream of checkpoints due to spur…
…ious parameter, code or software env changes
-
-
snakemake.vim: add envmodules keyword (snakemake#1925)
### Description <!--Add envmodules keyword to vim syntax--> ### QC <!-- Make sure that you can tick the boxes below. --> * [ ] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [ ] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
-
fix: remove redundant dot in expand call in multiext documentation (s…
…nakemake#1921) ### Description This just removes a single period in the documentation section about the multiext function, so that this, with a redundant period: ``` expand("some/plot.{ext}", ext=[".pdf", ".svg", ".png"]) ``` becomes this, matching the expected output: ``` expand("some/plot{ext}", ext=[".pdf", ".svg", ".png"]) ``` Fixes snakemake#1012. ### QC <!-- Make sure that you can tick the boxes below. --> * [X] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [X] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
-
chore(main): release 7.16.2 (snakemake#1928)
🤖 I have created a release *beep* *boop* --- ## [7.16.2](snakemake/snakemake@v7.16.1...v7.16.2) (2022-10-26) ### Bug Fixes * fix false rerun triggering downstream of checkpoints due to spurious parameter, code or software env changes ([638ea86](snakemake@638ea86)) * remove redundant dot in expand call in multiext documentation ([snakemake#1921](snakemake#1921)) ([278beaa](snakemake@278beaa)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
-
feat: allow to define the cache mode per rule (this enables to exclud…
…e software envs from the caching hash value, which can be handy e.g. for download rules where the software version does not affect the result) (snakemake#1933) ### Description <!--Add a description of your PR here--> ### QC <!-- Make sure that you can tick the boxes below. --> * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
-
perf: cached os.pathconf() call in _record_path() (snakemake#1920)
### Description The value of `os.pathconf(self._incomplete_path, "PC_NAME_MAX")` is cached in an attribute `Persistence._max_len` -- this avoids repeatedly querying the operating system for the same configuration information every time `marked_incomplete()` is called on a file/job output. (Note: this will dramatically speed up Snakemake DAG building on compute clusters where `os.pathconf()` is slow to return and the number of files checked for incompleteness is very large.) ### QC <!-- Make sure that you can tick the boxes below. --> * [X] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [X] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
-
chore(main): release 7.17.0 (snakemake#1935)
🤖 I have created a release *beep* *boop* --- ## [7.17.0](snakemake/snakemake@v7.16.2...v7.17.0) (2022-10-27) ### Features * allow to define the cache mode per rule (this enables to exclude software envs from the caching hash value, which can be handy e.g. for download rules where the software version does not affect the result) ([snakemake#1933](snakemake#1933)) ([715e618](snakemake@715e618)) ### Performance Improvements * cached os.pathconf() call in _record_path() ([snakemake#1920](snakemake#1920)) ([551badb](snakemake@551badb)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
-
fix: change source cache entries to keep the original name and folder…
… structure, such that imports from e.g. scripts also work with remote modules (if specified as additional input files with workflow.source_path) (snakemake#1936) ### Description <!--Add a description of your PR here--> ### QC <!-- Make sure that you can tick the boxes below. --> * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
-
chore(main): release 7.17.1 (snakemake#1937)
🤖 I have created a release *beep* *boop* --- ## [7.17.1](snakemake/snakemake@v7.17.0...v7.17.1) (2022-10-28) ### Bug Fixes * change source cache entries to keep the original name and folder structure, such that imports from e.g. scripts also work with remote modules (if specified as additional input files with workflow.source_path) ([snakemake#1936](snakemake#1936)) ([c34f3f6](snakemake@c34f3f6)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
-
fix: Consider source cache when setting search path for python script…
…s. This allows to import from Python modules next to scripts while deploying the workflow as a snakemake module, even from remote locations. (snakemake#1940) ### QC <!-- Make sure that you can tick the boxes below. --> * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
-
chore(main): release 7.17.2 (snakemake#1941)
🤖 I have created a release *beep* *boop* --- ## [7.17.2](snakemake/snakemake@v7.17.1...v7.17.2) (2022-10-28) ### Bug Fixes * Consider source cache when setting search path for python scripts. This allows to import from Python modules next to scripts while deploying the workflow as a snakemake module, even from remote locations. ([snakemake#1940](snakemake#1940)) ([27be1d4](snakemake@27be1d4)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
-
-
feat: first try to match output files against input files while persi…
…sting wildcard values from the consuming job. This can dramatically reduce ambiuity problems. Thanks to @descostesn! (snakemake#1939) Only fall back to generic regex based matching if the wildcard based approach does not work. This is inspired by a chat with @descostesn. <!-- Make sure that you can tick the boxes below. --> * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
-
chore(main): release 7.18.0 (snakemake#1944)
🤖 I have created a release *beep* *boop* --- ## [7.18.0](snakemake/snakemake@v7.17.2...v7.18.0) (2022-10-31) ### Features * first try to match output files against input files while persisting wildcard values from the consuming job. This can dramatically reduce ambiuity problems. Thanks to [@descostesn](https://github.com/descostesn)! ([snakemake#1939](snakemake#1939)) ([d093907](snakemake@d093907)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
-
chore(deps): Bump marocchino/sticky-pull-request-comment from 2.2.1 t…
…o 2.3.0 (snakemake#1945) Bumps [marocchino/sticky-pull-request-comment](https://github.com/marocchino/sticky-pull-request-comment) from 2.2.1 to 2.3.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/marocchino/sticky-pull-request-comment/releases">marocchino/sticky-pull-request-comment's releases</a>.</em></p> <blockquote> <h2>v2.3.0</h2> <ul> <li>Support glob path</li> <li>Add <code>follow_symbolic_links</code> for path</li> <li>Add <code>ignore_empty</code> for skip empty body</li> <li>Update README for new output syntax</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/marocchino/sticky-pull-request-comment/commit/adca94abcaf73c10466a71cc83ae561fd66d1a56"><code>adca94a</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/marocchino/sticky-pull-request-comment/issues/799">#799</a> from marocchino/dependabot/npm_and_yarn/octokit/graph...</li> <li><a href="https://github.com/marocchino/sticky-pull-request-comment/commit/af76947473de144ee8fd8a8a2a0e5b2f392f5626"><code>af76947</code></a> build(deps): Bump <code>@octokit/graphql-schema</code> from 12.11.0 to 12.21.0</li> <li><a href="https://github.com/marocchino/sticky-pull-request-comment/commit/e5883ea53c71dc8e52b346fd5c883eb1ea63ece7"><code>e5883ea</code></a> 🔀 Merge pull request <a href="https://github-redirect.dependabot.com/marocchino/sticky-pull-request-comment/issues/782">#782</a> from marocchino/dependabot/npm_and_yarn/eslint-plug...</li> <li><a href="https://github.com/marocchino/sticky-pull-request-comment/commit/2520dca5b1f09dab5376fc18715ff64dab2ceebb"><code>2520dca</code></a> 🔀 Merge pull request <a href="https://github-redirect.dependabot.com/marocchino/sticky-pull-request-comment/issues/790">#790</a> from marocchino/dependabot/npm_and_yarn/typescript-...</li> <li><a href="https://github.com/marocchino/sticky-pull-request-comment/commit/f4ae94070a1b2cc47b7a126d9271c48c3dd7e803"><code>f4ae940</code></a> 🔀 Merge pull request <a href="https://github-redirect.dependabot.com/marocchino/sticky-pull-request-comment/issues/793">#793</a> from marocchino/dependabot/npm_and_yarn/jest-circus...</li> <li><a href="https://github.com/marocchino/sticky-pull-request-comment/commit/97e16fd48982b5a81b0ef88b530b54e8995e27c4"><code>97e16fd</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/marocchino/sticky-pull-request-comment/issues/792">#792</a> from samkio/feature/glob</li> <li><a href="https://github.com/marocchino/sticky-pull-request-comment/commit/da2f89048f71a00bda8f6df93622691adf401be3"><code>da2f890</code></a> build(deps-dev): Bump eslint-plugin-jest from 27.1.2 to 27.1.3</li> <li><a href="https://github.com/marocchino/sticky-pull-request-comment/commit/ba82cbc35c1c35bab4a3bca541920a5d9e4d6f60"><code>ba82cbc</code></a> build(deps-dev): Bump <code>@typescript-eslint/parser</code> from 5.40.1 to 5.41.0</li> <li><a href="https://github.com/marocchino/sticky-pull-request-comment/commit/b8644f1ac559a999648c96168f58c769bb1af49a"><code>b8644f1</code></a> 🔀 Merge pull request <a href="https://github-redirect.dependabot.com/marocchino/sticky-pull-request-comment/issues/789">#789</a> from marocchino/dependabot/npm_and_yarn/eslint-8.26.0</li> <li><a href="https://github.com/marocchino/sticky-pull-request-comment/commit/95c224a6ab058ad5a70e45bb6d16f8b221078bfb"><code>95c224a</code></a> 🔀 Merge pull request <a href="https://github-redirect.dependabot.com/marocchino/sticky-pull-request-comment/issues/797">#797</a> from marocchino/dependabot/npm_and_yarn/types/node-...</li> <li>Additional commits viewable in <a href="https://github.com/marocchino/sticky-pull-request-comment/compare/v2.2.1...v2.3.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=marocchino/sticky-pull-request-comment&package-manager=github_actions&previous-version=2.2.1&new-version=2.3.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
-
-
Commits on Nov 4, 2022
Commits on Nov 5, 2022
Commits on Nov 9, 2022
Commits on Nov 18, 2022
Commits on Dec 8, 2022
Commits on Dec 9, 2022
Commits on Dec 14, 2022
Commits on Dec 15, 2022
Commits on Dec 19, 2022
Commits on Dec 22, 2022
Commits on Dec 28, 2022
Commits on Dec 29, 2022
Commits on Jan 4, 2023
Commits on Jan 10, 2023
Commits on Jan 17, 2023
Commits on Feb 7, 2023
-
refactor: update versioneer from 0.18->0.28, clean-up setup.py/.cfg (s…
…nakemake#2065) This PR does two main things: - Update the vendored versioneer, which uses a configuration in `pyproject.toml`, so clean-up `setup.cfg` - A follow-up from [snakemake#2026](snakemake#2026) is that most of `setup.py` was to be cleaned-up, but this somehow missed that PR! Keep the repeated `name="snakemake"` metadata in setup.py for now to avoid a [GitHub bug](https://github.com/orgs/community/discussions/6456) <!-- Make sure that you can tick the boxes below. --> * [ ] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [ ] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
-
feat: allow for human friendly resource definitions (e.g. mem="5GB", …
…runtime="1d") (snakemake#1861) ### Description <!--Add a description of your PR here--> ### QC <!-- Make sure that you can tick the boxes below. --> * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
-
fix: Catch missing error stream in Slurm executor (snakemake#2063)
### Description I ran into a bug in the `job_status` method in which the variable `sacct_error` is set to `None`, but then later is printed using `sacct_error.strip()`, which throws this error (with gently redacted filepaths): ``` Job 146 has been submitted with SLURM jobid 154129239 (log: .snakemake/slurm_logs/rule_sra_to_fastq/154129239.log) Traceback (most recent call last): File "/path_to_project/venv/lib/python3.8/site-packages/snakemake/executors/slurm/slurm_submit.py", line 310, in job_status out = subprocess.check_output( File "/shared/anaconda/python3-2020.07-mamba/lib/python3.8/subprocess.py", line 411, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/shared/anaconda/python3-2020.07-mamba/lib/python3.8/subprocess.py", line 512, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command 'scontrol show jobid -dd 154129189' returned non-zero exit status 1. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/path_to_project/venv/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 838, in _wait_thread asyncio.run(self._wait_for_jobs()) File "/shared/anaconda/python3-2020.07-mamba/lib/python3.8/asyncio/runners.py", line 43, in run return loop.run_until_complete(main) File "/shared/anaconda/python3-2020.07-mamba/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/path_to_project/venv/lib/python3.8/site-packages/snakemake/executors/slurm/slurm_submit.py", line 354, in _wait_for_jobs status = await self.job_status(j.jobid) File "/path_to_project/venv/lib/python3.8/site-packages/snakemake/executors/slurm/slurm_submit.py", line 322, in job_status f"Error getting status of slurm job {jobid}:\n sacct error: {sacct_error.strip()}\n scontrol error: {e.stderr.strip()}" AttributeError: 'NoneType' object has no attribute 'strip' Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2023-01-13T024214.712457.snakemake.log 'NoneType' object has no attribute 'strip' ``` If `sacct_error` is still `None` on line 322, I think the only explanations would be: 1. The call to `subprocess.check_output` on line 292 returned an `IndexError`, which was caught on line 304 and ignored, or 2. The call to `subprocess.check_output` on line 292 returned `None` for the standard error stream, which it shouldn't have done: [That only happens](https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.subprocess.Process.stderr) if called with the `stderr=None` parameter, which it wasn't. There's nothing I can see that's checking whether `sacct_error` is `None`, so my first thought was just to change the declaration of the variable on line 289 to something like `sacct_error = '(none)'` so it prints nicely, but that looks funny *and* doesn't avoid scenario number 2 above, which I suppose could happen if the parameters are changed one day, or something freaky happens with the `subprocess.PIPE` that I honestly don't understand very well. In any case, this fix isn't particularly sophisticated, but it fixes the `AttributeError` I ran into without modifying any of the logic and accommodates a path (`except IndexError as e`) that's at least common enough to be included here. ### QC <!-- Make sure that you can tick the boxes below. --> * [ ] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [ ] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake). Co-authored-by: Johannes Köster <johannes.koester@tu-dortmund.de> Co-authored-by: Johannes Köster <johannes.koester@uni-due.de>
-
refactor: move static metadata (again) to setup.cfg (snakemake#2067)
### Description This moves the static project metadata (again) to setup.cfg, which has had long-standing support with [setuptools](https://setuptools.pypa.io/en/latest/userguide/declarative_config.html). This resolves an issue with limitations of project metadata in pyproject.toml by release-please. ### QC * [ ] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [ ] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
-
chore(main): release 7.20.0 (snakemake#2018)
🤖 I have created a release *beep* *boop* --- ## [7.20.0](snakemake/snakemake@v7.19.1...v7.20.0) (2023-01-18) ### Features * add tes token ([snakemake#1966](snakemake#1966)) ([59a8fa0](snakemake@59a8fa0)) * Add token auth to GitLab/GitHub hosting providers ([snakemake#1761](snakemake#1761)) ([e03a3b4](snakemake@e03a3b4)), closes [snakemake#1301](snakemake#1301) * allow for human friendly resource definitions (e.g. mem="5GB", runtime="1d") ([snakemake#1861](snakemake#1861)) ([24610ac](snakemake@24610ac)) ### Bug Fixes * 🐛 - fix hyperlink ([snakemake#2046](snakemake#2046)) ([9519d31](snakemake@9519d31)) * Catch missing error stream in Slurm executor ([snakemake#2063](snakemake#2063)) ([c21fc7e](snakemake@c21fc7e)) * correctly parse empty values in config cli ([snakemake#2032](snakemake#2032)) ([1b0689d](snakemake@1b0689d)) * Correctly parse UserDicts in executors ([snakemake#2016](snakemake#2016)) ([e3926fa](snakemake@e3926fa)) * Fix handling of --jobs in no-exec state ([snakemake#2029](snakemake#2029)) ([e8e8222](snakemake@e8e8222)) * make `--show-failed-logs` handle empty log files ([snakemake#2039](snakemake#2039)) ([683c6f2](snakemake@683c6f2)), closes [snakemake#2023](snakemake#2023) * make python version check more robust ([snakemake#2058](snakemake#2058)) ([e685621](snakemake@e685621)) * parsing error when last line is comment ([snakemake#2054](snakemake#2054)) ([a928dd4](snakemake@a928dd4)) * prevent overriding of retries when set to 0 ([snakemake#2053](snakemake#2053)) ([a328f3e](snakemake@a328f3e)) * propagate attempt count from group to subjobs ([snakemake#2052](snakemake#2052)) ([da3f1c0](snakemake@da3f1c0)) * remove overflow from rulegraph div in report ([9a0aaa7](snakemake@9a0aaa7)) * skip type checks of missing dir in touch mode ([snakemake#2051](snakemake#2051)) ([ae00c25](snakemake@ae00c25)) * slurm default_resources quoting ([snakemake#2043](snakemake#2043)) ([47d3fc3](snakemake@47d3fc3)) * Update list of python versions in classifiers ([snakemake#2020](snakemake#2020)) ([7a98100](snakemake@7a98100)) * use short argument name for `--chdir` for compatibility with Slurm <=v17 ([snakemake#2040](snakemake#2040)) ([a9ed3ec](snakemake@a9ed3ec)) ### Documentation * Fix typo in SLURM help text ([snakemake#2049](snakemake#2049)) ([79b7025](snakemake@79b7025)) * mention XDG_CACHE_HOME ([snakemake#2057](snakemake#2057)) ([ec2ef45](snakemake@ec2ef45)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
-
feat: ability to encode paramspaces into a single wildcard, via the n…
…ewly introduced `single_wildcard` argument of `Paramspace`. (snakemake#2069) ### Description <!--Add a description of your PR here--> ### QC <!-- Make sure that you can tick the boxes below. --> * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
-
fix: Postprocess job groups in toposorted order for correct touch tim…
…es (snakemake#2073) ### Description If I understand the mechanics correctly, jobs in job groups executed on clusters get "postprocessed", and as part of this their output files are touched. However, the order in which this happened was not necessarily a topological order, meaning that output files of dependent rules could be touched in the "wrong" order. See snakemake#789 This simple change just ensures that jobs are post-processed in topological order, which should solve the above issue. If desirable, I could also writing a test for the above behaviour to catch potential future issues. ### QC * [ ] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake). Co-authored-by: Johannes Köster <johannes.koester@uni-due.de>
-
fix: more robust parsing of sacct output in slurm executor (snakemake…
…#2036) Fix error in parse jobid and jobidraw. jobid and jobidraw might be different. ### Description --slurm command report such error: ``` File "~/.local/lib/python3.11/site-packages/snakemake/executors/slurm/slurm_submit.py", line 328, in job_status return res[jobid] # == status ~~~^^^^^^^ KeyError: '1043166' ``` This is because sacct command report JobId but not JobIdRaw in the first column, which cause error in indexing the job status in the res dict. For example, ``` $ sacct -P -n --format=JobId,JobIdRaw,State -j 1043166 1043074_91|1043166|COMPLETED 1043074_91.batch|1043166.batch|COMPLETED 1043074_91.extern|1043166.extern|COMPLETED 1043074_91.0|1043166.0|COMPLETED ``` replacing `-b` with `--format=JobIdRaw,State` can have a more robust output. Co-authored-by: Johannes Köster <johannes.koester@tu-dortmund.de>
-
chore(deps): Bump amannn/action-semantic-pull-request from 4.6.0 to 5…
….0.2 (snakemake#1914) Bumps [amannn/action-semantic-pull-request](https://github.com/amannn/action-semantic-pull-request) from 4.6.0 to 5.0.2. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/amannn/action-semantic-pull-request/releases">amannn/action-semantic-pull-request's releases</a>.</em></p> <blockquote> <h2>v5.0.2</h2> <h3><a href="https://github.com/amannn/action-semantic-pull-request/compare/v5.0.1...v5.0.2">5.0.2</a> (2022-10-17)</h3> <h3>Bug Fixes</h3> <ul> <li>Upgrade <code>@actions/core</code> to avoid deprecation warnings (<a href="https://github-redirect.dependabot.com/amannn/action-semantic-pull-request/issues/208">#208</a>) (<a href="https://github.com/amannn/action-semantic-pull-request/commit/91f4126c9e8625b9cadd64b02a03018fa22fc498">91f4126</a>)</li> </ul> <h2>v5.0.1</h2> <h3><a href="https://github.com/amannn/action-semantic-pull-request/compare/v5.0.0...v5.0.1">5.0.1</a> (2022-10-14)</h3> <h3>Bug Fixes</h3> <ul> <li>Upgrade GitHub Action to use Node v16 (<a href="https://github-redirect.dependabot.com/amannn/action-semantic-pull-request/issues/207">#207</a>) (<a href="https://github.com/amannn/action-semantic-pull-request/commit/6282ee339b067cb8eab05026f91153f873ad37fb">6282ee3</a>)</li> </ul> <h2>v5.0.0</h2> <h2><a href="https://github.com/amannn/action-semantic-pull-request/compare/v4.6.0...v5.0.0">5.0.0</a> (2022-10-11)</h2> <h3>⚠ BREAKING CHANGES</h3> <ul> <li>Enum options need to be newline delimited (to allow whitespace within them) (<a href="https://github-redirect.dependabot.com/amannn/action-semantic-pull-request/issues/205">#205</a>)</li> </ul> <h3>Features</h3> <ul> <li>Enum options need to be newline delimited (to allow whitespace within them) (<a href="https://github-redirect.dependabot.com/amannn/action-semantic-pull-request/issues/205">#205</a>) (<a href="https://github.com/amannn/action-semantic-pull-request/commit/c906fe1e5a4bcc61624931ca94da9672107bd448">c906fe1</a>)</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/amannn/action-semantic-pull-request/blob/main/CHANGELOG.md">amannn/action-semantic-pull-request's changelog</a>.</em></p> <blockquote> <h3><a href="https://github.com/amannn/action-semantic-pull-request/compare/v5.0.1...v5.0.2">5.0.2</a> (2022-10-17)</h3> <h3>Bug Fixes</h3> <ul> <li>Upgrade <code>@actions/core</code> to avoid deprecation warnings (<a href="https://github-redirect.dependabot.com/amannn/action-semantic-pull-request/issues/208">#208</a>) (<a href="https://github.com/amannn/action-semantic-pull-request/commit/91f4126c9e8625b9cadd64b02a03018fa22fc498">91f4126</a>)</li> </ul> <h3><a href="https://github.com/amannn/action-semantic-pull-request/compare/v5.0.0...v5.0.1">5.0.1</a> (2022-10-14)</h3> <h3>Bug Fixes</h3> <ul> <li>Upgrade GitHub Action to use Node v16 (<a href="https://github-redirect.dependabot.com/amannn/action-semantic-pull-request/issues/207">#207</a>) (<a href="https://github.com/amannn/action-semantic-pull-request/commit/6282ee339b067cb8eab05026f91153f873ad37fb">6282ee3</a>)</li> </ul> <h2><a href="https://github.com/amannn/action-semantic-pull-request/compare/v4.6.0...v5.0.0">5.0.0</a> (2022-10-11)</h2> <h3>⚠ BREAKING CHANGES</h3> <ul> <li>Enum options need to be newline delimited (to allow whitespace within them) (<a href="https://github-redirect.dependabot.com/amannn/action-semantic-pull-request/issues/205">#205</a>)</li> </ul> <h3>Features</h3> <ul> <li>Enum options need to be newline delimited (to allow whitespace within them) (<a href="https://github-redirect.dependabot.com/amannn/action-semantic-pull-request/issues/205">#205</a>) (<a href="https://github.com/amannn/action-semantic-pull-request/commit/c906fe1e5a4bcc61624931ca94da9672107bd448">c906fe1</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/amannn/action-semantic-pull-request/commit/01d5fd8a8ebb9aafe902c40c53f0f4744f7381eb"><code>01d5fd8</code></a> chore: Release 5.0.2 [skip ci]</li> <li><a href="https://github.com/amannn/action-semantic-pull-request/commit/91f4126c9e8625b9cadd64b02a03018fa22fc498"><code>91f4126</code></a> fix: Upgrade <code>@actions/core</code> to avoid deprecation warnings (<a href="https://github-redirect.dependabot.com/amannn/action-semantic-pull-request/issues/208">#208</a>)</li> <li><a href="https://github.com/amannn/action-semantic-pull-request/commit/0a457e272673a33e52c9ade9318b3c115e566f60"><code>0a457e2</code></a> ci: Fix branch name in dist check (<a href="https://github-redirect.dependabot.com/amannn/action-semantic-pull-request/issues/209">#209</a>)</li> <li><a href="https://github.com/amannn/action-semantic-pull-request/commit/570204e54c9030a6cb7ef781c492e76c1798d1c0"><code>570204e</code></a> chore: Release 5.0.1 [skip ci]</li> <li><a href="https://github.com/amannn/action-semantic-pull-request/commit/6282ee339b067cb8eab05026f91153f873ad37fb"><code>6282ee3</code></a> fix: Upgrade GitHub Action to use Node v16 (<a href="https://github-redirect.dependabot.com/amannn/action-semantic-pull-request/issues/207">#207</a>)</li> <li><a href="https://github.com/amannn/action-semantic-pull-request/commit/7c194c28652f0faf98ad437c6cf291406d387b43"><code>7c194c2</code></a> docs: Use latest major [skip ci]</li> <li><a href="https://github.com/amannn/action-semantic-pull-request/commit/5369185dc9812f174fa58b03b13b79920b68f8e1"><code>5369185</code></a> chore: Release 5.0.0 [skip ci]</li> <li><a href="https://github.com/amannn/action-semantic-pull-request/commit/c906fe1e5a4bcc61624931ca94da9672107bd448"><code>c906fe1</code></a> feat!: Enum options need to be newline delimited (to allow whitespace within ...</li> <li><a href="https://github.com/amannn/action-semantic-pull-request/commit/b314c1bec341c714425c0aa43e142b35c12759a0"><code>b314c1b</code></a> docs: Improve example for composing outputs (<a href="https://github-redirect.dependabot.com/amannn/action-semantic-pull-request/issues/206">#206</a>)</li> <li>See full diff in <a href="https://github.com/amannn/action-semantic-pull-request/compare/v4.6.0...v5.0.2">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=amannn/action-semantic-pull-request&package-manager=github_actions&previous-version=4.6.0&new-version=5.0.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
-
fix: more robust encoding of params in persistent metadata storage. T…
…his way, pandas parameters do not lead to spurious rerun triggers. (snakemake#2080) ### Description <!--Add a description of your PR here--> ### QC <!-- Make sure that you can tick the boxes below. --> * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
-
feat: allow input, output, and params to be used in functions passed …
…to report mark arguments (snakemake#2081) ### Description <!--Add a description of your PR here--> ### QC <!-- Make sure that you can tick the boxes below. --> * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
-
chore(main): release 7.21.0 (snakemake#2074)
🤖 I have created a release *beep* *boop* --- ## [7.21.0](snakemake/snakemake@v7.20.0...v7.21.0) (2023-01-30) ### Features * ability to encode paramspaces into a single wildcard, via the newly introduced `single_wildcard` argument of `Paramspace`. ([snakemake#2069](snakemake#2069)) ([728ab3c](snakemake@728ab3c)) * allow input, output, and params to be used in functions passed to report mark arguments ([snakemake#2081](snakemake#2081)) ([93ff8b6](snakemake@93ff8b6)) ### Bug Fixes * more robust encoding of params in persistent metadata storage. This way, pandas parameters do not lead to spurious rerun triggers. ([snakemake#2080](snakemake#2080)) ([106a4c3](snakemake@106a4c3)) * more robust parsing of sacct output in slurm executor ([snakemake#2036](snakemake#2036)) ([fe651f8](snakemake@fe651f8)) * Postprocess job groups in toposorted order for correct touch times ([snakemake#2073](snakemake#2073)) ([10b5849](snakemake@10b5849)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
-
chore: testcase for parallel group job execution with slurm executor (s…
…nakemake#2092) ### Description <!--Add a description of your PR here--> ### QC <!-- Make sure that you can tick the boxes below. --> * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
-
-