Skip to content

Commit

Permalink
fix: ensure that log and benchmark files are uploaded to storage as w…
Browse files Browse the repository at this point in the history
…ell (#2545)

### Description

<!--Add a description of your PR here-->

### QC
<!-- Make sure that you can tick the boxes below. -->

* [x] The PR contains a test case for the changes or the changes are
already covered by an existing test case.
* [x] The documentation (`docs/`) is updated to reflect the changes or
this is not necessary (e.g. if the change does neither modify the
language nor the behavior or functionalities of Snakemake).
  • Loading branch information
johanneskoester committed Dec 15, 2023
1 parent 26eb4ba commit 6aabb5d
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 2 deletions.
9 changes: 8 additions & 1 deletion docs/getting_started/migration.rst
Expand Up @@ -22,7 +22,14 @@ Snakemake 8 removes the support for four syntactical elements, which are all off
* Support for marking output files as ``dynamic`` has been removed. You should instead use :ref:`checkpoints <snakefiles-checkpoints>`.
* Support for the ``version`` directive has been removed. You should use the :ref:`conda <integrated_package_management>` or :ref:`container <apptainer>` integration instead.
* Support for the ``subworkflow`` directive has been removed. You should use the :ref:`module directive <snakefiles-modules>` instead, which provides the same functionality in a more general way.
* Support for remote providers has been removed. You should use :ref:`storage plugins <storage-support>` instead.
* Support for remote providers has been removed. You should use :ref:`storage plugins <storage-support>` instead.
Most of the old remote providers have been migrated into the new storage plugins
(see the `Snakemake plugin catalog <https://snakemake.github.io/snakemake-plugin-catalog>`_.).
Two former remote providers have been migrated into Snakemake wrappers instead, namely
the NCBI and ENA remote providers, which are now replaced by the
`entrez/efetch <https://snakemake-wrappers.readthedocs.io/en/stable/wrappers/entrez/efetch.html>`_ and
the `ena <https://snakemake-wrappers.readthedocs.io/en/stable/wrappers/ena.html>`_ wrappers.


Command line interface
^^^^^^^^^^^^^^^^^^^^^^
Expand Down
4 changes: 3 additions & 1 deletion snakemake/dag.py
Expand Up @@ -349,11 +349,13 @@ async def store_storage_outputs(self):
logger.info("Storing output in storage.")
async with asyncio.TaskGroup() as tg:
for job in self.needrun_jobs(exclude_finished=False):
for f in job.output:
benchmark = [job.benchmark] if job.benchmark else []
for f in chain(job.output, job.log, benchmark):
if (
f.is_storage
and f
not in self.workflow.storage_settings.unneeded_temp_files
and f.exists_local()
):
tg.create_task(f.store_in_storage())

Expand Down

0 comments on commit 6aabb5d

Please sign in to comment.