Skip to content

Commit

Permalink
Add Fusion for Google storage docs
Browse files Browse the repository at this point in the history
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
  • Loading branch information
pditommaso committed Feb 22, 2023
1 parent 862ef02 commit 34fcfe0
Show file tree
Hide file tree
Showing 2 changed files with 29 additions and 7 deletions.
5 changes: 3 additions & 2 deletions docs/fusion.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Fusion is a distributed virtual file system for cloud-native data pipelines, opt

It bridges the gap between cloud-native storage and data analysis workflows by implementing a thin client
that allows any existing application to access object storage using the standard POSIX interface, thus simplifying
and speeding up most operations. Currently, it supports AWS S3.
and speeding up most operations. Currently, it supports AWS S3 and Google Storage.

Getting started
===============
Expand All @@ -22,7 +22,8 @@ Requirements
Fusion file system is designed to work with containerised workloads, therefore it requires the use of a container
engine such as Docker or a container native platform for the execution of your pipeline, e.g. AWS Batch or Kubernetes.

It also requires the use of :ref:`Wave containers<wave-page>` and Nextflow version ``22.10.0`` or later.
It also requires the use of :ref:`Wave containers<wave-page>` and Nextflow version ``22.10.0`` or later. The support
for Google storage requires Nextflow ``23.02.0-edge`` or later.

AWS S3 configuration
--------------------
Expand Down
31 changes: 26 additions & 5 deletions docs/google.rst
Original file line number Diff line number Diff line change
Expand Up @@ -139,15 +139,12 @@ Process definition
Processes can be defined as usual and by default the ``cpus`` and ``memory`` directives are used to find the cheapest machine
type available at current location that fits the requested resources. If ``memory`` is not specified, 1GB of memory is allocated per cpu.

The process ``machineType`` directive can be a list of patterns separated by comma. The pattern can contain a `*` to match
any number of characters and `?` to match any single character. Examples of valid patterns: `c2-*`, `m?-standard*`, `n*`.
As of version ``23.02.0-edge``, the process ``machineType`` directive can be a list of patterns separated by comma. The pattern can contain a ``*`` to match
any number of characters and ``?`` to match any single character. Examples of valid patterns: ``c2-*``, ```m?-standard*``, ```n*``.

Alternatively it can also be used to define a specific predefined Google Compute Platform `machine type <https://cloud.google.com/compute/docs/machine-types>`_
or a custom machine type.

When Fusion is enabled, by default, only machine types that allow to attach local SSD disks will be used. If you specify your own
machine type or machine series they should allow to attach local SSD disks, otherwise the job scheduling will fail.

Examples::

process automatic_resources_task {
Expand Down Expand Up @@ -215,6 +212,30 @@ if the virtual machine was terminated preemptively::
maxRetries = 5
}

Fusion file system
------------------

As of version ``23.02.0-edge``, Google Batch executor supports the use of :ref:`fusion-page`.

Fusion allows the use of Google Storage as a virtual distributed file system, optimising the data transfer
and speeding up most job I/O operations.

To enable the use of Fusion file system in your pipeline, add the following snippet in your Nextflow configuration file::

fusion.enabled = true
wave.enabled = true
process.scratch = false
tower.accessToken = '<YOUR ACCESS TOKEN>'

The `Tower <https://cloud.tower.nf>`_ access token is optional, but it enables higher API rate limits for the
:ref:`wave-page` service required by Fusion.

.. tip::
When Fusion is enabled, by default, only machine types that allow to attach local SSD disks will be used. If you specify your own
machine type or machine series they should allow to attach local SSD disks, otherwise the job scheduling will fail.



Supported directives
--------------------

Expand Down

0 comments on commit 34fcfe0

Please sign in to comment.