Skip to content

Commit

Permalink
Merge pull request #505 from hpc/michael-hennecke/daos-doc-1
Browse files Browse the repository at this point in the history 
daos: update documentation
  • Loading branch information
@Michael-Hennecke
Michael-Hennecke committed Nov 2, 2021
2 parents 95c3e8c + 0ae0733 commit 52788e5
Showing 1 changed file with 147 additions and 80 deletions.
227 changes: 147 additions & 80 deletions DAOS-Support.md
View file
Original file line number Diff line number Diff line change
@@ -1,36 +1,75 @@
# DAOS Support

[DAOS](https://github.com/daos-stack/daos) is supported as a backend storage system in
dcp, dsync, and dcmp. Custom serialization and deserialization for DAOS
containers to and from a POSIX filesystem is provided with daos-serialize and
daos-deserialize. A DAOS container and its associated
metadata can be serialized to an HDF5 file for storage on a POSIX filesystem using daos-serialize. The container data
can be later restored to DAOS via the use of daos-deserialize.

The build instructions for enabling DAOS support can be found here:
[Enable DAOS](https://mpifileutils.readthedocs.io/en/latest/build.html#build-everything-directly).
The following are ways that DAOS can be used to move data both across DAOS as well as POSIX
filesystems in dcp and dsync:

1. DAOS -> POSIX
2. POSIX -> DAOS
3. DAOS -> DAOS

For the DAOS->DAOS case, both the DFS API and DAOS Object API are supported.

For daos-serialize and daos-deserialize, any type of DAOS container is supported.
[DAOS](https://github.com/daos-stack/daos) is supported as a backend
storage system in the mpiFileUtils commands dcp, dsync, and dcmp.
The build instructions for enabling DAOS support in mpiFileUtils
can be found here:
[Enable DAOS](https://mpifileutils.readthedocs.io/en/latest/build.html#build-everything-directly).

## DAOS POSIX Data Movement Examples with dcp
With dcp and dsync, the DAOS backend can be used to move data across
two DAOS containers of the same type, as well as between a DAOS
POSIX container and a (non-DAOS) POSIX filesystem like Lustre or GPFS:

1. DAOS POSIX container -> POSIX FS
2. POSIX FS -> DAOS POSIX container
3. DAOS POSIX container -> DAOS POSIX container
4. DAOS non-POSIX container -> DAOS non-POSIX container

When the source or the destination is a (non-DAOS) POSIX filesystem,
the DAOS File System (DFS) API will be used.
When both the source and the destination are DAOS POSIX containers,
both the DFS API and the DAOS Object API are supported to perform
the copy/sync operations (by default, the DFS API will be used).
When both the source and the destination are DAOS non-POSIX containers
(of the same type),
the DAOS Object API will be used to perform the copy/sync operations.

When DAOS and HDF5 support is enabled, mpiFileUtils also provides the
daos-serialize and daos-deserialize commands.
These commands can be used to save the contents of a DAOS container
and its associated metadata into a POSIX filesystem,
and to restore that DAOS container with its associated metadata
back into a DAOS storage system at a later time.
This functionality is also known as "container parking",
and it is supported for any type of DAOS container.
The typical use case is the "archiving" of a DAOS container to a
lower cost, higher capacity storage system
to free up space in the DAOS storage system for other projects.
This step is performed with the daos-serialze command.
When the DAOS container needs to be accessed again, it can be re-established
in the DAOS storage system by using the daos-deserialize command.
The daos-serialize and daos-deserialize commands are MPI-parallel
applications.
They use HDF5 to write and read the DAOS container data and metadata
as "opaque" HDF5 files, one HDF5 file per MPI rank.
In order to use these commands, HDF5 needs to be installed and
mpiFileUtils needs to be built with both DAOS support and HDF5 support.
See the above build instructions for details.



## Data Movement Examples with dcp, using the DFS API

#### Example One

Show a copy from a regular /tmp source path to a DAOS container.
A DAOS Unified Namespace path is used as the destination, which allows you to lookup
the pool and container UUID from the path. This feature can only be used if the
container is created with a path.
Perform a copy from a regular /tmp source path to a DAOS POSIX container.
A DAOS Unified Namespace (DUNS) path is used as the destination,
which enables the tool to look up the pool UUID and container UUID from
the DUNS path. This feature can only be used if the
container has been created with the ` --path=` option
to store the container information as extended attributes in a DUNS path.

Note: The DUNS path should be a directory in a globally accessible filesystem.
All nodes in the MPI job that execute the dcp command will need to
extract the DAOS pool and container information.
If the path given to the `daos cont create` command is in a local
filesystem on the node that runs this daos command
(like a node-local /tmp), that information will only be visible locally
on that node and the DUNS lookup on other nodes will fail.

```shell
$ mpirun -np 3 dcp -v /tmp/$USER/s /tmp/$USER/conts/p1cont1
$ mpirun -np 3 dcp -v /tmp/$USER/s /lustrefs/$USER/conts/p1cont1
[2020-04-23T17:04:15] Items: 6
[2020-04-23T17:04:15]   Directories: 3
[2020-04-23T17:04:15]   Files: 3
Expand All @@ -39,10 +78,12 @@ $ mpirun -np 3 dcp -v /tmp/$USER/s /tmp/$USER/conts/p1cont1

#### Example Two

Show a copy where the pool and container UUID are passed in directly. This option
can be used if the type of container is POSIX, but the container was not created
with a path. The destination is the relative path within the DAOS container, which
in this example is the root of the container.
Perform a copy from a POSIX filesystem to a DAOS POSIX container,
where the destination pool and container UUIDs are passed in directly.
This option can be used if the type of the DAOS container is POSIX,
but the container was not created with a DUNS path.
The destination is a relative path within the DAOS container, which
in this example is the root of the container.

```shell
$ mpirun -np 3 dcp -v /tmp/$USER/s daos://$pool/$cont
Expand All @@ -51,13 +92,17 @@ $ mpirun -np 3 dcp -v /tmp/$USER/s daos://$pool/$cont
[2020-04-23T17:17:51]   Files: 3
[2020-04-23T17:17:51]   Links: 0
```

#### Example Three

Show a copy from one DAOS container to another container that exists in the same
pool. A DAOS Unified Namespace path is used as the source and the destination.
Perform a copy from one DAOS POSIX container to another DAOS POSIX container.
The two containers can reside in the same pool, or in different pools.
In this example, a DAOS Unified Namespace path is used as the source and
as the destination.
The complete source container will be copied to the destination.

```shell
$ mpirun -np 3 dcp -v /tmp/$USER/conts/p1cont1 /tmp/$USER/conts/p1cont2
$ mpirun -np 3 dcp -v /lustrefs/$USER/conts/p1cont1 /lustrefs/$USER/conts/p1cont2
[2020-04-23T17:04:15] Items: 6
[2020-04-23T17:04:15]   Directories: 3
[2020-04-23T17:04:15]   Files: 3
Expand All @@ -66,9 +111,11 @@ $ mpirun -np 3 dcp -v /tmp/$USER/conts/p1cont1 /tmp/$USER/conts/p1cont2

#### Example Four

This example passes in the pool and container UUID directly. The source path
is the relative path within the DAOS container, which in this case is a subset of
the DAOS container.
Perform a copy from one DAOS POSIX container to another DAOS POSIX container.
The two containers can reside in the same pool, or in different pools.
This example passes in the pool and container UUIDs directly. The source path
is a relative path within the DAOS container, which in this case is a subdirectory within
the source DAOS container. Only this subdirectory tree will be copied to the destination.

```shell
$ mpirun -np 3 dcp -v daos://$pool1/$cont1/s/biggerfile daos://$pool2/$cont2
Expand All @@ -80,49 +127,56 @@ $ mpirun -np 3 dcp -v daos://$pool1/$cont1/s/biggerfile daos://$pool2/$cont2

#### Example Five

This example copies data from a DAOS container to /tmp, where a DAOS
Unified Namespace path is used as the source.
This example copies data from a DAOS POSIX container to /tmp,
where a DAOS Unified Namespace path is used as the source.

```shell
$ mpirun -np 3 dcp -v /tmp/$USER/conts/p1cont1 /tmp/$USER/d
$ mpirun -np 3 dcp -v /lustrefs/$USER/conts/p1cont1 /tmp/$USER/d
[2020-04-23T17:17:51] Items: 6
[2020-04-23T17:17:51]   Directories: 3
[2020-04-23T17:17:51]   Files: 3
[2020-04-23T17:17:51]   Links: 0
```
## DAOS non-POSIX Data Movement Examples with dcp

#### Example One

Show the copy from one DAOS container to another, where both of the DAOS
containers are of the POSIX type, but a user would like to copy DAOS
containers at the object level.
## Data Movement Examples with dcp, using the DAOS API

#### Example One

Perform a copy from one DAOS POSIX container to another DAOS POSIX container.
The two containers can reside in the same pool, or in different pools.
in this example, the user requests to copy the DAOS POSIX container
using the DAOS object level API instead of the DFS API.

```shell
$ mpirun -np 3 dcp -v --daos-api=DAOS daos://$pool1/$p1cont1 daos://$pool1/$p1cont2
[2021-01-20T16:16:25] Successfully copied to DAOS Destination Container.
```
#### Example Two

Show the copy from one DAOS container to another, where both of the DAOS
containers are not of the POSIX type. This run does not require passing
in the --daos-api=DAOS flag as it will detect the containers as non-POSIX.
Perform a copy from one DAOS container to another DAOS container,
where both DAOS containers are *not* of type POSIX.
Both containers must be of the same type.
This scenario does not need the `--daos-api=DAOS` option:
It will be automatically detected that the DAOS containers are of a non-POSIX
container type, and thus the DAOS object API has to be used.

```shell
$ mpirun -np 3 dcp -v daos://$pool1/$p1cont1 daos://$pool1/$p1cont2
[2021-01-20T16:16:25] Successfully copied to DAOS Destination Container.
```

## DAOS POSIX Data Movement Examples with dsync

The usage for dsync is similar to the usage for dcp. The source and destination
can be DAOS, POSIX, or UNS paths.
## Data Movement Examples with dsync

The usage for dsync is similar to the usage for dcp.
The source and destination can be DAOS, POSIX, or DUNS paths.

#### Example One

Show the sync from one DAOS container to another, where both of the DAOS
containers are of the POSIX type, and both the source and destination use
a path relative to the root of each container.
Perform the sync from one DAOS container to another, where both of the DAOS
containers are of type POSIX. In this example, both the source and
the destination use a path relative to the root of each container.

```shell
$ mpirun -np 3 dsync -v daos://$pool1/$cont1/source daos://$pool2/$cont2/dest
Expand All @@ -134,8 +188,8 @@ $ mpirun -np 3 dsync -v daos://$pool1/$cont1/source daos://$pool2/$cont2/dest

#### Example Two

Show the sync from one DAOS container to another, where both of the DAOS
containers are not of the POSIX type.
Perform the sync from one DAOS container to another, where both of the DAOS
containers are *not* of type POSIX. Both containers must be of the same type.

```shell
$ mpirun -np 3 dsync -v daos://$pool1/$cont1 daos://$pool1/$cont2
Expand All @@ -148,9 +202,10 @@ $ mpirun -np 3 dsync -v daos://$pool1/$cont1 daos://$pool1/$cont2

#### Example Three

Show the same sync from the previous example, but where the destination
already contains the source data. In this case, the data in the destination
is compared to the source, and nothing is written.
Perform the same sync as in Example Two, but here the destination container
already contains the same data as the source container.
In this case the data in the destination is compared to the source,
and nothing needs to be written.

```shell
$ mpirun -np 3 dsync -v daos://$pool1/$cont1 daos://$pool1/$cont2
Expand All @@ -161,16 +216,17 @@ $ mpirun -np 3 dsync -v daos://$pool1/$cont1 daos://$pool1/$cont2
[2021-03-04T19:15:03] Bytes written : 0.000 B (0 bytes)
```

## DAOS POSIX Data Comparison Examples with dcmp

Similar to dcp and dsync, dcmp supports DAOS, POSIX, and UNS paths,
but dcmp currently only supports POSIX-type DAOS containers.
## Data Comparison Examples with dcmp

Similar to dcp and dsync, dcmp supports DAOS, POSIX, and UNS paths.
dcmp currently only supports DAOS containers of type POSIX.

#### Example One

Show the comparison between two DAOS containers, where both of the DAOS
containers are of the POSIX type, and both containers use a path relative
to the root of each container.
Perform the comparison between two DAOS containers, where both of the DAOS
containers are of type POSIX. In this example, both containers use a
path relative to the root of each container.

```shell
$ mpirun -np 3 dcmp -v daos://$pool1/$cont1/source daos://$pool2/$cont2/dest
Expand All @@ -182,20 +238,25 @@ Number of items that exist in both directories and have the same content: 1 (Src
Number of items that exist in both directories and have different contents: 0 (Src: 0 Dest: 0)
```

## DAOS Serialization and Deserialization Examples
## DAOS Container Serialization and Deserialization Examples

daos-serialize and daos-deserialize can be used on any type of DAOS container.
They are DAOS only tools that require HDF5.

daos-serialize will serialize a DAOS container to an HDF5 file. Depending on
the amount of data, multiple files may be written for each rank specified in the job.
daos-serialize will serialize a DAOS container to one or more HDF5 files.
Depending on the amount of data, multiple files may be written
for each rank specified in the job.

daos-deserialize will deserialize or restore the HDF5 files written into a new
DAOS container, and a pool UUID to deserialize the data to must be specified.
daos-deserialize will deserialize or restore the HDF5 files that have been
previously created by daos-serialize. A pool UUID to deserialize the data to
must be specified, and a container with the original data and metadata
will be created in that DAOS pool.

#### Example One

Show the serialization of a DAOS container
Perform the serialization of a DAOS container into a set of HDF5 files.
In this example, the HDF5 files are stored in the current working directory
within a global parallel filesystem.

```shell
$ mpirun -np 3 daos-serialize -v daos://$pool1/$cont1
Expand All @@ -206,17 +267,21 @@ Serializing Container to 7bf8037d-823f-4fa5-ac2a-c2cae8f81f57_rank2.h5

#### Example Two

Show the serialization of a DAOS container by specifying output directory
Perform the serialization of a DAOS container into a set of HDF5 files.
In this example, a directory in which to store the HDF5 files is specified.

```shell
$ mpirun -np 3 daos-serialize -v -o serialize daos://$pool1/$cont1
Serializing Container to serialize/7bf8037d-823f-4fa5-ac2a-c2cae8f81f57_rank0.h5
Serializing Container to serialize/7bf8037d-823f-4fa5-ac2a-c2cae8f81f57_rank1.h5
Serializing Container to serialize/7bf8037d-823f-4fa5-ac2a-c2cae8f81f57_rank2.h5
$ mpirun -np 3 daos-serialize -v -o serialized-cont daos://$pool1/$cont1
Serializing Container to serialized-cont/7bf8037d-823f-4fa5-ac2a-c2cae8f81f57_rank0.h5
Serializing Container to serialized-cont/7bf8037d-823f-4fa5-ac2a-c2cae8f81f57_rank1.h5
Serializing Container to serialized-cont/7bf8037d-823f-4fa5-ac2a-c2cae8f81f57_rank2.h5
```
#### Example Three

Show the deserialization of an HDF5 file
#### Example Three

Perform the deserialization of a DAOS container that was previously serialized
into a number of HDF5 files. In this example, the HDF5 files are individually
specified on the command line.

```shell
$ mpirun -np 3 daos-deserialize -v --pool $pool1 7bf8037d-823f-4fa5-ac2a-c2cae8f81f57_rank0.h5
Expand All @@ -230,13 +295,15 @@ Successfully created container cbc52064-303e-497d-afaf-fa554c18e08f

#### Example Four

Show the deserialization of HDF5 files from a specified directory
Perform the deserialization of a DAOS container that was previously serialized
into a number of HDF5 files. In this example, the directory in which
he HDF5 files are residing is specified on the command line.

```shell
$ mpirun -np 3 daos-deserialize -v serialize
$ mpirun -np 3 daos-deserialize -v serialized-cont

Successfully created container 8d6a6083-4009-4afe-8364-7caa5ebaa72b
Deserializing filename: serialize/7bf8037d-823f-4fa5-ac2a-c2cae8f81f57_rank0.h5
Deserializing filename: serialize/7bf8037d-823f-4fa5-ac2a-c2cae8f81f57_rank1.h5
Deserializing filename: serialize/7bf8037d-823f-4fa5-ac2a-c2cae8f81f57_rank2.h5
Deserializing filename: serialized-cont/7bf8037d-823f-4fa5-ac2a-c2cae8f81f57_rank0.h5
Deserializing filename: serialized-cont/7bf8037d-823f-4fa5-ac2a-c2cae8f81f57_rank1.h5
Deserializing filename: serialized-cont/7bf8037d-823f-4fa5-ac2a-c2cae8f81f57_rank2.h5
```

0 comments on commit 52788e5

Please sign in to comment.