The Yorc DDI plugin implements a Yorc (Ystia orchestrator) plugin as described in Yorc documentation, allowing the orchestrator to use LEXIS DDI (Distributed Data Infrastructure) API to manage asynchronous data transfers requests.
This code repository is a result / contains results of the LEXIS project. The project has received funding from the European Union’s Horizon 2020 Research and Innovation programme (2014-2020) under grant agreement No. 825532.
This plugin provides the following TOSCA components defined in the TOSCA file a4c/ddi-types-a4c.yaml that can be uploaded in Alien4Cloud catalog of TOSCA components:
Job executing a transfer of dataset from DDI to Cloud staging area
Job executing a transfer of dataset from Cloud staging area to DDI
Job deleting a dataset from Cloud staging area
Job executing a transfer of dataset from DDI to HPC in a directory for a given task in the job
Job executing a transfer of dataset from DDI to HPC in a directory for a given task in the job, the dataset path being computed at runtime
Job executing a transfer of data from a HPC job directory to DDI
Job executing a transfer of dataset from Cloud staging area to HPC in a directory for a given task in the job
Job executing a transfer of dataset from a HPC job directory to Cloud Staging area
Job monitoring a HEApPE job and transferring new files produced by this HEAppE job to DDI, until this HEAppE job ends
Job monitoring a HEApPE job and transferring new files produced by this HEAppE job to DDI until this HEAppE job ends, and grouping these files in datasets according to a pattern.
Job waiting for a dataset to appear in DDI, and optionally waiting for files of a given pattern to appear in this dataset
Job executing a transfer of files in dataset from DDI to Cloud staging area, the dataset being provided at runtime by an associated component
Job executing a transfer of files in a dataset from DDI to HPC in a directory for a given task in the job
Job executing a request to get a DDI dataset info (size, number of files, number of small files of size <= 32MB)
Job executing requests to replicated a dataset to different sites
Component whose start operation provide info on a directory in a compute instance (size, number of files, number of small files of size <= 32MB)
Component whose start operation provide info on files produced by a HEAppE job (size, number of files, number of small files of size <= 32MB)
SSHFS mount a dataset in the Cloud staging area on a compute instance directory.
You need first to have a working Go environment. Then to build, execute the following instructions:
mkdir -p $GOPATH/src/github.com/lexis-project
cd $GOPATH/src/github.com/lexis-project
git clone https://github.com/lexis-project/yorc-ddi-plugin
cd yorc-ddi-plugin
make
The plugin is then available at bin/ddi-plugin
.
This plugin is licensed under the Apache 2.0 License.