Welcome to the SemWES/CloudiFacturing service-development resources!
This repository is meant to be a center for documentation relevant for everyone who develops services and workflows on the SemWES platform stack in the CloudiFacturing project. Here, you will find everything from high-level descriptions of the concepts behind the platform to step-by-step tutorials for advanced topics in service development.
- Documentation version
- Contributing
- Platform overview
- Service implementation: concepts, examples, tutorials
- Workflow creation, execution, and monitoring
- Stream Data Integration
- emGORA Integration
- Reference documentation of platform services
Current documentation version: 3.20.0
See the Changelog for versioning details.
Everyone is welcome to contribute to this documentation center. Typos, clarifications, interesting code examples, or also just questions – anything is valuable. To contribute, choose one of the following options:
- Fork the repository, add your changes and create a pull request.
- Create an issue in the GitHub issue tracker.
- Write an email with your ideas to Robert (discouraged, rather use one of the options above).
New to the SemWES platform? Read all about its concepts and background here.
-
Getting access: Need access to the SemWES platform in CloudiFacturing? Look here.
-
Demos: If you want to get your hands dirty immediately and try out some demo workflows on the platform, look here.
-
Users, projects, and authentication: Gives a short overview of how user authentication and data-access restriction works on the SemWES platform.
-
Workflows and services in the SemWES cloud – an overview: Gives a compact overview over the concepts behind SemWES and what services and workflows are and how they are executed. Also explains the nomenclature used in this repository.
-
The SemWES service types: Description and requirements of synchronous services, asynchronous services, and applications. Read this if you're wondering what kind of service you need to develop for a certain use case.
-
Accessing cloud storage: Generic Storage Services (GSS): On the SemWES platform, different cloud storages can be accessed in a simple, unified way. Learn about the basic concepts of the SemWES Generic Storage Services and the available storage solutions in CloudiFacturing.
All workflows in the SemWES platform are basically a series of calls to individual web services. This section provides information on how to develop such web services in a way that they can be registered in the SemWES platform.
All documentation here deals with things done "in code" (as opposed to via the graphical tools provided on the portal).
- Available parameters: The workflow manager offers a set of "global" parameters which are available to every service. Here, we take a closer look at these parameters and their main use cases.
- Check out the SemWES service types for a high-level description of synchronous services and their required interface.
- Code example (Python): synchronous service: A very simple synchronous calculator service implemented in Python
- Code example (Java): synchronous-service skeleton: Bare-bone skeleton of a synchronous service
- Tutorial: Deploy and modify synchronous calculator webservice: This tutorial teaches you how to deploy and modify one of the code examples, namely the synchronous Calculator webservice implemented in Python. Good as a starting point in the service development.
- Check out the SemWES service types for a high-level description of asynchronous services and their required interface.
- Code example (Python): Waiter: An asynchronous service which does nothing but waiting
- Tutorial: Create a simple asynchronous service: This tutorial guides you through the deployment steps of a simple asynchronous service. Starting from a simply Python script representing a long-running computation, we will wrap an asynchronous web service around it which can be deployed in the SemWES infrastructure stack.
- Check out the SemWES service types for a high-level description of applications and their required interface.
- Code example (Python): Dialog: Simplest possible example application containing a one-button dialog.
- Code example (Python): Debugger: Debug application for workflows; pauses a workflow and displays parameter contents.
-
Service-deployment concept: Learn about the SemWES deployment strategy here.
-
Service-deployment manual: Describes how services can be deployed in SemWES
- Testing SOAP services: Explains how deployed SOAP services can be tested without having to go through the workflow manager.
- If you haven't done so already, it is recommended to read the general principles of accessing cloud storage on the SemWES platform.
- High-level file access using GSS libraries: While it is perfectly possible to directly interact with the GSS API, it is highly recommended to use the provided GSS client libraries for file access in SemWES. This article gives an overview over existing libraries and their usage.
-
HPC access through the SemWES platform: The SemWES platform abstracts access of specific HPC resources with a generic API, making it possible to run computations on different HPC resources without any change to the computation code. This article explains the concepts and technical background of this solution.
-
Packaging software in Singularity images: All software that should be run on the HPC resources accessible through the SemWES platform must be wrapped into Singularity images which are then executed as isolated containers on an HPC cluster. Learn how to create, upload, and register such images in this article.
-
Communicating with a running HPC job: Sometimes, communication with a running HPC job is important, for example to be able to control or abort a simulation if required. This article explains how to set up a Singularity image for this kind of communication.
-
Debugging HPC applications: Debugging HPC applications in SemWES can be difficult due to many layers of abstraction between the running application and the user. This article gives some hints on debugging and loggin.
-
Singularity and MPI applications: Learn how to prepare your Singularity image for parallel execution using MPI.
-
GPU support for Singularity images: Learn how to prepare your Singularity image for access of underlying GPU cores.
-
Error handling in SOAP services: Explains how to gracefully handle errors inside SOAP services such that the outside world can process them.
-
Using authentication services inside a service: When a service is executed inside a workflow, the portal takes care that proper user authentication. If necessary, however, the authentication manager can also be used inside services.
Once all required web services are developed, they need to be registered and hooked up to make a workflow. This section deals with all things done "graphically" via the tools provided on the portal.
-
Don't forget to have a look at the demo workflows to learn from some pre-defined workflow examples.
-
Tutorial: Overview over the portal GUI: In this tutorial you will get to know the portal GUI, start a workflow and inspect its results.
-
Tutorial: Basic workflow editing: In this tutorial you will load, modify, and save an existing workflow using the graphical workflow editor.
-
Tutorial: Registration of new services: Any newly created service needs to be registered properly to be usable in the SemWES platform. Learn about all details and caveats of service registration here.
-
Upgrading services: Sometimes, an already deployed and registered service needs to be upgraded, possibly with changes to the input and output parameters. Read here what to keep in mind when performing such upgrades.
-
Overview over the generic HPC service: Introduces the generic HPC service and showcases how it can be used to execute Singularity images on the available HPC resources.
-
Converting from GSS URIs to file paths and back: Files and folders are handled using GSS URIs within the SemWES platform, but on an HPC cluster, absolute paths are required. Learn how to convert between the two here.
-
Pre- and post-processor services: Explains how one can interface with the generic HPC service by writing pre- and post-processor services.
-
Launching HPC jobs in the background: Explains how HPC jobs can be launched in the background, such that other parts of the workflow can run in parallel.
-
File selection using the FileChooser service: The FileChooser service is one of two ways to easily let the user upload and select files for a workflow. Learn how to integrate the file chooser into your workflows.
-
Automatic creation of graphical interfaces for user input: Want an HTML form for user input at the start of a workflow? Then use this tool which automatically creates one for you with minimal input on your side.
-
DFKI’s utility services: Want to show some HTML during a workflow? Need a user decision somewhere inside the workflow? DFKI's utility suite offers ready-made services just for that.
- Parameter debugger: This simple application offers the option to pause a workflow and display any parameters that are currently in use. Great for debugging failing workflows or services. Offered as a code example with complete source code.
-
Using workflows inside workflows: One of the great strengths of the SemWES workflow concepts is that one can use complete workflows as services inside another workflow. Learn about this workflow nesting here.
-
Branching and looping: Read how you can implement basic branches and loops using only the workflow editor.
- Stream data handling: Due to its flexible webservice-based workflow structure, SemWES does natively support the integration of stream data into SemWES workflows. This section features a python based prototype and explanations, and links to demo services and workflows.
- emGORA integration manual: This section will address how to integrate SemWES workflows with the emgora marketplace.
If you need information on a specific method of one of the platform services, have a look at our API references: