Ola Spjuth edited this page Jul 9, 2018 · 140 revisions
Clone this wiki locally

PhenoMeNal H2020 wiki

The PhenoMeNal infrastructure provides a Cloud Research Environments (CRE) for interoperable and scalable metabolomics analysis. This wiki explains general usage and status of development.

Overview of the PhenoMeNal e-infrastructure

The PhenoMeNal infrastructure is available for deployment on public/private clouds, as well as on local servers. The main access for users will be through our Cloud Research Environment (CRE).

Creating PhenoMeNal Cloud Research Environments

Users can create their own Cloud Research Environment (CRE) through the PhenoMeNal Portal. We support the installation of the CRE on Amazon Web Services (AWS), Google Cloud Engine (GCE) and Open Stack (in beta testing).

Creating PhenoMeNal Cloud Research Environments follows a specific scheme: When the deployment is invoked from the cloud portal, the configuration details are queried through the portal web application. When used from the command line, the requested number of machines, their flavours and other details that can be site specific, are configured through files such as config.tfvars. While the interfaces and options found at the public cloud providers like Google, Amazon and Microsoft are the same for all customers, OpenStack installations (local and commercial ones alike) usually differ between each installations. These differences can range from the installed OpenStack version and available OpenStack modules, but also in terms of the available networks and virtual machine flavours. These settings are fully configurable and the documentation Starting-up-a-PhenoMeNal-VRE-on-OpenStack is showing where to find this information in different OpenStack clusters.

The PhenoMeNal infrastructure deployment is enabled by KubeNow and is applicable for installations on local computers/clusters and public or private cloud environments.

For software provisioning, i.e. the deployment and configuration of software inside the virtual machine infrastructure, the administration automation engine Ansible is used. The software management tasks are defined in so-called playbooks. In PhenoMeNal, they are split into 1) the generic playbooks for setting up the operating system and Kubernetes cluster, and 2) specific playbooks defining the running PhenoMeNal services in the kubernetes cluster. The latter are rather simple, since a Helm package manager is used for the actual definition. Helm is a commonly used package manager for Kubernetes application deployments. It allows administrators and users to manage parameterised deployments, upgrades and keeping track of the deployment history (which versions were deployed, with which variables, rollback after an update if needed, etc).

We have written Helm charts for the application deployment of Galaxy, which have been contributed back to the general Galaxy community. The documentation for developers who want to extend the functionality are in the README.md and values.yaml files. Other helm charts are for Jupyter and the Portal. We also use Helm for the continuous delivery of testing inside Kubernetes clusters for tools with larger data.

See the links below for further general information:

Workflow Tutorials

While CREs are usually deployed by more technical users, end-users such as clinicians or biochemists can simply use the Galaxy workflow management system therein. We have dedicated significant resources to build reproducible workflows, accompanied by detailed documentation and tutorials so that end-users can directly start using them. More tutorials and workflows will follow in the future.

Examples of Galaxy Tools for metadata tracking:

Examples of reproducible computational workflows:

Developer Resources

This section contains technical documentation for developers, both from within the project and externally, interested in learning more about the PhenoMeNal architecture, frameworks, and evaluate capabilities offered.

Public release information

Guidelines for tool developers

Guidelines for workflow developers

Guidelines for testing

Guidelines for portal developers

Technical documents and tutorials

If you aim to contribute to this wiki, please read the wiki contribution guidelines before making changes.