Home

PhenoMeNal H2020 wiki

The PhenoMeNal infrastructure provides a Cloud Research Environments (CRE) for interoperable and scalable metabolomics analysis. This wiki explains general usage and status of development.

Overview of the PhenoMeNal e-infrastructure

The PhenoMeNal infrastructure is available for deployment on public/private clouds, as well as on local servers. The main access for users is through our Cloud Research Environment (CRE).

• Introduction to PhenoMeNal
• PhenoMeNal and the e-infrastructure landscape
• Introduction to PhenoMeNal architecture and components
• Security Approach and Roadmap
• Vocabulary and definitions in PhenoMeNal

Creating PhenoMeNal Cloud Research Environments

Users can create their own Cloud Research Environment (CRE) through the PhenoMeNal Portal. We support the installation of the CRE on Amazon Web Services (AWS), Google Cloud Engine (GCE) and Open Stack.

Creating PhenoMeNal Cloud Research Environments follows a specific scheme: When the deployment is invoked from the cloud portal, the configuration details are queried through the portal web application. When used from the command line, the requested number of machines, their flavours and other details that can be site specific, are configured through files such as config.tfvars. While the interfaces and options found at the public cloud providers like Google, Amazon and Microsoft are the same for all customers, OpenStack installations (local and commercial ones alike) usually differ between each installations. These differences can range from the installed OpenStack version and available OpenStack modules, but also in terms of the available networks and virtual machine flavours. These settings are fully configurable and the documentation Starting-up-a-PhenoMeNal-VRE-on-OpenStack is showing where to find this information in different OpenStack clusters.

The PhenoMeNal infrastructure deployment is enabled by KubeNow and is applicable for installations on local computers/clusters and public or private cloud environments.

For software provisioning, i.e. the deployment and configuration of software inside the virtual machine infrastructure, the administration automation engine Ansible is used. The software management tasks are defined in so-called playbooks. In PhenoMeNal, they are split into 1) the generic playbooks for setting up the operating system and Kubernetes cluster, and 2) specific playbooks defining the running PhenoMeNal services in the kubernetes cluster. The latter are rather simple, since a Helm package manager is used for the actual definition. Helm is a commonly used package manager for Kubernetes application deployments. It allows administrators and users to manage parameterised deployments, upgrades and keeping track of the deployment history (which versions were deployed, with which variables, rollback after an update if needed, etc).

We have written Helm charts for the application deployment of Galaxy, which have been contributed back to the general Galaxy community. The documentation for developers who want to extend the functionality are in the README.md and values.yaml files. Other helm charts are for Jupyter and the Portal. We also use Helm for the continuous delivery of testing inside Kubernetes clusters for tools with larger data.

See the links below for further general information:

• Understanding the PhenoMeNal Gateway interface
• Starting a PhenoMeNal CRE via PhenoMeNal Portal
• Starting a PhenoMeNal CRE on a public or private cloud provider

Workflow Tutorials

While CREs are usually deployed by more technical users, end-users such as clinicians or biochemists can simply use the Galaxy workflow management system therein. We have dedicated significant resources to build reproducible workflows, accompanied by detailed documentation and tutorials so that end-users can directly start using them. More tutorials and workflows will follow in the future.

Examples of Galaxy Tools for metadata tracking:

• Study Design ISA Creation
• ISA Creation, Validation, Visualization and Deposition/Preregistration to EMBL-EBI Metabolights

Examples of reproducible computational workflows:

• Fluxomics
• MS MetFrag XCMS
• NMR1d
• Univariate and Multivariate Statistics
• Eco-Metabolomics workflow

Developer Resources

This section contains technical documentation for developers, both from within the project and externally, interested in learning more about the PhenoMeNal architecture, frameworks, and evaluate capabilities offered.

Public release information

• Schedule
• Galaxy release procedure
• Tool spreadsheet

Guidelines for tool developers

• How to make your software tool available through PhenoMeNal, including:
  • Dockerfile creation
  • Addition of containers to Jenkins
  • Container release procedures
• How to update your software tool in PhenoMeNal
• Guideline for creating a container GitHub Respository README.md
• Using Docker

Guidelines for workflow developers

• Installation of Local PhenoMeNal Workflows using Galaxy on MiniKube
• Uninstallation of Local PhenoMeNal Workflows using Galaxy on MiniKube

Guidelines for testing

• Continuous Integration of software tools in PhenoMeNal
• Testing container with real datasets inside Kubernetes
• Notes on container streamlining, testing & statistics (Merge-Google-doc-here)
• Using Galaxy with Kubernetes to run containerised tools

Guidelines for portal developers

• PhenoMeNal Portal Infrastructure

Technical documents and tutorials

• de.NBI Summer school cloud computing 2017
• Logging and Monitoring Tools
• Data and Compute Federation in PhenoMeNal

If you aim to contribute to this wiki, please read the wiki contribution guidelines before making changes.