Skip to content
An Open Computational Genomics Analysis platform for big data processing and analysis in genomics
Branch: develop
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github/ISSUE_TEMPLATE Update issue templates May 24, 2019
.vscode Container pool workaround. Refactor mount.py for calling from python Jan 3, 2019
opencga-analysis test: fix custom analysis test Jun 12, 2019
opencga-app Merge branch 'develop' of https://github.com/opencb/opencga into develop Jun 17, 2019
opencga-catalog catalog: use entity_name from PanelApp as region id when present Jun 26, 2019
opencga-client app: fix minor cli issues Jun 12, 2019
opencga-core core: Avoid NPE on missing configuration sections. Jun 4, 2019
opencga-server clinical: refactor clinical analysis for implementing RoleInCancerMan… Jun 10, 2019
opencga-storage storage: Improve approximate count. Get real numVariantsFromPrimary Jun 3, 2019
opencga-test Test: change return code for partial DELETE and group remove paramete… May 14, 2019
opencga-web catalog: use entity_name from PanelApp as region id when present Jun 26, 2019
.dockerignore Update init script to configure cellbase to use local mongo. Run conf… Feb 22, 2019
.gitignore Merge pull request #1225 from mgviz/pyOpenCGA-mgviz May 13, 2019
.gitmodules web: basic structure of the web application Apr 8, 2016
Jenkinsfile Jenkinsfile: add ARM validation and test back Jun 10, 2019
LICENSE Apache License v2 copied into the project, this and previous commit c… Apr 26, 2015
README.md Fix typo in readme May 23, 2019
checkstyle.xml catalog: improve internal methods, closes #1245 Apr 4, 2019
pom.xml pom: upgrade biodata version Jun 17, 2019

README.md

Overview

OpenCGA is an open-source project that aims to provide a Big Data storage engine and analysis framework for genomic scale data analysis of hundreds of terabytes or even petabytes. OpenCGA provides a scalable and high-performance Storage Engine framework to index biological data such as BAM or VCF files using different NoSQL databases, currently only MongoDB has been fully developed. A data analytics and genomic Analysis layer interface has been implemented over this big data storage index. A metadata Catalog has also been developed to provide authentification and ACLs and to keep track all of files and sample annotation. All these can be queried through a comprehensive RESTful web services API or using the command line interface.

OpenCGA constitutes the big data analysis component of OpenCB initiative. It is used by other projects such as EMBL-EBI EVA, Babelomics or BierApp.

Documentation

You can find OpenCGA documentation and tutorials at: https://github.com/opencb/opencga/wiki.

Issues Tracking

You can report bugs or request new features at GitHub issue tracking.

Release Notes and Roadmap

Releases notes are available at GitHub releases.

Roadmap is available at GitHub milestones. You can report bugs or request new features at GitHub issue tracking.

Versioning

OpenCGA is versioned following the rules from Semantic versioning.

Maintainers

We recommend to contact OpenCGA developers by writing to OpenCB mailing list opencb@googlegroups.com. Current main developers and maintainers are:

Former Contributors
Contributing

OpenCGA is an open-source and collaborative project. We appreciate any help and feedback from users, you can contribute in many different ways such as simple bug reporting and feature request. Dependending on your skills you are more than welcome to develop client tools, new features or even fixing bugs.

How to build

OpenCGA is mainly developed in Java and it uses Apache Maven as building tool. OpenCGA requires Java 8, in particular JDK 1.8.0_60+, and other OpenCB dependencies that can be found in Maven Central Repository.

Stable releases are merged and tagged at master branch, you are encourage to use latest stable release for production. Current active development is carried out at develop branch and need Java 8, in particular JDK 1.8.0_60+, only compilation is guaranteed and bugs are expected, use this branch for development or for testing new functionalities. Dependencies of master branch are ensured to be deployed at Maven Central Repository, but dependencies for develop branch may require users to download and install the following git repositories from OpenCB:

Cloning

OpenCGA is an open-source and free project, you can download default develop branch by executing:

imedina@ivory:~$ git clone https://github.com/opencb/opencga.git
Cloning into 'opencga'...
remote: Counting objects: 20267, done.
remote: Compressing objects: 100% (219/219), done.
remote: Total 20267 (delta 105), reused 229 (delta 35)
Receiving objects: 100% (20267/20267), 7.23 MiB | 944.00 KiB/s, done.
Resolving deltas: 100% (6363/6363), done.

Latest stable release at master branch can be downloaded executing:

imedina@ivory:~$ git clone -b master https://github.com/opencb/opencga.git
Cloning into 'opencga'...
remote: Counting objects: 20267, done.
remote: Compressing objects: 100% (219/219), done.
remote: Total 20267 (delta 105), reused 229 (delta 35)
Receiving objects: 100% (20267/20267), 7.23 MiB | 812.00 KiB/s, done.
Resolving deltas: 100% (6363/6363), done.

Build

You can build OpenCGA by executing the following command from the root of the cloned repository:

$ mvn clean install -DskipTests

For changing particular settings during buildings you can create a profile in ~/.m2/settings.xml in the <profiles> section using this template:

    <?xml version="1.0" encoding="UTF-8"?>
    <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
        <profiles>
            <profile>
                <id>custom-config</id>
                <activation>
                    <activeByDefault>true</activeByDefault>
                </activation>
                <properties>
                    <opencga.war.name>opencga-${opencga.version}</opencga.war.name>

                    <!-- General -->
                    <OPENCGA.INSTALLATION.DIR>/opt/opencga</OPENCGA.INSTALLATION.DIR>
                    <OPENCGA.USER.WORKSPACE>file:///opt/opencga/sessions/</OPENCGA.USER.WORKSPACE>
                    <OPENCGA.JOBS.DIR>${OPENCGA.USER.WORKSPACE}/jobs/</OPENCGA.JOBS.DIR>
                    <OPENCGA.DB.PREFIX>opencga</OPENCGA.DB.PREFIX>
                    <OPENCGA.EXECUTION.MODE>LOCAL</OPENCGA.EXECUTION.MODE>

                    <!-- Client -->
                    <OPENCGA.CLIENT.REST.HOST>http://localhost:8080/${opencga.war.name}</OPENCGA.CLIENT.REST.HOST>
                    <OPENCGA.CLIENT.GRPC.HOST>http://localhost:9091</OPENCGA.CLIENT.GRPC.HOST>
                    <OPENCGA.CLIENT.ORGANISM.SCIENTIFIC_NAME>Homo sapiens</OPENCGA.CLIENT.ORGANISM.SCIENTIFIC_NAME>
                    <OPENCGA.CLIENT.ORGANISM.COMMON_NAME>human</OPENCGA.CLIENT.ORGANISM.COMMON_NAME>
                    <OPENCGA.CLIENT.ORGANISM.TAXONOMY_CODE>9606</OPENCGA.CLIENT.ORGANISM.TAXONOMY_CODE>
                    <OPENCGA.CLIENT.ORGANISM.ASSEMBLY></OPENCGA.CLIENT.ORGANISM.ASSEMBLY>

                    <OPENCGA.SERVER.REST.PORT>9090</OPENCGA.SERVER.REST.PORT>
                    <OPENCGA.SERVER.GRPC.PORT>9091</OPENCGA.SERVER.GRPC.PORT>
                    <OPENCGA.MONITOR.PORT>9092</OPENCGA.MONITOR.PORT>

                    <!-- Catalog -->
                    <OPENCGA.CATALOG.DB.HOSTS>localhost:27017</OPENCGA.CATALOG.DB.HOSTS>
                    <OPENCGA.CATALOG.DB.USER></OPENCGA.CATALOG.DB.USER>
                    <OPENCGA.CATALOG.DB.PASSWORD></OPENCGA.CATALOG.DB.PASSWORD>
                    <OPENCGA.CATALOG.DB.AUTHENTICATION_DATABASE></OPENCGA.CATALOG.DB.AUTHENTICATION_DATABASE>
                    <OPENCGA.CATALOG.DB.CONNECTIONS_PER_HOST>20</OPENCGA.CATALOG.DB.CONNECTIONS_PER_HOST>

                    <!-- Storage -->
                    <OPENCGA.STORAGE.DEFAULT_ENGINE>mongodb</OPENCGA.STORAGE.DEFAULT_ENGINE>
                    <OPENCGA.STORAGE.CACHE.HOST>localhost:6379</OPENCGA.STORAGE.CACHE.HOST>
                    <OPENCGA.STORAGE.SEARCH.HOST>http://localhost:8983/solr/</OPENCGA.STORAGE.SEARCH.HOST>
                    <OPENCGA.STORAGE.STUDY_METADATA_MANAGER></OPENCGA.STORAGE.STUDY_METADATA_MANAGER>

                    <!-- Storage Variants general -->
                    <OPENCGA.STORAGE.VARIANT.DB.HOSTS>localhost:27017</OPENCGA.STORAGE.VARIANT.DB.HOSTS>
                    <OPENCGA.STORAGE.VARIANT.DB.USER></OPENCGA.STORAGE.VARIANT.DB.USER>
                    <OPENCGA.STORAGE.VARIANT.DB.PASSWORD></OPENCGA.STORAGE.VARIANT.DB.PASSWORD>

                    <!-- Storage Alignments general -->
                    <OPENCGA.STORAGE.ALIGNMENT.DB.HOSTS>localhost:27017</OPENCGA.STORAGE.ALIGNMENT.DB.HOSTS>
                    <OPENCGA.STORAGE.ALIGNMENT.DB.USER></OPENCGA.STORAGE.ALIGNMENT.DB.USER>
                    <OPENCGA.STORAGE.ALIGNMENT.DB.PASSWORD></OPENCGA.STORAGE.ALIGNMENT.DB.PASSWORD>

                    <!-- Storage-mongodb -->
                    <OPENCGA.STORAGE.MONGODB.VARIANT.DB.AUTHENTICATION_DATABASE></OPENCGA.STORAGE.MONGODB.VARIANT.DB.AUTHENTICATION_DATABASE>
                    <OPENCGA.STORAGE.MONGODB.VARIANT.DB.CONNECTIONS_PER_HOST>20</OPENCGA.STORAGE.MONGODB.VARIANT.DB.CONNECTIONS_PER_HOST>

                    <!-- Storage-hadoop -->
                    <!--If empty, will use the ZOOKEEPER_QUORUM read from the hbase configuration files-->
                    <OPENCGA.STORAGE.HADOOP.VARIANT.DB.HOSTS></OPENCGA.STORAGE.HADOOP.VARIANT.DB.HOSTS>
                    <OPENCGA.STORAGE.HADOOP.VARIANT.DB.USER></OPENCGA.STORAGE.HADOOP.VARIANT.DB.USER>
                    <OPENCGA.STORAGE.HADOOP.VARIANT.DB.PASSWORD></OPENCGA.STORAGE.HADOOP.VARIANT.DB.PASSWORD>
                    <OPENCGA.STORAGE.HADOOP.VARIANT.HBASE.NAMESPACE></OPENCGA.STORAGE.HADOOP.VARIANT.HBASE.NAMESPACE>

                    <!-- Email server -->
                    <OPENCGA.MAIL.HOST></OPENCGA.MAIL.HOST>
                    <OPENCGA.MAIL.PORT></OPENCGA.MAIL.PORT>
                    <OPENCGA.MAIL.USER></OPENCGA.MAIL.USER>
                    <OPENCGA.MAIL.PASSWORD></OPENCGA.MAIL.PASSWORD>

                    <!-- cellbase -->
                    <OPENCGA.CELLBASE.VERSION>v4</OPENCGA.CELLBASE.VERSION>
                    <OPENCGA.CELLBASE.REST.HOST>http://bioinfodev.hpc.cam.ac.uk/cellbase-4.5.0-beta1.1/</OPENCGA.CELLBASE.REST.HOST>
                    <OPENCGA.CELLBASE.DB.HOST>localhost:27017</OPENCGA.CELLBASE.DB.HOST>
                    <OPENCGA.CELLBASE.DB.USER></OPENCGA.CELLBASE.DB.USER>
                    <OPENCGA.CELLBASE.DB.PASSWORD></OPENCGA.CELLBASE.DB.PASSWORD>
                    <OPENCGA.CELLBASE.DB.AUTHENTICATION_DATABASE></OPENCGA.CELLBASE.DB.AUTHENTICATION_DATABASE>
                    <OPENCGA.CELLBASE.DB.READ_PREFERENCE>secondaryPreferred</OPENCGA.CELLBASE.DB.READ_PREFERENCE>
                </properties>
            </profile>
        </profiles>
    </settings>

See the description of each property in https://github.com/opencb/opencga/wiki/OpenCGA-installation.

Remember that develop branch dependencies are not ensured to be deployed at Maven Central, you may need to clone and install develop branches from OpenCB biodata, datastore, cellbase and hpg-bigdata repositories. After this you should have this file structure in opencga/build:

opencga/build/
├── bin/
├── conf/
├── examples/
├── libs/
├── LICENSE
├── opencga.war
├── README.md
└── tools/

You can copy the content of the build folder into the installation directory such as /opt/opencga.

Testing

You can run the unit tests using Maven or your favorite IDE. Just notice that some tests may require of certain database back-ends such as MongoDB or Apache HBase and may fail if they are not available.

Command Line Interface (CLI)

If the build process has gone well you should get an integrated help by executing:

./bin/opencga.sh --help

You can find more detailed documentation and tutorials at: https://github.com/opencb/opencga/wiki.

Other Dependencies

We try to improve the admin experience by making the installation and build as simple as possible. Unfortunately, for some OpenCGA components and functionalities other dependencies are required.

Loading data

At this moment the only fully developed storage engine plugin is MongoDB. MongoDB is free and open-source and can be downloaded from here. Currently Apache HBase plugin is under heavy development and will be ready soon.

AES encryption

For AES encryption please download UnlimitedJCEPolicyJDK7.zip from https://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html . Then unzip the file into $JAVA_HOME/jre/lib/security

You can’t perform that action at this time.