Generate and manage your own genomic data portal.
- Getting Started
- What is a "Data Portal"?
- Development Details
- Development Setup
Setting up the project, and prepare things to make changes
# 1. clone the repository git clone firstname.lastname@example.org:overture-stack/arranger.git # 2. enter the project's folder cd arranger # 3. install the dependencies npm i # 4. install the module's own dependencies npm run bootstrap
Now you should be able to start the following processes from the project's root folder:
# watch all modules and rebuild them when you make changes npm run watch # test all modules at once npm test # run the server (on port 5050) npm run server # serve the component dashboard (on port 6060) npm run dashboard # serve the component portal (on port 7070) npm run portal # run storybook (on port 8080) npm run storybook
- Dockerized Setup
A bit more friendly "quickstart", if you just want to get things started
# Start all services at once, using some default settings. # This runs the following services: Elasticsearch, kibana, arranger-server, and arranger-ui make start # ^^^ which runs the following command behind the scenes: # ES_USERNAME=elastic ES_PASSWORD=myelasticpassword docker-compose -f docker-compose.yml up -d -build # Note: these ES_* values may be customized when running your own Arranger instance --- # Afterwards, in another bash process, you may seed an example file_centric index make init-es # ^^^ which runs the following command behind the scenes: # ./docker/elasticsearch/load-es-data.sh ./docker/elasticsearch elastic myelasticpassword # That SH script may give you ideas on how to automate uploading indexes to your instance. --- # Bonus: ----------------------------- # # See other preprogrammed make targets make help # e.g. utilities to list the indexes, or clear the Elasticsearch; list the running docker containers, etc.
- International Cancer Genome Consortium (ICGC) Data Portal
- Genomic Data Commons (GDC) Data Portal (joint effort with University of Chicago)
Although they are not identical in architecture, available data or overall purpose, there is tremendous amount of overlap in how they function and how users interact with them, despite being implemented differently. It's no coincidence. The GDC Data Portal was directly influenced by the ICGC Data Portal.
With new projects ahead of us, there is an opportunity to create a framework designed to act as a core library for any given data portal, similar to what Elastic's Kibana accomplishes; but based on the features of our existing portals, and the expectation of continuous improvement and domain specific customization.
There are many potential benefits:
- Reduce duplicate code
- Ability to fix bugs and add features to many projects at once
- Pool developer resources
- Increase cross-team communication
- Encourage open source contribution
What Is A "Data Portal"?
cli tool for bootstrapping new projects
Provide all necessary modules to implement searching functionality
- Dynamic GraphQL schema generation
- API Server (GraphQL endpoint)
- Query / Aggregation building middleware
- Response middleware (ie. removing null aggregations)
- UI Components
- Simple view
- Advanced View
- Results Table
- SQON Display
- Provide editor interface to expose common transformations (similar to the Babel or bodybuilder REPLs)
- Elasticsearch Mappings -> GraphQL Schema
- GraphQL Query -> Elasticsearch Queries
- Kibana Plugin
- Hosted Data Portal generating service
npm run tag <version>
- Publishing process will be run by Jenkins
* The GDC contains many features that are out of Arranger's scope