Skip to content

menchelab/VRNetzer

main
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 

VRNetzer - A Virtual Reality Framework for Network Visualization and Analytics

alt text

Networks provide a powerful representation of complex systems of interacting components. In addition to a wide range of available analytical and computational tools, networks also offer a visual interface for exploring large data in a uniquely intuitive fashion.
However, the size and complexity of many networks render static visualizations on common screen or paper sizes impractical and result in proverbial 'hairballs'. Here, we introduce an immersive Virtual Reality (VR) platform that overcomes these limitations and unlocks the full potential of visual, interactive exploration of large networks.
Our platform is designed towards maximal customization and extendability, with key features including import of custom code for data analysis, easy integration of external databases, and design of arbitrary user interface elements. As a proof of concept, we show how our platform can be used to interactively explore genome-scale molecular networks for identifying genetic aberrations responsible for rare diseases and develop hypotheses on their respective pathobiological mechanisms.
Our platform represents a first-of-its-kind, general purpose VR data exploration platform in which human intuition can work seamlessly together with state-of-the-art analysis methods for large and diverse data.

Note: This project is still in an infant state of development. This is a public beta release, meaning it is not fit to be used in any critical applications yet and may still contain some bugs. You can get in touch with us if you would like to contribute to the further development of this project.

Content

Hardware Requirements

Installation: Quick Start

Installation: Stand Alone

Desktop Mode

Data Logging

Take screenshots and record videos

Tutorial 1: Using the Uploader to add your own network

VRNetzer Architecture Overview

Tutorial 2: Creating custom User Interfaces

Tutorial 3: Creating a route on the backend

USE CASE Gene prioritization

VRnet API Documentation

Dependencies

License

Hardware Requirements

We are running the platform on an MSI gaming laptop with the following specs:

  • CPU: i7 - 7820HK
  • RAM: 16 GB
  • GPU: NVIDIA 1070 TI
  • OS: WINDOWS 10

and a VR HEADSET - we tested the following:

  • HTC VIVE VR Headset + controllers
  • OCULUS QUEST (tethered to windows with link) + controllers

Installation: Quick Start

Time required ~ 10min

For a quick start, you can just download the VR Module executable and run VR_Module.exe on your windows computer with a SteamVR compatible headset. Please watch this video that will help to get you started. The VR Module comes preconfigured to connect to the other modules that we already installed on our server to make it easy for you to get a first impression. It has the ability to upload your own datasets, but we don't recommend it. Do NOT upload any sensitive data here, this is only for demonstration purposes and for the whole world to see. We don't guarantee your data's safety! If you want to work with your own data you should use the Stand Alone version.

Installation: Stand Alone

Time required ~ 60min

However, if you want to implement your own datasets and functionalities, you can also run everything localy on your machine (the SAFE option if you are dealing with sensitive data) or run the Analytics Module and UI Module on a networkserver or cloud service (if you need scalability and processing power).
Here is a step by step guide how to do a local installation on a windows computer. Note that the Analytics Module and the UI Module can also run on a (linux) server, while the VR Module needs to run on a windows 10 machine as gaming hardware is required for Virtual Reality which is best supported under windows.

1.) Set up a MySql Database

  • download and install MySql Workbench
  • download the .dump file of our database from here
  • Create a new database, user and password and upload .sql file to it

you have now a clone of our database on your local machine.

2.) Clone the Analytics Module repository into a new folder somewhere on your computer called "VRNetzer"

  • install python >3.6 <3.9, make sure its added to path variable

  • install pip for python

  • edit VRNetzer/Analytics_Module/db_config.py to match the database host and user you created before and save

  • right click on VRNetzer/Analytics_Module/runAnalytics_Module.ps1 -> "run with power shell"
    alt text

  • if the output of the console window that just opened ends with "Running on http://127.0.0.1:1337/" you are good to go, if there are errors you will need to install dependencies.

3.) Clone the UI Module repository into "VRNetzer"

  • install python >3.6, make sure its added to path variable

  • install pip for python

  • edit VRNetzer/uiserver/static/js/UI_Module_Config.js and set "dbprefix" to the address of your dataServer , here: 'http://127.0.0.1:1337' and save.

  • edit VRNetzer/UI_Module/static/js/ UI_Module_Config.js and set "dbprefix" to the address of your Analytics Module, here: 'http://127.0.0.1:1337' and save.

  • right click on VRNetzer/UI_Module/run_UI_Module.ps1 -> "run with power shell"

  • if the output of the console window that just opened ends with "Running on http://127.0.0.1:5000/" you are good to go, if there are errors you will need to install dependencies.

4.) Download the VR Module executable and extract it to "VRNetzer"

  • if you haven't already, make a Steam account and install SteamVR and test your headset
  • edit VRNetzer/VRnet/viveNet/Content/data/Config.txt and change the address to the one where your UI Module is, here http://127.0.0.1:5000/
  • run VR_Module.exe

Desktop Mode

It is possible to run the VR Module in desktop mode which is useful for development tasks but does not provide the full functionality as with a headset.
To do so, change the value "vr":true in the config file of the VR Module to "vr":false and restart the VR Module.

Data Logging

The VR Module can produce log files that contain a list of all that has happened during a VR session. This produces quite big textfiles and is therefore turned off by default.
To enable logging change the value "logging":false in the config file of the VR Module to "logging":true and restart the VR Module. The log files are generated in VRNetzer/VR_Module/viveNet/Content/data.

Take screenshots and record videos

If you want you can take screenshots and videos of the running application using nvidia geforce experience, which is a free download for nvidia graphics cards, find it here: geforce experience. After installation the default key combination for taking a screenshot is alt+f1 and to start or stop a video recording you press alt+f9. Depending on your setup you might need to enable the ingame overlay and enable desktop recording in the privacy settings in the geforce experience settings if the hotkeys do not work right away.

Tutorial 1: Using the Uploader to add your own network

WINDOWS

  • right click on VRNetzer/ Analytics_Module/runAnalyticsModule.ps1 -> "run with power shell" to start the Analytics Module
  • right click on VRNetzer/ UI_Module/run_UI_Module.ps1 -> "run with power shell" to start the UI Module
  • open the web frontend of the UI Module in a browser /http://127.0.0.1:5000/upload

MAC/LINUX

  • open terminal, cd to Analytics_Module and run app.py with python3
  • open terminal, cd to UI_Module and run json_io.py with python3

  • select .csv files to upload, they must be formatted after these guidelines as sketched in the picture above.
  • Required is as least a layout file with xyz coordinated (normalized each in [0,1]) and colors and you will obtain a 3D point cloud.
  • If you add an edge list that refers to the identifiers given in the 1st column of the layout file you will obtain a network with nodes and edges.
  • Uploading a node file with symbols, names and functions/features (optional, can remain empty) you will find the symbols as labels in VR and the node characteristics on the node panel when clicking on a node.
  • restart the VR Module and load your project

VRNetzer Architecture Overview

The VRNetzer platform consists of 5 Modules:
alt text

VRNet - the virtual reality module written in Unreal Engine

Unreal Engine is one of the industry leaders in the videogame world. We chose it as the base for our VR Module for it's astounding graphics performance, continuous support of upcoming VR hardware and because it's open source.

UI Module - a jQuery and html website

The UI Module is running in the browser of your local machine and can SEND GET and POST requests TO the Analytics Module. Nearly all actions originate here (exept the ones in the VR Module, like if the user touches a node or moves the network with the VR controllers).
Think of the UI Module as the frontend of a website and the Analytics Module as it's backend.
A User clicks on a button on the frontend, this makes it to send a post request to a specific URL (route) on the backend (the Analytics Module), await it's response and finally display the result as text or as a graph on the user interface and/or call functions in the VR Module to alter the appearence of the network displayed in VR.
The Analytics Module can only RESPOND to those requests, meaning the Analytics Module can never send something to the frontend without being asked. Every communication is Initiated by the UI Module.

Now here is what's special about the UI Module:

  • It can also SEND api function calls to the VR Module


AND

  • It can RECEIVE calls from the VR Module

In contrast to the Analytics Module, the VR Module CAN initiate communication with the UI Module and call special functions set up in the UI Module. This picture illustrates the different routes of communication in the framework.

alt text

Analytics Module - a Flask/Python webserver

The Analytics Module is the backend that performs all the data science tasks. It has

  • separate routes (URLs) defined for each task
    When the UI Module sends a request to one of these, it parses the input parameters, makes database queries and performs calculations, and finally returns it's response to the UI Module.
  • it can run on the local machine or on a powerful cloud server if more power is needed

SQL database

All data is stored within a MySQL database. The following schema shows the minimal set of tables that describes one single network.

drawing

The schema and the related tables are auto-generated and auto-populated upon user upload. To upload your own network, see Tutorial 1.

For advanced users:

Additionally, you may want to include more data to contextualize your nodes or edges. For example, in our proof of concept application, we use additional tables to store:

  • hierarchical information of our taxonomic node annotations (attribute_taxonomies)
  • articles mentioning specific genes (articles, nodes_articles)
  • gene expression levels of genes in different tissues (gtex_values)

The possibilities are endless! Functions to query or manipulate the database are in tables.py of the Analytics Module.

Tutorial 2: Creating custom User Interfaces

The User Interface in the virtual reality module is a website made with jQuery. In this tutorial you will learn how to make UI elements like buttons or dropdown lists and how to communicate with the VR and Analytics Module.
Basic JavaScript and HTML skills required.
Prerequisites: install the VRNetzer framework on your local machine Installation: Stand Alone

Step 1: Create a button

  • in Visual Studio Code (or the text editor of your choice) File -> Open Folder -> navigate to your UI Module Folder

  • right click on VRNetzer/Analytics_Module/runAnalyticsModule.ps1 -> "run with power shell"
    alt text

  • right click on VRNetzer/UI_Module/runUIModule.ps1 -> "run with power shell"

  • open the VR Module's config.txt file located at VRNetzer/VR_Module/viveNet/Content/data in your text editor and change "vr":true to "vr":false and "UIServerAdress":"....." to "UIServerAdress":"http://127.0.0.1:5000/" or where your UI Module is running and save the file

  • run VR_Modul.exe

  • it might promt you to install DirectXRuntime, follow the instructions

  • Once the VR Module opens, you will see the main UI to which we will add something in the right-most tab in the left upper corner of the 3D view.

  • press Alt + Enter to switch from fullscreen to windowed mode.

  • click on the user interface in the left corner and hit 'Ctrl + Shift + i' to open the developer tools, click on "Network" and tick the "Disable Cache" checkbox. Note the "Console" window, where debugging output is displayed.
    alt text

  • open VRNetzer/UI_Module/templates/main.html in your editor

  • at the end of the file, after <div id="tabs-7"> add <button id="MyNewButton"> EXIT </button>

  • save changes to main.html and refresh the browser by clicking in the red area and hit F5
    alt text

  • if you now navigate to the right most tab again, the button appeared. It looks grey though, not like the other buttons.

The different elements of the page are set up in the html documents. All the logic happens in the JavaScript, in the UI_Module\static\js folder.

VRNetzer_API.js is the most important, it has all the functions to communicate between the Analytics Module and the VR Module.

Then there is a file for each of the html files (main_UI.js). These all have this initialization function $(document).ready(function () {....} and in there are functions that bind to the UI elements created in Html. For our button we do this:

  • at the end of main_UI.js, but still inside the document.ready() function put this code:
$(function () {
        $("#MyNewButton").button();
        $("#MyNewButton").click(function (event) {
            event.preventDefault();
            logger("click!") 
        });
    });
  • save and refresh the browser
    alt text

Now that jQuery is aware of the new button the css styling also works. Click it and the console outputs the message so we know it works.

Step 2: Communicate with the VR Module

  • put this code in the MyNewButton.click() function
var args = {
    "content": "somecoolName",
    "route": "tutorial"
};

ue4("GetSelection", args);

Everything that looks like ue4("name","argument") are calls to the VR module, the so called VRnet API documented here.

The variable "args" is a JavaScript object and follows the JSON syntax.

The "GetSelection" call above returns the currently selected nodes in the VR module to this return function ue.interface.getSelection specified in VRNetzer_API.js around line 30

//// FUNCTIONS CALLED BY UE4

ue.interface.getSelection = function (data) {
    logger(data);

    switch (data.route) {
     case "saveSelection":
         SaveSelectionDB(data);
         break;
     case "reLayout":
         ReLayoutSubSet(data);
         break;
     case "GSEA":
         GSEASubSet(data);
         break;

    }

};

Because the GetSelection call is used for different things, it's arguments contain "route": "tutorial" that is used in the return function to determine what to do with the response.

  • in the return function above, add a case "tutorial" to the switch statement
    case "tutorial":
        LogOnUIServer(data);
        break;

LogOnUIServer(data); prints out the data we receive from the VR Module on our python console.

That's it! let's start the VR Module and check if everything works.

  • start viveNet.exe
  • open a layout
    alt text
  • load a selection
    alt text
  • click on MyNewButton

You should see some printout in the UI Module console window. We have now initiated communication from the UI module to the VR module (by pressing the button) and received a response. In the next step we will see how to do the same with the backend, the Analytics Module.

Tutorial 3: Creating a route on the backend

Basic Python knowledge required
Prerequisites:

  • local installation of the VRNetzer framework Installation: Stand Alone

  • Tutorial 2 finished

  • in a new instance of Visual Studio Code (or the text editor of your choice)
    File -> Open Folder -> navigate to your ** Analytics_Module ** folder

  • in app.py, at the end of the document but before if __name__ == "__main__": insert

@app.route('/api/<string:db_namespace>/MyNewRoute', methods=['POST'])
@cross_origin()
def my_new_route(db_namespace):
    data =request.get_json()
    node_ids = [int(x) for x in data['node_ids']]
    print(node_ids)
    return jsonify(node_ids)
  • save app.py and bring up the Analytics Module console, it should detect the change and restart automatically
    alt text

That’s it! Now we will go back to the UI Module and forward the output of the VR module that we created previously in tutorial 2.
For this we will use a POST request.

  • go to the UI Module project and open it in a new Visual Studio Instance.
  • paste this code at the end of VRNetzer_API.js
function MyNewPostRequest(data) {

    payload = JSON.stringify(data);
    //logger(payload);
    path = dbprefix + "/api/" + thisNamespace.namespace + "/MyNewRoute";
    $.ajax({
        type: "POST",
        url: path,
        contentType: "application/json",
        data: payload,
        dataType: "json",
        headers: {
            "Authorization": "Basic " + btoa(dbuser + ":" + dbpw)
        },
        success: function (response) {
        // DO SOMETHING WITH RESPONSE HERE!
            logger(response);
        },
        error: function (err) {
            logger(err);
            logger(data);
        }
    });

}

This is a blank POST request that calls the route we created before.

path = dbprefix + "/api/" + thisNamespace.namespace + "/MyNewRoute" is the target URL, dbprefix comes from UI_Module_Config.js and thisNamespace.namespace is the name of the project. path = dbprefix + "/api/" + thisNamespace.namespace + "/MyNewRoute" is the target URL, dbprefix comes from Analytics_ModuleConfig.js and thisNamespace.namespace is the name of the project.

  • also in VRNetzer_API.js, in the function ue.interface.getSelection put MyNewPostRequest(data); instead of LogOnUIServer(data);

CSV file formats

You can find examples of all used .csv formats HERE to to use as templates for the formatting.

Note: You need to assign unique node integer IDs (like EntrezID in case of genes) for your files to upload
Don't use commas in string fields, as they are reserved delimiters.

Layout Lists look like this:

8473,0.4993,0.4544,0.640,188,20,26,100,3dportrait
...

where each line is a node with the following data:

8473 0.4993 0.4544 0.640 188 20 26 100 3dportrait
NodeID X-Pos Y-Pos Z-Pos R G B A Namespace

Note: Please provide XYZ coordinates for the layout that are normalised between 0 - 1

RGBA colours range from 0 - 255 Note: "A" (in RGBA) value's should be 100. Bigger values makes nodes glow, smaller values make them darker.

Node Lists

It is recommended to provide a Node file that converts NodeIDs given in the Layout file into readable symbols and names. There is also the possibility to fill the 4th column with functions, descriptions or other properties that should show up on the node panel within the VR. Please make sure that no commas appear within these fields as the are used as delimiters!

The format could look like this:

8473 OGT O-linked N-acetylglucosamine (GlcNAc) transferase protein coding 3dportrait
NodeID Symbol Name Function Namespace

Symbols will appear as labels directly at selected nodes and at nodes in the proximity.

Link Lists

1267,2945
...

where

1267 2945
Start End

The identifiers refer to the NodeIDs given in the Layout file. For now, unweighted link lists are supported and only one link list per project is allowed.

Attributes

For every node multiple types of attributes can be given. The type can be specified as namespace (e.g. DISEASE). One gene can have multiple attributes (such as a gene can be associated to multiple diseases). And vice versa one specific attribute can be distributed over several nodes.

node_id attribute_id namespace name description
3543 DOID:2825 DISEASE nose disease none
25836 DOID:2825 DISEASE nose disease none
4852 DOID:2825 DISEASE nose disease none
4524 DOID:1936 DISEASE atherosclerosis none
4524 DOID:7148 DISEASE rheumatoid arthritis none
4524 DOID:4535 DISEASE hypotrichosis none

Note The database structure allows also attributes that are organized as an ontology tree. However it is not part of the uploader functionality yet. We plan to provide it with the next release.

Selection Lists

An important feature are selection lists. They define subsets of your nodes that are of interest for specific task. (Such as a gene subset representing a list of patient variants.) They can be loaded separately in a VR session to always have them highlighted. Moreover they serve as the initial set for several analytical procedures such as seeds for a random walk.

Selection lists are a sub category of attributes and can be uploaded in the following way:

node_id attribute_id namespace name description
3543 my_ID:1 SELECTION my_set_1 none
4524 my_ID:1 SELECTION my_set_1 none
25836 my_ID:2 SELECTION my_set_2 none
4852 my_ID:2 SELECTION my_set_2 none

Please provide as attribute_id an user defined ID for your selection. The name can be picked up in the VR session.

Labels

x_loc y_loc z_loc text namespace
0.5000000 0.5000000 0.5000000 C E L L U L A R C O M P O N E N T S my_layout

A simple way to add textlabels at certain positions to a specific layout (namespace)

Note

USE CASE Gene prioritization

This refers to the chapter 'Application to prioritization of genomic variants' from the publication describing one specific use case for identifying a single disease causing gene out of a set of given variant genes.

In the VRNetzer release 1.1 candidate and seed genes are already contained as a prepared data set. For this specific task we include an extra patient panel that looks like this


alt text

It may include patient and diagnosis data and a list of variants. The second tab shows the manually curated seed genes:


alt text

From both lists the genes symbols can be clicked to be highlighted in the network while the gene panel shows the gene characteristics for individual inspection. As described in the publication the sets can be analyzed in terms of their network properties for example. In order to show their interconnectivity or lcc size select the network analytics tab on the control panel (turn your left wrist/controller) and use the isolate and re-layout functions. (NOTE: the LINKS needs to be loaded)


alt text

To contextualize the variant genes in the functional set of seeds run the random walk on the 3rd tab of side panel. It will highlight the variants (shiny red), seeds (shiny yellow) and linker genes (shiny blue) in the 3D network. A bar chart within the side panel presents a list of genes resulting from the random walk ranked by their visiting probabilty. Depending on the choice of the restart parameter value (high r means higher localization around initial gene set) you will find DOCK2 among the first few candidates.


alt text

A force-graph layout of the resulting random-walk network can be inspected separately in the 5th tab of the side panel. The color legend is the same as in the 3D network and the nodes are interactive so that clicking on them refers directly to the 3D network where the gene panel opens as well.


alt text

Saving your results makes them accessible from outside the VR session: http://127.0.0.1:5000/_side

VRnet API Documentation

The following function calls are sent from jQuery to the VR module.

The syntax looks like this:

ue4("rw_result", response);

where "rw_results" is the function name and "response" the parameters.

This list will grow in the future.

function parameters type description
LoadDbNodeList node list object json loads a node list into channel A of the layout
LoadDbNodeListB node list object json loads a node list into channel B of the layout (only xyz, no colors)
LoadDbLinkList link list object json loads a link list from db
LoadDbLabelList label list object json loads a label list for channel A of the layout
LoadDbLabelListB label list object json loads a label list for channel B of the layout
Init --- --- start initialization routine, OnDocumentReady()
Morph "A" or "B" string Morph between layout channel A and B
GetSelection { "content": "Name","route": "saveSelection"} json Get active selection and do different things depending on route (save it on db in this case)
SetScale value float set Network Scale
SetNodeSize value float set Network SetNodeSize (linksize is also affected by this)
SetLinkSize value float set Network SetLinkSize ATTENTION: big diameters cause lag
SetLinkAlpha value float set Network Link transperancy
SetLight value float set scene light intensity
ShortestPathPoint "p1" or "p2" string get selected node ID from VR and define it either as start or endpoint
ExitIsolate --- --- exit Isolate Selection Mode (show all links)
Rw_Result rw object json display the random walk results in VR
Julia {"seeds":[{"node_id":123}...],"variants":[...],"linker":[...]} json Gene Priorization example on side panel
ActivateNode id int select and highlight single node in network
ReLayout node list object json show newly created layout of subset in VR (this resorts the link list)
VRkeyboard route string opens a keyboard in VR - after user presses ENTER, typed string is returned to a .js function by the same name as route - so you need to create this
loadSelection name string Deprecated Load selection from csv file

Dependencies

Analytics Module UI Module VR Module
Win, Mac, Linux Win, Mac, Linux Win 10
python==3.6
click==7.1.2
decorator==4.4.2
fisher==0.1.9
Flask==1.1.2
Flask-Cors==3.0.8
itsdangerous==1.1.0
Jinja2==2.11.2
MarkupSafe==1.1.1
networkx==2.4
numpy==1.19.4
pandas==1.2.0
PyMySQL==0.9.3
python-dateutil==2.8.1
pytz==2020.5
six==1.15.0
Werkzeug==0.16.1
jquery==3.4.1
jquery-ui
python==3.6
Flask==1.1.2
Flask-Cors==3.0.8
jquery==3.4.1
jquery-ui
d3v4
Epic Unreal Engine 4.23
WebUI by Tracer Interactive
Direct X Runtime
SteamVR

Data availability

source url version/realease date
GO taxonomy http://geneontology.org/docs/download-ontology/ Mar 2019
GO annotations http://current.geneontology.org/products/pages/downloads.html Mar 2019
DO taxonomy https://github.com/DiseaseOntology/HumanDiseaseOntology/
tree/main/src/ontology/releases
Dec 2018
DisGeNET https://www.disgenet.org/downloads Dec 2018
OMIM https://www.omim.org/ Feb 2019
HPO taxonomy https://hpo.jax.org/app/download/ontology Sep 2018
HPO annotation https://hpo.jax.org/app/download/annotation Sep 2018
KeGG https://www.genome.jp/kegg/genes.html Feb 2019
BioGRID https://downloads.thebiogrid.org/ Jun 2019
REACTOME https://reactome.org/download-data Jul 2019
Gtex https://gtexportal.org/home/datasets V7
Pubmed articles from INDRA INDRA module May 2019
INDRA https://github.com/sorgerlab/indra May 2019
HIPPIE (Interactome) http://cbdm-01.zdv.uni-mainz.de/~mschaefer/hippie/download.php Jul 2017

Software

Python==3.6
JavaScript==ES6
MySQL==8.0
Apache==2.4.29 (Ubuntu)
EPIC Unreal Engine 4

License

Copyright (c) 2021 Menche Lab

This project is licensed under the terms of the MIT license. Check the LICENSE.md file for details.

This Readme:
DOI
UI Module:
DOI
Analytics Module:
DOI

About

A Virtual Reality platform for interactive, immersive network exploration.

Topics

Resources

License

Stars

Watchers

Forks