-
Notifications
You must be signed in to change notification settings - Fork 4
Configuration of BioRels backend
-
Postgres database:
-
Depending on the scope – up to 25Tb of disk space
-
Can be located on AWS or other cloud environments.
-
Please have at hand the host, port, user, password and database values handy. To create a new database, check here
-
-
Computing power:
-
SGE Cluster
-
OR Single CPU - with limited features
-
OR Any other cluster application (With a little work)
-
-
Singularity container application - Installation — Singularity container 3.0 documentation (sylabs.io)
-
Disk space:
-
depending on the scope – it can be from 500Gb to 35 Tb. For instance, clinical trials/drugs/ontologies/taxon/pubmed can be stored with less than 500Gb. Adding proteome or genomes can greatly increase that amount necessary.
-
More disk space might be necessary if wanting to keep the previous dataset versions
-
We suggest that you clone the BioRels repository from github to get the latest code update and bug fixes.
git clone https://github.com/EliLillyCo/BioRels.git
If you wish to develop your own scripts, you should create your own branch. In the case you wish to publish your branch, a good recommendation would be to name your branch after your lab/organization name.
Git checkout -b \<branch_name\>
This will download all the code necessary to run the backend infrastructure (BACKEND) and the website (WEBSITE). Once you have downloaded the source code, let’s configure everything.
The setenv.sh file contains all the global variables required to properly execute the different scripts. It is located in BACKEND/SCRIPT/SHELL/setenv.sh. This specific script will be called during the execution of every script to load all those global variables.
The table below provides a list of the different variables that should be modified to fit your infrastructure. In addition, any proxy variables should be defined in this file.
| Variable name | Description |
|---|---|
| REQUIRED: | |
| TG_DIR | Root directory path of biorels repository |
| PGPASSWORD | Postgres password |
| PGUSER | Postgres user |
| DB_PORT | Postgres database port |
| DB_HOST | Postgres database hostname |
| DB_NAME | Postgres database name |
| DB_SCHEMA | Postgres database schema |
| OPTIONAL: | |
| SINGULARITY_BIND |
Singularity allows you to mount directories on your host system to directories within your container using bind mounts. This allows you to read and write data on the host system with ease. More information: Bind Paths and Mounts — Singularity container 3.0 documentation (sylabs.io) $SGE_ROOT must be added to enable cluster job submission. |
| SCHEMA_PRIVATE |
Optional – Name of the private schema If you want a private schema in parallel to the public schema. |
| FOR GRID ENGINE: | |
| SGE_QMASTER_PORT | specifies the tcp port on which sge_qmaster is expected to listen for communication requests |
| SGE_EXECD_PORT | The port for sge_execd is currently set as service. Default: 6435 |
| SGE_CLUSTER_NAME | Name of this cluster (used by SMF as a service instance name) |
| DRMAA_LIBRARY_PATH | Libdrmaa.so path for the Distributed Resource Management Application API (DRMAA) |
| LD_LIBRARY_PATH | Provides location of dynamically linked libraries |
| PATH |
Directories to be searched to find a command. lx-amd64 directory must be added to that path |
| PROXY (OPTIONAL): | |
| http_proxy | Path to your proxy – if any |
| https_proxy | |
| HTTP_PROXY | |
| HTTPS_PROXY | |
| HTTP_proxy | |
| HTTPS_proxy | |
Important
One of the most important variable here is TG_DIR as it represents the absolute root path directory of the repository. All scripts, files and documents are using TG_DIR.
If you want to use another cluster application, please follow this guidelines to set them up: link
If you wish to add more variables, please ensure that are also set in the environment file for the singularity contained, located in BACKEND/CONTAINER/env-file.txt. This file serves as a mapping file between setenv and the singularity container to know which environment variables to load in the container.
To simplify the code execution, several aliases have been created in setenv.sh
- Biorels_exe to execute a script with some parameters.
biorels_exe php test.php
- Biorels_php is an alias to biorels_exe to execute php scripts:
biorels_php test.php
-
Biorels_run allows you to get in the container
-
Biorels_monitor run the monitoring script
-
Biorels_python is an alias to biorels_exe to execute python scripts
biorels_python test.py
- Biorels_sql is an alias to biorels_exe to run psql
Once set, you will need to source the setenv.sh script. To do so, please run the following command from the root directory of the repository:
source BACKEND/SCRIPT/SHELL/setenv.sh
No output should be returned.
Tip
You can check that all your configuration is successful by running a couple commands:
-
ls $TG_DIRshould list you the BioRels directories: BACKEND, WEBSITE ... - biorels_sql and other commands will not work until the container is built
Important
Every time you will open a new console, you will need to source this file
To execute any code within Biorels, you will first need to compile the singularity container that will provide all the necessary third-party packages and tools. If you haven’t already installed singularity, please follow this link: Installation — Singularity container 3.0 documentation (sylabs.io).
All the configurations, third party packages and tools are defined in biorels.sing.txt, located in $TG_DIR/BACKEND/CONTAINER. To compile the container, make sure to have the environment variables set by reviewing the section on Environment variables – setenv.sh if you haven’t already. Once done, execute the following commands:
cd $TG_DIR/BACKEND/CONTAINER
singularity build biorels_container.sif biorels.sing.txt
The process can take up to 90 minutes (about 1 and a half hour). If all goes well – which it should, you should have a file called biorels_container.sif in $TG_DIR/BACKEND/CONTAINER. You can test if your container is successfully compiled by running the following command:
biorels_php -v
You should have something like this as a result:
PHP 8.3.9 (cli) (built: Jul 5 2024 12:04:09) (NTS gcc x86_64)
Copyright (c) The PHP Group
Zend Engine v4.3.9, Copyright (c) Zend Technologies
Note
If you run into issues, we sincerely apologize for it. We want to give you the best experience possible, but we cannot test it on all platforms. Please, don't hesitate to raise a ticket and we will be happy to help you as soon as we can.
Now that all the environment variables have been set and sourced, we can now install the database. Please execute the following commands:
cd \$TG_DIR/BACKEND/INSTALL/
biorels_php 1_runInstall.php
- First, it will ask you to confirm that TG_DIR and DB_SCHEMA that you have set in setenv.sh are the correct values.
- Once you confirm those two values, the script will then generate the database schema.
- It will read biorels_public.sql which contains the generic schema
- It will modify it to adapt the schema template name to your $DB_SCHEMA value, and then create a file called schema_ready.sql.
- This sql file will then be executed to create the schema $DB_SCHEMA, the tables and sequences necessary for the infrastructure.
- If you have set SCHEMA_PRIVATE in setenv.sh file, it will do the same, this time reading biorels_red.sql to generate private_schema_ready.sql that will be executed to create the $SCHEMA_PRIVATE schema and its tables and sequences.
Following the schema generation, you will need to create the wrapper scripts specific to the container type (here in this case Singularity).
cd \$TG_DIR/BACKEND/INSTALL/
biorels_php 2_genContainerFile.php
All scripts in $TG_DIR/BACKEND/SCRIPT/SHELL/ will be reviewed to ensure the called script exists and a new shell in $TG_DIR/BACKEND/CONTAINER_SHELL will be generated so it can be executed within a container.
We strive at making things simple. For this reason, we created a script that will guide you through the different configuration steps. To run it, please execute the following commands:
cd \$TG_DIR/BACKEND/INSTALL/
biorels_php 3_prepConfigJob.php
This script is in 5 distinct steps. You can stop the process at any step to come back to it later as long as you finish the process within the same day. The reason for this is that it will generate a directory named by the current date and will generate in this directory one file per step. Therefore, if you want to:
-
Restart the process: delete the directory with the current date
-
Restart a finished step: delete the corresponding file.
-
Genome selection: GENOMIC and GENOMIC_RULESET
-
Proteome selection: PROTEOMES
-
Data sources selection: DATASOURCES
-
Global options: GLOBAL_OPTIONS
-
Final step: NEW_CONFIG_USER
-
With this process and the generated directory, you can come back to see the options you have selected at any given time.
You can also look at the BioRels wiki page on BioRels-Infrastructure-in-depth, section config_user--user-configuration-file to learn more.
The first step is the selection of genomes:
STEP 1 - Select organisms
Do you want to process genomic assemblies? Y/N.
You will be prompted to choose whether you want to process genome assemblies or not. If not (Answer: N), you can proceed to the next section. If you wish to process genome assembly – answer Y. This will trigger the download of RefSeq and Ensembl list of assemblies, grouped by organisms. You will then be prompted to provide the NCBI Taxonomy ID of the organisms of interest (one at the time).
Please provide the NCBI Taxonomy Identifier for the organism of interest
or N to stop.
Note: If don't know it, please search for it here:
<https://www.ncbi.nlm.nih.gov/taxonomy>
Example: If you wish to consider Human, type down 9606. It will then provide you with a list of assemblies for this organism. Below is an example of that list – we only present the first few columns to make it more readable. In this instance (December 4th 2023) – there are 3 assemblies available for Homo Sapiens. 2 from RefSeq, GRCh38.p14 which is the Human Reference Genome and T2T-CHM13v2.0 from Telomere 2 Telomere project. Ensembl only offers GRCh38.p14. Each assembly is defined by an ID that you can use to select the assembly. If you want to get GRCh38.p14 from both resources, then type 0,2.
ID SOURCE ASSEMBLY_ACCESSION ASSEMBLY_NAME
0 REFSEQ GCA_000001405.29 GRCh38.p14 GCF_000001405.40 Homo sapiens latest
1 REFSEQ GCA_009914755.4 T2T-CHM13v2.0 GCF_009914755.1 Homo sapiens latest
2 ENSEMBL GCA_000001405.29 GRCh38.p14 2014-01-Ensembl/2023-03 Human N/A
Please choose among the following assemblies below
If you wish multiple assemblies, list the IDs separated by comma.
Example: 1,3
Once done, you will come back to the previous prompt asking to provide the NCBI Taxonomy ID so you can select assemblies from other organisms. When you went through all the organisms, input N to stop.
The second step will focus on selecting proteomes from UniProt. If you don’t wish to consider specific proteomes, type N and move on to the next step.
Do you want to process proteomes? Y/N. Then press return: Y
Please wait while we download the list of proteomes
24004 proteomes listed
Please provide the NCBI Taxonomy Identifier for the organism of interest
or N to stop.
Note: If don't know it, please search for it here:
https://www.ncbi.nlm.nih.gov/taxonomy
Your value:
The list of proteomes offered by UniProt will then be downloaded and will be searchable by NCBI Taxonomy Id. If you don’t know the corresponding NCBI Tax ID, you can go to https://www.ncbi.nlm.nih.gov/taxonomy and search for it.
Your value: 9606
PROTEOME_ID SUPERREGNUM SPECIES_NAME
0 UP000005640 eukaryota Homo sapiens (Human)
Please choose among the following proteomes below
If you wish multiple proteomes, list the IDs separated by comma.
Example: 1,3
Uniprot usually provides only 1 proteome per organism. However, to remain generic enough, we provide the list of all proteomes and prompt you to select the one(s) you want. If you wish here to select the proteome UP000005640, type 0. You will then come back to the first Proteome prompt asking about the next NCBI Taxonomy ID if you want to consider multiple proteomes. Once done, you can type N to move on to the next step
Biorels offers more than 30 data sources, with various levels of integration. The first few lines of the prompt will summarize which data source you will need based on the genome and proteome selection steps. Therefore, in this third step, you will be prompted to select which data sources you want to process. If the data sources selected from the previous steps are enough, you can move forward by typing N/A. If you want them all, type ALL. Otherwise, it will list all available resources. You will then have to type the ones you want, separated by space. Do not worry about dependencies, the script will for it afterward.
Based on genome information and options, you will need the following
data sources:
REFSEQ GENOME ENSEMBL UNIPROT
Please provide the list of databases you wish to consider among the list
below, each separated by space:
Or N/A if already covered by previous steps
Or ALL if you want all resources
TAXONOMY
GENE
SEQ_ONTO
GENOME
VARIANT
TRANSCRIPTOME
PUBLI
In the example below, we ask to add the TRANSLATE process, which will try to translate mRNA to protein.
Your choice(s): TRANSLATE
You have selected TRANSLATE, REFSEQ, GENOME, ENSEMBL, UNIPROT
You will need those datasources too: GENE, SEQ_ONTO, PUBLI, ECO, GO,
TAXONOMY
Crudely estimated database size: 268905Mb
Do you want to proceed Y/N. Then press return
From your selection, the process will also identify the necessary dependencies. In the case of UniProt, it will be Taxonomy, ECO ontology, Gene Ontlogy, and Publication. RefSeq/Ensembl will require Taxonomy, Gene and Sequence Ontology. It will then ask you to confirm if this selection is correct or not.
If you reached this step, you will have been prompted to select genome assemblies, proteomes and data sources. The script will then proceed in providing you a summary of your current choices:
##################
##################
##################
Summary of current selection:
-\> Genomes:
9606 9606 REFSEQ GCA_000001405.29 GRCh38.p14 GCF_000001405.40 Homo sapiens latest Patch reference genome 2023/10/02 vertebrate_mammalian
9606 9606 ENSEMBL GCA_000001405.29 GRCh38.p14 2014-01-Ensembl/2023-03 Human N/A N/A N/A N/A N/A
-> Proteomes:
9606 UP000005640 9606 HUMAN eukaryota 20596 84119 108303 Homo sapiens (Human) 9606
-> Data source needed based on Genome/Proteome selection:
REFSEQ
GENOME
ENSEMBL
UNIPROT
-> Data source you requested or depends upon:
TRANSLATE REQUESTED
REFSEQ REQUESTED
GENOME REQUESTED
ENSEMBL REQUESTED
UNIPROT REQUESTED
GENE ADDED
SEQ_ONTO ADDED
PUBLI ADDED
ECO ADDED
GO ADDED
TAXONOMY ADDED
In this example, we have selected 2 Human GRCh38.p14 assemblies, one from RefSeq and one from Ensembl. We also asked to consider the Human Proteome from UniProt and requested TRANSLATE data source. Overall this will configure the system to process 11 data sources: 5 requested and 6 added due to dependencies. You will then be asked a series of question to further customized the infrastructure.
A private schema name has been set in setenv.sh. Do you want to use it? (Y/N) – If you have specified SCHEMA_PRIVATE in setenv.sh, this will ask to confirm that you indeed want to populate that schema. If you answer Y, it will set PRIVATE_ENABLED to T in CONFIG_USER.
Do you want to keep files from previous releases? (Y/N) – In some instances, such as regulatory purposes, you might be required to keep previous releases. If such case applies to you, type Y.
Please provide an email address to send issues to: When a script fails, Biorels will send an email to this email address
Please provide an email address from which the email will be sent from: When a script fails, Biorels will send an email using this email address.
Please provide a prefix for the job names (Default BR_) - Max 5 letters: This prefix is used in all job submission to the cluster to identify them from other non biorels jobs.
ChEMBL: Genes are not required by default for ChEMBL. Do you want to add Gene annotations for ChEMBL records? If you requested ChEMBL but not GENE, you will be prompted this question. If you choose Y, this will allow to connect Gene annotations to ChEMBL assays.
X-Ray: Genes are not required by default for X-Ray. Do you want to add Gene annotations for X-Ray records? If you requested XRAY but not GENE, you will be prompted this question. If you choose Y, this will allow to connect Gene annotations to X-Ray structures.
DrugBank: Please provide your Drugbank API login (Format: USER:PASSWORD). DrugBank API isn’t a free resource. Therefore, you will need to provide your own User & password to enable it. Otherwise it will be disabled.
OMIM: Please provide your OMIM API Key. OMIM API isn’t a free resource for all. Therefore you will need to provide your own API Key or OMIM will be disabled.
Uniprot: Do you want to download/process Swiss-Prot? (Y/N). Biorels provides the option – on top and independently of the proteomes – to process the whole Swiss-Prot. If you want to do so, Type Y.
Uniprot: Do you want to download TrEMBL Y/N . In addition to Swiss-Prot and proteome, Biorels also can download TrEMBL. However, due to the size and high level of changes occurring between releases, it will not process and push it to the database. If you wish to download TrEMBl, Biorels will generate a blastp database from it and will use it to get additional records associated with either the Organisms you’ve selected, ChEMBL records, X-Ray records.
Pubmed: Do you want to store abstracts? Y/N. Abstracts can take around 35Gb in the database. If you want to store abstracts, type Y
Pubmed: Do you want to store citations? Y/N. Citations will take 2.5Gb.
Twice a week, Biorels will query each gene, drugs, disease, tissue to get the latest papers. Depending on your data sources, you might those questions:
Pubmed: Do you want to query pubmed for genes? Y/N
Pubmed: Do you want to query pubmed for drugs? Y/N
Pubmed: Do you want to query pubmed for disease? Y/N
Pubmed: Do you want to query pubmed for tissues? Y/N
API Key can be generated in Pubmed to enable a higher number of queries per second:
Pubmed: Please provide your Pubmed API Key (N/A if none)?
for more information: https://support.nlm.nih.gov/knowledgebase/article/KA-05317/en-us
Pubmed API Key:N/A.
Gene annotation
Based on genomes/proteomes, genes from those taxId will be considered: 9606
Do you want to consider other organisms for gene annotations?
List taxonomic Identifiers, seperated by space.
This last question aims at optimizing GENE processing which can take up to 16h if you process the whole gene collections. It will therefore limit the processing to the organisms of interest. The default will be based on your genome, proteome, and data source selection – ChEMBL, SwissProt and X-Ray. If you wish to consider more organisms, you can provide the list of NCBI Taxonomy identifiers.
Thank you for your patience! If you have done all the steps, the process will generate the CONFIG_USER file in your INSTALL/[CURRENT_DATE] directory. You can review the different files if you wish. To enable that new CONFIG_USER file, please execute the prompted command which will look very much like this:
cp ./\[DATE\]/NEW_CONFIG_USER \$TG_DIR/BACKEND/SCRIPT/CONFIG/CONFIG_USER
This command will copy your newly generated configuration file to Biorels configuration directory.
Please follow this page if you wish to configure any other cluster application