BioRels Infrastructure in depth
While Biorels is processing the data sources you want, please take some time to understand a little bit more about how the infrastructure works, from its infrastructure to the job definitions.
For now, we are going to focus on the architecture of the backend infrastructure. There are 4 directories at the root of Biorels.
| Directory | Description |
|---|---|
| BACKEND | All the backend data logistic |
| DOCS | Provides the licenses for 3rd party tools and data sources as well as this documentation. |
| PRD_DATA | Aliases to the latest version of each data source |
| PROCESS | Where all data processing happens. 1 directory per data source |
| PRIVATE_PROCESS | Where all the private data process happens. 1 directory per data source |
| WEBJOBS | Where executed web jobs can run |
| WEBSITE | Website files |
The BACKEND directory contains all necessary scripts and configuration files to run the backend infrastructure. Please take a look at the Table 2 below listing the roles of the different directories.
| Directory | Description |
|---|---|
| BACKEND | All the backend data logistic |
| |=> CONTAINER | Configuration and environment file to create backend container scripts |
| |=> DEVELOP | Experimental! Script to generate new data source scripts |
| |=> INSTALL | SQL files to create schemas and tables |
| |=> LOG | LOG directory providing a status for each script |
| |=> SGE_LOG | Standard out and error logs for each submitted job |
| |=> MONITOR | Monitoring files: job running/timestamps |
| |=> PRIVATE_SCRIPT | Directory of all your personal scripts, not to be shared |
| |=> SCRIPT | Contains all the individual job scripts and necessary libraries |
| |=> SCRIPT/LIB | Contains php library files |
| |=> SCRIPT/LIB_PYTHON | Contains python library files |
| |=> SCRIPT/CONFIG | Configuration files |
| |=> SCRIPT/SHELL | Shell wrapper for each script file |
| |=> SCRIPT/API | API function calls |
| |=> SCRIPT/BIORJ | Import/export |
| |=> SCRIPT/monitor_job.php | Master script |
| |=> CONTAINER_SHELL | Shell wrapper for each script, to be run using the container |
| |=>SRC | Source code for some of the tools |
| |=> STATIC_DATA | Predefined data or sets of rule to help in data processing |
Table 2 - Directory structure for Backend infrastructure
Let’s break it down a bit.
-
Monitor_job.php is the conductor of the overall machinery.
-
CONTAINER contains the configuration files to build the container, which has all the tools and libraries required to execute the scripts
-
INSTALL has all the installation and configuration scripts
-
DEVELOP will generate all the files and configuration you’ll need to add a new data source. It’s however experimental
-
STATIC_DATA possess all the static files, i.e. some ruleset or additional mapping information.
-
MONITOR contains the list of running scripts so that if you stop monitor_jobs, it knows what was running
-
CONFIG holds the core of the machinery, your configuration file and the jobs configuration
-
SCRIPT/PRIVATE_SCRIPT contains the processing scripts
-
CONTAINER_SHELL will have scripts that are container wrappers of the shell scripts located in SCRIPTS/SHELL
-
LOG will have:
-
MONITOR files that is a snapshot of the decision making to run a job as defined by monitor_jobs
-
SGE_LOG: log files of the actual scripts.
-
SCRIPTS is the key scientific directory since all the scientific data processing are defined in this directory. Some core directories that are part of Biorels engine, as listed in Table 3. In addition, each data source will have a directory under SCRIPTS, that will contain one to many processing scripts.
| Directory | Description |
|---|---|
| API | Scripts for API Database query. See API Section |
| BIORJ | Scripts for Biorj import/export |
| CONFIG | Configuration files. See Configuration files section |
| LIB | PHP Library files with functions to enable the processing scripts |
| LIB_PYTHON | Python Library files with functions to enable the processing scripts |
| SHELL | Shell wrapper for each processing script. Includes setenv.sh |
| WEBJOBS | Specific web jobs that can be run in the backend. See Webjob section |
Table 4 – List of SCRIPTS directories that are part of Biorels engine
PROCESS directory will be the data core. Every file where being downloaded, being generated or copied over will be saved under PROCESS. Similarly, to SCRIPTS, each data source will have its own directory. Under each data source directory will be all the different versions of that resource, each being stored in a unique subdirectory named either after the version or the date of the download.
For NCBI GENE for instance, the directories will look as such:
$TG_DIR/PROCESS/GENE/2023-11-29/
$TG_DIR/PROCESS/GENE/2023-11-30/
The name of the data source will be defined in red, right under PROCESS, while the date will be created under gene to note the version – in orange.
The same will apply for private scripts where files will be stored in PRIVATE_PROCESS
Different customers have different needs, and some of them wants to access to the latest version of a data source easily. PRD_DATA answers such a need. For each data source, an alias of the same name as the data source will be created – with a link to the latest version of the data. An example of a PRD_DATA content by executing ls -l command:
BIOASSAY_ONTO -> $TG_DIR/PROCESS/BIOASSAY_ONTO/2022-11-16
CLINVAR -> $TG_DIR/PROCESS/CLINVAR/2023-11-18
EFO -> $TG_DIR/PROCESS/EFO/2023-11-18
GO -> $TG_DIR/PROCESS/GO/2023-11-22
REACTOME -> $TG_DIR/PROCESS/REACTOME/2023-09-14
SWISSLIPIDS -> $TG_DIR/PROCESS/SWISSLIPIDS/2023-10-12
TRANSLATE -> $TG_DIR/PROCESS/TRANSLATE/2023-10-16
UBERON -> $TG_DIR/PROCESS/UBERON/2023-10-28
UNIPROT -> $TG_DIR/PROCESS/UNIPROT/2023-09-14
We can see in the example above 9 symbolic links representing the latest version for 9 data sources.
The core of Biorels configuration resides in $TG_DIR/BACKEND/SCRIPT/CONFIG and is made of 3 files:
-
CONFIG_GLOBAL: The file contains global variables, such as different directory and tool path names.
-
CONFIG_JOB: This file lists the different processes and how/when they can run.
-
CONFIG_USER: This defines the different rules the user has provided during Biorels configuration – Data sources, Genomes, Proteomes step.
BACKEND/SCRIPT/CONFIG/CONFIG_GLOBAL contains all the different links, path, parameters, and options that Biorels requires to properly function. Many of them don’t need to be modified unless there is some critical conflict with your environment. However, a few of them should be reviewed before running the system. 3 types of variables are defined in that file: GLOB for global variables, LINK for external links, and TOOL, which can be used to define both tools path and the parameters of those tools.
Below are the different global variables located in BACKEND/SCRIPT/CONFIG/CONFIG_GLOBAL. None of it should be modified by the user during configuration. The format of the Global variable line is as follow:
| Column 1 | Column 2 | Column 3 |
Column 4 (optional) |
|---|---|---|---|
| GLOB | Link name | Link path | Description |
Note: The number of tabs between column does not matter as long as there are at least 3 values – to make it more readable.
| Variable name | Description |
|---|---|
| CORE BIORELS CONFIGURATION | |
| PROCESS_DIR | Directory in which downloaded data will be stored – shouldn’t be modified |
| PRIVATE_PROCESS_DIR | Directory in which private data will be stored |
| PRIVATE_SCRIPT_DIR | Directory where all private scripts are stored – shouldn’t be modified |
| LOG_DIR | Log directory– shouldn’t be modified |
| MONITOR_DIR | Monitoring files directory– shouldn’t be modified |
| SCRIPT_DIR | Script directory – shouldn’t be modified |
| STATIC_DIR | Static data – shouldn’t be modified |
| PRD_DIR | Location of symbolic links pointing to the latest version of each data |
| SRC_DIR | Source code directory -shouldn’t be modified |
| TIMESTAMP | Name of the timestamp file – shouldn’t be modified |
| JOBARRAY | File name to submit job arrays – do not modify |
Table 5 - Global variables
To simplify eventual changes in external resources path, all http(s) paths are defined in the CONFIG_GLOBAL file as a LINK. A LINK record is composed of 3 values, separated by tabs. For the installation process, you shouldn’t change anything.
Note: The number of tabs does not matter as long as there are 3 values – to make it more readable.
| Column 1 | Column 2 | Column 3 |
|---|---|---|
| LINK | Link name | Link path |
All tools are built and shipped in the singularity container. The location of each tool within the container is defined in the CONFIG_GLOBAL in the TOOL section. A TOOL record is composed of 3 values, separated by tabs and provides the path for one application. For the installation process, you shouldn’t change anything. In addition, parameters to different tools can be specified in this section. In this case, the name should have the suffix _PARAM
Note: The number of tabs does not matter as long as there are 3 values – to make it more readable.
| Column 1 | Column 2 | Column 3 |
|---|---|---|
| TOOL | tool name | tool path within the container |
| TOOL | TOOLNAME_PARAM | List of parameters |
If you want to add a tool, we suggest you follow the documentation on developing biorels [SECTION]
CONFIG_JOB is the heart of biorels. It defines the numerous jobs, when they are triggered, if they are triggered and what are the dependencies. Please review the table below describing the different columns in CONFIG_JOB.
Note: The number of tabs between column does not matter as long as there are exactly 13 values – to make it more readable. A value of 0 is equivalent in PHP to nothing, so it will be ignored – reason with Nothing is defined by a value of -1.
| Column | Name | Description |
|---|---|---|
| 1 | SC | Header tag to know it’s a script |
| 2 | JOB_ID |
Unique numeric identifier for the job Private job are prefixed with P |
| 3 | JOB_NAME | Name of the job. Must be the same as the job filename |
| 4 | REQUIRED |
List of job identifiers, separated by |, that are required to be updated prior to trigger this job if they are enabled -1 if no required job |
| 5 | REQUIRED_TRIGGER | List of job identifiers, separated by |, that would trigger the job if the REQUIRED jobs and the REQUIRED_UPDATED jobs have been run successfully |
| 6 | REQUIRED_UPDATED |
List of job identifiers, separated by |, that are required to be successfully run at least once -1 if no parent job |
| 7 | DIRECTORY | Working directory name in PROCESS/. Must be the same name as the data source |
| 8 | REQUIREMENT |
Type of requirement, based on parent jobs: C: All parent jobs must be updated (Complete) A: Any parent jobs must be updated to trigger the updated D: All parent jobs that are NOT disabled must be updated |
| 9 | JOB_TYPE |
D: Processing job P: Moving to PRD job |
| 10 | UPDATE_FREQUENCY |
For jobs with no parent – defines a frequency:
For jobs with parent:
|
| 11 | JOB_RUNTYPE |
S: Script R: Runtime |
| 12 | CONCURRENT | List of jobs Identifiers separated by |. If any of such jobs are running, this job will wait. |
| 13 | MEMORY |
-1 if no extra memory necessary Otherwise value in Mb |
| 14 | DESCRIPTION | Simple textual description of the job purpose. |
To better explain this, we will look at 2 data sources, taxonomy and gene. Below is a snapshot of the CONFIG_JOB in which we have defined 7 scripts, 1 related to taxonomy and 6 for genes.
In the example above, we are covering two data sources, TAXONOMY and GENE – which we usually define in the directory name for convention. TAXONOMY (column 6) is managed by 1 script, called wh_taxonomy (column 3). From a scientific perspective, the taxonomy is on the top of the scientific description, since all genomic information is ultimately associated to a given taxon. Therefore, there is no parent dependency (column 4 & 5). We want to maintain the data updated frequently, so we make sure it’s enabled (column 10) set the update to be triggered 10 minutes past midnight (column 9). It is a regular script (column 11) with no extra memory required (column 13). We will come back on the CONCURRENT column after explaining the gene.
We also want to cover gene information, and as such, we make sure that all GENE scripts are enabled (column 10). The first script dl_gene, will download all gene-related files. Since it’s just downloading files, there is no dependency on other scripts (column 4 & 5) and we can trigger it every day at 10 minutes past midnight (column 9) to keep our database up to date. So far, we have been looking at scripts with no dependencies (parents). However, db_gene – which process gene information and stores it in the database – will require to have the gene-related files (from dl_gene) as well as the taxonomy fully loaded in the database (from wh_taxonomy) – before it can run. Thus, db_gene will have two parents: wh_taxonomy and dl_gene, with job id 1 and 10 respectively (Column 4). The frequency column (column 9) is changed to P (for Parents), to make sure it will be triggered only when both wh_taxonomy and dl_gene have been successfully run. In the even where there is a failure in downloading the gene files or processing taxons, db_gene will not be triggered.
Concurrent jobs: We define here as concurrent jobs any job that will modify the content of tables in the database that either the job of interest will modify or will rely upon. In the case of TAXONOMY and GENE data, we have a clear relationship between the two, since a gene is defined in a given taxon. Thus, if we decide to modify the taxon table while modifying the gene table, there is a possibility of conflict where either a taxon is missing, deleted or replaced and therefore a gene couldn’t be mapped to a taxon. For this reason, db_gene (job id 11) is concurrent for wh_taxonomy (job id 1, column 12) and wh_taxonomy is concurrent of db_gene. This imply that if db_gene is running, wh_taxonomy cannot run. Similarly, if wh_taxonomy is running, db_gene cannot run. However, the latter situation is redundant with the db_gene parent requirement. This additional rule exists in which db_gene parent requirements are met AND it has been more than 24h that wh_taxonomy hasn’t been run. This situation would happen in the case of a reboot.
Once db_gene has completed successfully, db_gene_syn and db_orthologs will be triggered since their only parent script is db_gene. They will run in parallel. Once db_gene_syn is completed, db_mv_gene will be run. At last, once db_mv_gene and db_orthologs are successfully completed, prd_gene will be executed.
CONFIG_USER is the configuration file generated during the Biorels configuration – Data sources, Genomes, Proteomes step. It provides all the information about high level options, genomes, proteomes and jobs to process.
Note: You will need to stop and re-run monitor_jobs to enable those changes. See Running Biorels
EMAIL and EMAIL_FROM are respectively email addresses for Biorels to communicate to and from, respectively.
To minimize the risk of collision with other applications, Biorels offers the possibility to specify a prefix to use for all the jobs Biorels will submit. Changing JOB_PREFIX will update all future job submission. You can specify the prefix for jobs submitted from the website with WEBJOB_PREFIX.
Warning: Please ensure that no jobs or web jobs are currently running, otherwise Biorels will lose track of them while they are still running, will resubmit them which will create collisions.
Warning: Do not use the same job prefix and webjob prefix.
A few additional configurations are possible:
-
WEBJOB_LIMIT: Maximal number of web jobs that can run simultaneously. Default: 20
-
CHECK_ITER: Time in seconds in which the system will run a check when no jobs are running. Default: 3600
-
CHECK_RUN: Time in seconds in which the system will run a check when some jobs are running
-
KEEP_PREVIOUS_DATA: T if you want to keep and archive the previous versions of each data source. F to delete them
Once one parameter exists in CONFIG_USER to enable private processes. It is used as an additional safeguard. Please set PRIVATE_ENABLED to T if you want to use private processes, F otherwise. It is strongly recommended to review the PRIVATE_SECTION.
UniProt offers access to different files depending on your needs. Swiss-Prot encompass all the UniProt records that have been reviewed, while TrEMBL provides all the records that haven’t been reviewed and have been programmatically annotated. Uniprot also offers proteomes specific to given organism. To ensure we offers the same level of granularity, BioRels allow users to configure Uniprot in a similar fashion. If you wish to cover only specific species, please refer to Species configuration section. Additionally, you can select whether you want to consider SwissProt, TrEMBL or both.
If you wish to enable SwissProt, set WITH_UNIPROT_SP to T. F otherwise
If you wish to enable TrEMBL, set WITH_UNIPROT_TREMBL to T. F otherwise.
Process behavior: If you have requested to process some type of Uniprot data, dl_swissprot and dl_trembl will be enabled. However, if WITH_UNIPROT_SP or WITH_UNIPROT_TREMBL are set to F, dl_swissprot and dl_trembl will stop immediately and send a SUCCESS status, allowing the rest to proceed.
Important note: TrEMBL is too big for Biorels to proceed. Setting up WITH_UNIPROT_TREMBL will download TrEMBL data from UniProt and generate a blastp database. If you have requested some organism’s proteomes, it will also add TrEMBL records from those organisms that are not in the proteomes. In addition, if you need specific TrEMBL entries from ChEMBL or X-Ray, it will extract them from the file rather than downloading them.
How to choose?
-
If you work on a specific organism, the proteome will be enough and you will not need SwissProt or TrEMBL.
-
If you work on multiple organisms – comparing them, the proteomes will be enough
-
If you work on a specific target and wants to look at more distant information, SwissProt should be considered
-
TrEMBL should only be considered for large entities, companies or university’s platforms to wants to provide broader access and capabilities.
Some data sources will require licenses or API Key. 2 data sources are currently considered in Biorels: DrugBank and OMIM. OMIM will require an API Key, which you can set with OMIM_API_KEY. DrugBank will require a login/password, that can be set with DRUGBANK_LOGIN. N/A value can be set if you don’t have login/key information.
API Key:
Biorels performs a lot of queries on Pubmed to search for genes, drugs, clinical trials, diseases, tissues. NCBI Pubmed API is limited by default at 3 queries per second. However, They offer the possibility to increase that number to 10 queries by second with an API Key. To request an API key – which is free, please follow this link: https://ncbiinsights.ncbi.nlm.nih.gov/2017/11/02/new-api-keys-for-the-e-utilities/. Once NCBI generates the NCBI API Key, you can set PUBMED_API_ID value to this API Key. Otherwise, please set it to N/A
Abstract:
Pubmed abstracts provides a lot of useful information when looking at a paper. However, it also takes a lot of disk space – 33Gb – without being additionally processed in Biorels. Therefore, Biorels has the option to not store abstracts in the database: PUBLI_W_ABSTRACT. Set it to Y if you wish to store them, N otherwise.
Citations:
In addition to abstracts, Pubmed provides citations recorded in publications. Biorels has the option to not store such citations in the database: PUBLI_W_CITATIONS. Set it to Y if you wish to store them, N otherwise.
You can decide how many nucleotides on the 5’ end you want to consider to be the promoter region. PROMOTER_RANGE defines that range and by default it is set to 5000. Not currently in use
According to UniProt definition, “A proteome is the set of proteins thought to be expressed by an organism. The majority of the UniProt proteomes are based on the translation of a completely sequenced genome, and will normally include sequences that derive from extra-chromosomal elements such as plasmids or organellar genomes in organisms where these occur. Some proteomes may also include protein sequences based on high quality cDNAs that cannot be mapped to the current genome assembly due to sequencing errors or gaps. These are only included in the proteome following manual review of the supporting evidence, including careful analysis of homologous sequences from closely related organisms.” (Source: https://www.uniprot.org/help/proteome).
You can select one or multiple proteomes to be processed in Biorels. Uniprot reference file for proteomes is defined in CONFIG_GLOBAL > LINK > FTP_UNIPROTEOME. If you wish to add a proteome, download the UniProt proteome file and select your proteome of interest. Then, copy paste the line between #[PROTEOME] and $[/PROTEOME] sections. Add PROTEOME\t at the beginning of the line.
Here is the description of the different columns in the Proteome section:
-
Column 1: PROTEOME tag
-
Column 2: Uniprot Proteome ID
-
Column 3: TaxID
-
Column 4: OSCODE
-
Column 5: SUPERREGNUM
-
Column 6: Number of entries in main fasta (canonical)
-
Column 7: Number of entries in additional fasta (isoforms)
-
Column 8: Number of entries in gene2acc mapping file
-
Column 9: Species name
Biorels allows to process the genomes and transcriptome of different organisms, provided by RefSeq or Ensembl. The Genome section in CONFIG_USER allows you to define such genomes. It is not recommended to modify that section by yourself. Instead use the Genome selection process of the installer.
For this configuration item, we consider all information related to a gene: gene symbol, identifier, name, synonyms, chromosomal and cytogenetic location – as provided by NCBI Gene.
Considering all genes from all organisms greatly increase computing time, most of that time is being used for transient genes, such as LOC genes. By default, Biorels will optimize the list of processed genes based on the genomes and proteomes requested. However, you have additional configuration rules you can modify.
Uniprot: If you enable Uniprot as a data source, you have the possibility to process the genes from all organisms defined in SwissProt or TrEMBL, depending on your choice. This is in addition to the defined proteomes. To enable such rule, set UNIPROT_GENE to Y, N otherwise.
ChEMBL: If you enable ChEMBL as a data source, you have the possibility to process the genes from all organisms defined in ChEMBL. To enable such rule, set CHEMBL_GENE to Y, N otherwise.
Additional taxons: If you wish to add specific organisms that are not defined by a proteome or a genome, you can add it in the TAXON_LIMIT parameter. Each taxon is defined by its NCBI taxonomy identifier and must be separated by a pipe (|). If you have chosen a proteome or a genome, TAXON_LIMIT will already be populated with the corresponding NCBI taxonomic identifiers for the corresponding organisms. To consider all possible genes of all possible organisms, please set TAXON_LIMIT to N/A – WARNING – This is computationally expensive!!
Finding the NCBI Taxonomy identifier: To consider all genes from a given list of organisms, you will need to provide the list of Taxonomic Identifiers – as provided by NCBI Taxonomy – separated by |. To retrieve a Taxonomic Identifier, you can follow this link as an example: https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi?mode=Info&id=9606 – which is related to Homo Sapiens. Type in the organism name in the search box (Red arrow in Figure 1- NCBI Taxonomy website), click Go. The Taxonomy ID will be the value of interest here (Green arrow in Figure 1- NCBI Taxonomy website).
Biorels offers the possibility to have a private schema, script and process directories. The idea behind it was two-fold. First, we want to have the possibility to compare public data with public+internal data using the same processes and visualization tools. This allows on the website to show either one depending on their security access. We also wanted to have the possibility to create internal workflows that are specific to our business needs. Unless you want such capabilities, which imply some degree of duplication in the data, using the “public” version will be good enough since your data will not be published, and you can fork the repository for your own needs.
How to decide whether you need the private schema and section then?
-
if you have a lot of internal data and you want to able to compare private vs private+public.
-
If you want different security levels. The public schema for the public data only while the private schema is used for public + internal data with a more restricted access.
To enable the private schema, you should have already set a schema name in setenv.sh. If not, please review the Environment variables – setenv.sh section. In addition, you should have run the runInstall.php to create the tables in the private schema - see Database Installation. Be aware that this will delete the content of the public and the private schema. Once done, you can change PRIVATE_ENABLED to T in CONFIG_USER