System Requirements

This page describes the software and hardware requirements for the computing system (workstation or server) to run ExSeq processing pipeline.

Software requirements

To run ExSeq processing pipeline, the following OS and software are required:

• Linux: A Linux OS is required. ExSeq assumes Linux paths and tools are available. Testing has been done on CentOS 7.
• Bash: The main pipeline script which launches the other MATLAB scripts uses bash (runPipeline.sh).
• MATLAB: Most of the functionality of ExSeq is contained in MATLAB scripts and is required.
• (optional) CUDA: Allows for using NVIDIA GPUs to accelerate the pipeline. This is not required to run the pipeline, but, if available, it can be used to significantly reduce the time taken to run.
• (optional) Python: Is used for setting parameters in a GUI or an interactive command line interface. It is also necessary for using a performance profile option of the pipeline. Python major versions 2 and 3 are supported. The pipeline can run without Python if the performance profile option is not used. The parameters can also be set, without Python, by pipeline script options or by setting them in a parameters file.
• (optional) Jupyter, numpy, pandas, matplotlib: These are additional Python tools and libraries necessary for displaying the results of the performance profile option but are not used elsewhere.

Hardware requirements

While there are no strict requirements for the computer specifications, a high-performance system is recommended in order to run ExSeq pipeline for a large data set (100 GB - 10 TB) in a reasonable time. The following are typical configurations.

• CPU: 20 cores at 2.0+ GHz
  While the pipeline runs on any number of cores, it is highly parallelized and benefits from more cores.
• Main memory (RAM): 192 GB
  RAM requirements depend on the size of the input data set as well as the degree of parallelization used. Parallelization reduces the time taken by the pipeline but requires having more memory. The pipeline will attempt to automatically set parallelization parameters to use most (with some overhead) of the system's memory, but they can also be set manually in a parameters file.
• Data storage (HDD/SSD): 10-20 TB
  Disk usage also depends on the input data size. The color correction and registration stages create full copies of the data and normalization stage creates one file per round. These stages also optionally create downsampled versions further increasing the size. While those intermediate files may be deleted once final results are obtained, at least temporarily they increase disk usage to roughly 3-4 times the original size after running the pipeline. Using fast drives (SSD) in place of HDD is moderately beneficial. If more space is needed to handle more/larger data sets, the data storage may be implemented as network attached storage (NAS) connected through no less than 10 Gb Ethernet.
• (optional) GPU: 2 of NVIDIA GeForce GTX 1080 Ti or similar
  The pipeline runs significantly faster on CUDA-enabled GPUs. Any number of GPUs may be used, and more GPUs (up to around 4) with more GPU memory (1080 Ti has 11 GB) lead to a shorter runtime. If no GPUs are present in the system, the pipeline runs entirely on the CPUs. It is important to note that some steps of the pipeline require using temporary storage when using GPUs. If the temporary storage is on HDD or other slow storage, the GPU version can be slower than the CPU version of the pipeline.
• (optional) NVMe SSDs: 2 of 1-2 TB drives bundled together in RAID0 configuration
  The pipeline runs on GPUs even faster if NVMe SSDs are present so they work as a high-speed, high-capacity data buffer to/from GPUs. Slower drives (SATA SSDs) are unlikely to help for this purpose.