Merge pull request #19 from SlavovLab/DO-MS-DIA

release v2.0.5

README.md

@@ -2,55 +2,42 @@
 
 <u>D</u>ata-Driven <u>O</u>ptimization of <u>M</u>ass <u>S</u>pectrometry Methods
 
+![Python Package](https://github.com/SlavovLab/DO-MS-DIA/actions/workflows/python-package.yml/badge.svg)
 ![GitHub release](https://img.shields.io/github/release/SlavovLab/DO-MS.svg)
 ![GitHub](https://img.shields.io/github/license/SlavovLab/DO-MS.svg)
 
 * [Website](https://do-ms.slavovlab.net)
 * [Get started now](#getting-started)
 * [Download](https://github.com/SlavovLab/DO-MS/releases/latest)
-* [Manuscript](https://pubs.acs.org/doi/10.1021/acs.jproteome.9b00039)
+* [BioArxiv Preprint](https://www.biorxiv.org/content/10.1101/2023.02.02.526809v1)
 
-<img src="https://do-ms.slavovlab.net/assets/images/TOC_Graphic_noOutline_abstract_v18.png" width="70%">
+<img src="https://do-ms.slavovlab.net/assets/images/do-ms-dia_title_v2.png" width="70%">
 
 
 ## Getting Started
 
 Please read our detailed getting started guides:
-* [Getting started on the application](https://do-ms.slavovlab.net/docs/getting-started-application)
-* [Getting started on the command-line](https://do-ms.slavovlab.net/docs/getting-started-command-line)
+* [Getting started with DIA preprocessing](https://do-ms.slavovlab.net/docs/getting-started-preprocessing)
+* [Getting started with DIA reports](https://do-ms.slavovlab.net/docs/getting-started-dia-app)
+* [Getting started with DDA reports](https://do-ms.slavovlab.net/docs/getting-started-dda-app)
 
 ### Requirements
-
-This application has been tested on R >= 4.0.0 and R <= 4.0.2, OSX 10.14 / Windows 7/8/10. R can be downloaded from the main [R Project page](https://www.r-project.org/) or downloaded with the [RStudio Application](https://www.rstudio.com/products/rstudio/download/). All modules are maintained for MaxQuant >= 1.6.0.16 and MaxQuant < 2.0
+This application has been tested on R >= 3.5.0, OSX 10.14 / Windows 7/8/10/11. R can be downloaded from the main [R Project page](https://www.r-project.org/) or downloaded with the [RStudio Application](https://www.rstudio.com/products/rstudio/download/). All modules are maintained for MaxQuant >= 1.6.0.16 and DIA-NN > 1.8.1.
 
 The application suffers from visual glitches when displayed on unsupported older browsers (such as IE9 commonly packaged with RStudio on Windows). Please use IE >= 11, Firefox, or Chrome for the best user experience.
 
-### Installation
-
-Install this application by downloading it from the [release page](https://github.com/SlavovLab/DO-MS/releases/latest).
-
-### Running
+### Running the Interactive Application
 
 The easiest way to run the app is directly through RStudio, by opening the `DO-MS.Rproj` Rproject file
 
-<img src="https://github.com/SlavovLab/DO-MS/raw/master/docs/assets/images/do-ms-proj.png" height="100px">
+![](https://do-ms.slavovlab.net/assets/images/do-ms-proj.png){: width="70%" .center-image}
 
 and clicking the "Run App" button at the top of the application, after opening the `server.R` file. We recommend checking the "Run External" option to open the application in your default browser instead of the RStudio Viewer.
 
-<img src="https://github.com/SlavovLab/DO-MS/raw/master/docs/assets/images/do-ms-run.png" height="100px">
+![](https://do-ms.slavovlab.net/assets/images/do-ms-run.png){: width="70%" .center-image}
 
 You can also start the application by running the `start_server.R` script.
 
-### Automated Report Generation
-
-You can automatically generate PDF/HTML reports without having to launch the server by running the `do-ms_cmd.R` script, like so:
-
-```
-$ Rscript do-ms_cmd.R config_file.yaml
-```
-
-This requires a configuration file, and you can [find an example one here](https://github.com/SlavovLab/DO-MS/blob/master/example/config_file.yaml). See [Automating Report Generation](https://do-ms.slavovlab.net/docs/automation) for more details and instructions.
-
 ### Customization
 
 DO-MS is designed to be easily user-customizable for in-house proteomics workflows. Please see [Building Your Own Modules](https://do-ms.slavovlab.net/docs/build-your-own) for more details.
@@ -59,9 +46,9 @@ DO-MS is designed to be easily user-customizable for in-house proteomics workflo
 
 Please see [Hosting as a Server](https://do-ms.slavovlab.net/docs/hosting-as-server) for more details.
 
-### Search Engines Other Than MaxQuant
+### Supporting other Search Engines
 
-This application is currently maintained for MaxQuant >= 1.6.0.16. Adapting to other search engines is possible but not provided out-of-the-box. Please see [Integrating Other Search Engines ](https://do-ms.slavovlab.net/docs/other-search-engines) for more details.
+This application is currently maintained for (MaxQuant)[https://www.nature.com/articles/nbt.1511] >= 1.6.0.16 and  (DIA-NN)[https://www.nature.com/articles/s41592-019-0638-x] >= 1.8. Adapting to other search engines is possible but not provided out-of-the-box. Please see [Integrating Other Search Engines ](https://do-ms.slavovlab.net/docs/other-search-engines) for more details.
 
 ### Can I use this for Metabolomics, Lipidomics, etc... ?
 
@@ -71,13 +58,8 @@ While the base library of modules are based around bottom-up proteomics by LC-MS
 
 ## About the project
 
-<!--
-DO-MS is a project...
--->
-
 The manuscript for this tool is published at the Journal of Proteome Research: [https://pubs.acs.org/doi/10.1021/acs.jproteome.9b00039](https://pubs.acs.org/doi/10.1021/acs.jproteome.9b00039)
-
-The manuscript is also freely available on bioRxiv: [https://www.biorxiv.org/content/10.1101/512152v1](https://www.biorxiv.org/content/10.1101/512152v1).
+The manuscript for the extended version 2.0 can be found on bioArxiv: [https://www.biorxiv.org/content/10.1101/2023.02.02.526809v1](https://www.biorxiv.org/content/10.1101/2023.02.02.526809v1)
 
 Contact the authors by email: [nslavov\{at\}northeastern.edu](mailto:nslavov@northeastern.edu).
 
@@ -87,21 +69,11 @@ DO-MS is distributed by an [MIT license](https://github.com/SlavovLab/DO-MS/blob
 
 ### Contributing
 
-Please feel free to contribute to this project by opening an issue or pull request.
-
-<!--
-### Data
-All data used for the manuscript is available on [UCSD's MassIVE Repository](https://massive.ucsd.edu/ProteoSAFe/dataset.jsp?task=ed5a1ab37dc34985bbedbf3d9a945535)
--->
-
-<!--
-### Figures/Analysis
-Scripts for the figures in the DART-ID manuscript are available in a separate GitHub repository, [https://github.com/SlavovLab/DART-ID_2018](https://github.com/SlavovLab/DART-ID_2018) 
--->
+Please feel free to contribute to this project by opening an issue or pull request in the [GitHub repository](https://github.com/SlavovLab/DO-MS).
 
 -------------
 
 ## Help!
 
-For any bugs, questions, or feature requests, 
+For any bugs, questions, or feature requests,
 please use the [GitHub issue system](https://github.com/SlavovLab/DO-MS/issues) to contact the developers.

do-ms_cmd.R

@@ -244,7 +244,7 @@ for(i in 1:length(config[['load_misc_input_files']])) {
   prnt(paste0('Loading misc file: ', name))
 
   # read in as data frame (need to convert from tibble)
-  data[[name]] <- as.data.frame(read_tsv(file=path))
+  data[[name]] <- as.data.frame(read_tsv(file=path, col_types = cols()))
   # rename columns (replace whitespace or special characters with '.')
   colnames(data[[name]]) <- gsub('\\s|\\(|\\)|\\/|\\[|\\]', '.', 
                                        colnames(data[[name]]))
@@ -363,6 +363,8 @@ for(f in config[['load_input_files']]) {
 # sort the raw files
 raw_files <- sort(raw_files)
 
+
+
 
 # load naming format
 file_levels <- rep(config[['exp_name_format']], length(raw_files))
@@ -377,8 +379,12 @@ file_levels <- str_replace(file_levels, '\\%i', as.character(seq(1, length(raw_f
 # folder name is stored as the names of the raw files vector
 file_levels <- str_replace(file_levels, '\\%f', names(raw_files))
 
+
 # replace %e with the raw file name
 file_levels <- str_replace(file_levels, '\\%e', raw_files)
+print(file_levels)
+
+print(config[['exp_name_pattern']])
 
 # apply custom string extraction expression to file levels
 if(!is.null(config[['exp_name_pattern']])) {
@@ -387,11 +393,13 @@ if(!is.null(config[['exp_name_pattern']])) {
   file_levels[is.na(file_levels)] <- 'default'
 }
 
+print(file_levels)
+
 # apply custom names, as defined in the "exp_names" config field
 if(!is.null(config[['exp_names']]) & length(config[['exp_names']]) > 0) {
   file_levels[1:length(config[['exp_names']])] <- config[['exp_names']]
 }
-
+print(file_levels)
 # ensure there are no duplicate names
 # if so, then append a suffix to duplicate names to prevent refactoring errors
 
@@ -477,4 +485,4 @@ generate_report(input, f_data, raw_files, config[['output']], progress_bar=FALSE
 
 # prnt(paste0('Report written to: ', config[['output']]))
 
-prnt('Done!')
+prnt('Done!')

docs/Gemfile

@@ -1,2 +1,6 @@
 source "https://rubygems.org"
 gemspec
+
+gem "rexml", "~> 3.2"
+
+gem "webrick", "~> 1.8"

docs/_config.yml

@@ -26,17 +26,15 @@ search_enabled: true
 
 # Aux links for the upper right navigation
 aux_links:
-  "DO-MS Preprint":
-    - "https://www.biorxiv.org/content/10.1101/512152v1"
-  "JPR Article":
-    - "https://pubs.acs.org/doi/10.1021/acs.jproteome.9b00039"
+  "BioArxiv Preprint":
+    - "https://www.biorxiv.org/content/10.1101/2023.02.02.526809v1"
   "GitHub Repository":
     - "https://github.com/SlavovLab/DO-MS"
   "Slavov Lab":
     - "http://slavovlab.net"
 
 github_link: https://github.com/SlavovLab/DO-MS
-preprint_link: https://www.biorxiv.org/content/10.1101/512152v1
+preprint_link: https://www.biorxiv.org/content/10.1101/2023.02.02.526809v1
 jpr_link: https://pubs.acs.org/doi/10.1021/acs.jproteome.9b00039
 
 # Color scheme currently only supports "dark" or nil (default)

docs/docs/DO-MS_examples.md

@@ -1,7 +1,7 @@
 ---
 layout: default
 title: DO-MS Examples
-nav_order: 3
+nav_order: 5
 permalink: docs/DO-MS_examples
 parent: Getting Started
 ---
@@ -10,6 +10,13 @@ parent: Getting Started
 
 Bellow are links to example DO-MS Reports
 
+## DIA-NN DIA Reports
+1. [MS2 number optimization](https://drive.google.com/uc?id=1mNrJsk6uaI3ljtQEadr0XwR5mlyO4xZg&export=download)
+2. [Different MS2 strategies](https://drive.google.com/uc?id=1eOJXC2Zb0lmVwbsaj7Zlf6wm3SPGrbWl&export=download)
+3. [Gradient length optimization](https://drive.google.com/uc?id=1dQdo3kR-WxWT8zEeOxztxXmJE1rSMsEM&export=download)
+4. [MS1 sampling optimization](https://drive.google.com/uc?id=1YcT3al9ICu36_ornSsLZgHj1eTBBPOxb&export=download)
+5. [Single cell quality control](https://drive.google.com/uc?id=1v62qJ8JVZ_TSkUa3K0i6nWoJDbW9EQHO&export=download)
 
+## MaxQuant DDA Reports
 1. [Isobaric carrier optimization](https://scope2.slavovlab.net/mass-spec/Isobaric-carrier-optimization#do-ms-reports)
 2. [Technical improvements of SCoPE2](https://scope2.slavovlab.net/mass-spec/Increased-accuracy-and-throughput)

docs/docs/getting_started.md

@@ -8,4 +8,19 @@ has_children: true
 
 # Getting Started
 
-DO-MS can be run either from the command-line or as interactive application. Follow the links below to get started using the implementation of your choice. For more details on the data display, read the [DO-MS article](https://pubs.acs.org/doi/10.1021/acs.jproteome.9b00039).  
+DO-MS can be run either from the command-line or as interactive application. Follow the links below to get started using the implementation of your choice. For more details on the data display, read the [DO-MS 2.0 article](https://www.biorxiv.org/content/10.1101/2023.02.02.526809v1). 
+
+Before starting DO-MS the first time, the input data type has to be selected. DO-MS can work with both DDA results coming from MaxQuant as well as with DIA results coming from DIA-NN.
+The mode can be set in the config.yaml file. Open the file in R-Studio or your editor of choice.
+![]({{site.baseurl}}/assets/images/do-ms-dia_mode.png){: width="70%" .center-image}
+
+If You wish to analyze MaxQuant DDA results, change the parameter to `max_quant`, otherwise leave it as `dia-nn`. This setting needs to be changed before SO-MS is started or the R environment is initialized. It is also possible to keep both versionas in two separate folders simultanously.
+![]({{site.baseurl}}/assets/images/do-ms-dia_config.png){: width="70%" .center-image}
+
+# Generating DO-MS Reports
+Please read our detailed getting started guides:
+
+* [DIA Preprocessing]({{site.baseurl}}/docs/getting-started-preprocessing)
+* [DIA Reports using the app]({{site.baseurl}}/docs/getting-started-dia-app)
+* [DDA Reports using the app]({{site.baseurl}}/docs/getting-started-application)
+* [DDA Reports using the command line]({{site.baseurl}}/docs/getting-started-application)

docs/docs/getting_started_app.md

@@ -1,12 +1,12 @@
 ---
 layout: default
-title: Interactive Application
-nav_order: 1
-permalink: docs/getting-started-application
+title: DDA Reports in the App
+nav_order: 3
+permalink: docs/getting-started-dda-app
 parent: Getting Started
 ---
 
-# Getting Started -- Interactive DO-MS Application
+# Getting Started -- DDA Reports in the App
 
 DO-MS is an application to visualize mass-spec data both in an interactive application and static reports generated via. the command-line. In this document we'll walk you through analyzing an example dataset in the interactive application.
 
@@ -31,13 +31,8 @@ The only constraint for data in DO-MS is that it must be from MaxQuant version >
 
 ## Installation
 
-Download the application via. a zip or tar archive from the [GitHub release page](https://github.com/SlavovLab/DO-MS/releases). Unzip the archive, and then open the `DO-MS.Rproj` to load the project into RStudio
-
-![]({{site.baseurl}}/assets/images/do-ms-proj.png){: width="60%" .center-image}
-
-To start the app open `server.R` in RStudio and on the top right corner of the editor click on the "Run App" button. To run the application in your browser (preferred option) rather than in RStudio, click the dropdown menu and select "Run External".
-
-![]({{site.baseurl}}/assets/images/do-ms-run.png){: width="85%" .center-image}
+Please make sure that you installed DO-MS as descibed in the [installation]({{site.baseurl}}/docs/installation) section. 
+<span style="color:red">Note that DO-MS has to be configured for MaxQuant data. [Find out More]({{site.baseurl}}/docs/getting-started)</span> 
 
 ## Data Import
 

docs/docs/getting_started_cmd.md

@@ -1,7 +1,7 @@
 ---
 layout: default
-title: Command Line
-nav_order: 2
+title: DO-MS Command Line
+nav_order: 4
 permalink: docs/getting-started-command-line
 parent: Getting Started
 ---

docs/docs/getting_started_dia_app.md

@@ -0,0 +1,100 @@
+---
+layout: default
+title: DIA Reports in the App
+nav_order: 2
+permalink: docs/getting-started-dia-app
+parent: Getting Started
+---
+
+# Getting Started -- DIA Reports in the App
+
+DO-MS is an application to visualize mass-spec data both in an interactive application and static reports generated via. the command-line. In this document we'll walk you through analyzing an example dataset in the interactive application.
+
+**Table of Contents**
+
+1. [Example Data](#example-data)
+2. [Installation](#installation)
+3. [Data Import](#data-import)
+    1. [Adding Folders](#adding-folders)
+    2. [Renaming Experiments](#renaming-experiments)
+4. [Interacting with Modules](#interacting-with-modules)
+5. [Generate Report](#generate-report)
+
+
+
+## Example Data
+
+We have provided an example data set online, which contains parts of the MS2 number optimization in the paper. This data can also be obtained by follwoing the guide on the preprocessing or can be downloaded as a zip file here: [https://drive.google.com/file/d/1BzWVKghIThtgYItgGy9vt6M214mHB74q/view?usp=share_link](https://drive.google.com/file/d/1BzWVKghIThtgYItgGy9vt6M214mHB74q/view?usp=share_link).
+
+
+![]({{site.baseurl}}/assets/images/do-ms-dia-example.png){: width="70%" .center-image}
+
+## Installation
+
+Please make sure that you installed DO-MS as descibed in the [installation]({{site.baseurl}}/docs/installation) section. 
+
+
+## Data Import
+<span style="color:red">Make sure DO-MS has been configured for DIA-NN data. [Find out More]({{site.baseurl}}/docs/getting-started)</span> 
+
+![]({{site.baseurl}}/assets/images/do-ms-dia-overview.png){: width="90%" .center-image}
+
+### Adding Folders
+
+DO-MS is designed to load _folders_ of analyses rather than individual files. To allow quick access to your analyses, as well as to allow analyzing multiple searches simultaneously, DO-MS provides a searchable "folder table" for all of your analyses. We will start loading the folder with the example output.
+
+Start by clicking the "Add Folder" button at the top of the table
+
+![]({{site.baseurl}}/assets/images/do-ms-dia-import.png){: width="70%" .center-image}
+
+Then add the path of your folder into the textbox, as shown:
+
+![]({{site.baseurl}}/assets/images/do-ms-dia-foldername.png){: width="70%" .center-image}
+
+A folder path is the folder's absolute location on your machine. On Windows, you can get the folder path by navigating to it in Explorer, clicking on the top file path bar, and copying the resulting text with Ctrl+C. On Mac/OSX, you can get the folder path by right clicking on the folder while presing the 'option' key.
+
+![]({{site.baseurl}}/assets/images/do-ms-dia-pathname.png){: width="70%" .center-image}
+
+Note that in the example above we checked "Add Single Folder" to add just the folder path we pasted in. If for example, you have a folder that contains many MaxQuant searches, you can select "Add Child Folders" to add all subfolders of the path specified, or "Add Recursively" to add _all_ folders that are below the path specified.
+
+Click "Confirm" and now you should see your folder added to the folder table. When you have multiple folders loaded into the table you can select more than one.
+
+![]({{site.baseurl}}/assets/images/do-ms-dia-load.png){: width="70%" .center-image}
+
+Finally, load the files from the selected folders by scrolling down and clicking on the big "Load Data" button
+
+### Renaming Experiments
+
+An important aspect of data visualization is easy-to-read labels for your experiments. DO-MS provides an accessible interface for renaming your raw file names so that generated figures will be easier to interpret.
+
+Once your data is loaded, scroll down to the "Renaming Experiments" section. Here you will find a table of your loaded raw files and their associated "labels" that will be used when plotting. Double click a label and type enter the new labe:
+
+![]({{site.baseurl}}/assets/images/do-ms-dia-rename.png){: width="100%" .center-image}
+
+Please rename all experiments as following:
+
+```
+wGW027 => 4 MS2
+wGW028 => 8 MS2
+wGW029 => 10 MS2
+```
+
+## Interacting with Modules
+
+Each module has a short description that can be accessed by hovering over the question mark icon next to the module title.
+
+![]({{site.baseurl}}/assets/images/do-ms-dia-info.png){: width="70%" .center-image}
+
+You can also download the module plot as a PNG or PDF, by clicking on the download buttons below each module plot. You can also download the underlying data used for the plot by clicking on the "Download Data" button. This tab-delimited file can be imported into many other visualization packages.
+
+## Generate Report
+
+Click on "Generate Report" in the sidebar to access the report generation page.
+
+![]({{site.baseurl}}/assets/images/do-ms-dia-generate-report.png){: width="70%" .center-image}
+
+Here you will find some options to customize your report. While we support PDF reports and PDF images, we strongly recommend that you generate your reports in HTML format with PNG image plots. Other configurations may result in graphical glitches or unwanted behavior. In addition we recommend that you check your `pandoc` installation ([more details here]({{site.baseurl}}/docs/known-issues#pandoc-not-found)) as any issues will prevent the report generation.
+
+Click the "Download Report" button to begin generating the report. This takes a while as all plots have to be remade. A progress bar at the bottom of the page informs you of the progress.
+
+All images in the report are embedded in the markup, so feel free to share this single file to your colleagues/collaborators and don't worry about having to include anything else.