Merge pull request #168 from scipion-em/devel

Devel

README.rst

@@ -44,8 +44,6 @@ You should also consider having VMD on your system for visualization.
 We assume that VMD is installed on your system in "/usr/local/lib/vmd".
 If VMD is installed but does not work, you may run the command "scipion3 config" and look for VMD_HOME in the config file (the config file is usually at ~/scipion3/config/scipion.conf)
 
-Note: GENESIS is not installed by default in continuousflex. To install GENESIS, you can use the Plugin Manager, or run the command line "scipion3 installb MD-NMMD-Genesis-1.0"
-
 Note: Matlab with its image processing toolbox is optional. It will only be needed if missing-wedge correction using Monte Carlo or volume denoising using BM4D are to be used
 We assume that Matlab is installed on your system in "~/programs/Matlab".
 If Matlab is installed but does not work, you may run the command "scipion3 config" and look for MATLAB_HOME in the config file (the config file is usually at ~/scipion3/config/scipion.conf)
@@ -58,17 +56,18 @@ versions > 3.3.0
 Protocols
 ---------
 
-* HEMNMA: Hybrid Electron Microscopy Normal Mode Analysis method to interpret heterogeneity of a set of single particle cryo-EM images in terms of continuous macromolecular conformational transitions, based on normal mode analysis [1-3]
-* StructMap: Structural Mapping method to interpret heterogeneity of a set of single particle cryo-EM maps in terms of continuous conformational transitions, based on normal mode analysis [4]
-* HEMNMA-3D: Extension of HEMNMA to continuous conformational variability analysis of macromolecules in cryo-ET subtomograms (in vitro and in situ) [5]
-* TomoFlow: Method for analyzing continuous conformational variability of macromolecules in cryo-ET subtomograms (in vitro and in situ) based on 3D dense optical flow [6]
-* NMMD: Software to perform cryo-EM flexible fitting using a combination of Normal Mode (NM) analysis and Molecular Dynamics (MD) simulations  implemented in GENESIS [7]
-* DeepHEMNMA: A deep learning extension of HEMNMA  [8]
+* **HEMNMA**: Hybrid Electron Microscopy Normal Mode Analysis method to interpret heterogeneity of a set of single particle cryo-EM images in terms of continuous macromolecular conformational transitions, based on normal mode analysis [1-3]
+* **StructMap**: Structural Mapping method to interpret heterogeneity of a set of single particle cryo-EM maps in terms of continuous conformational transitions, based on normal mode analysis [4]
+* **HEMNMA-3D**: Extension of HEMNMA to continuous conformational variability analysis of macromolecules in cryo-ET subtomograms (in vitro and in situ) [5]
+* **TomoFlow**: Method for analyzing continuous conformational variability of macromolecules in cryo-ET subtomograms (in vitro and in situ) based on 3D dense optical flow [6]
+* **NMMD**: Software to perform cryo-EM flexible fitting using a combination of Normal Mode (NM) analysis and Molecular Dynamics (MD) simulations  implemented in GENESIS [7]
+* **DeepHEMNMA**: A deep learning extension of HEMNMA  [8]
+* **MDSPACE**: Approach for extracting atomic-resolution landscapes of continuous conformational variability of biomolecular complexes from cryo electron microscopy (cryo-EM) single particle images based on a new 3D-to-2D flexible fitting method, which uses molecular dynamics (MD) simulation and is embedded in an iterative conformational-landscape refinement scheme. [11]
 
 Notes:
 
 * The plugin additionally provides the test data and automated tests of the protocols in Scipion 3. The following two types of tests of HEMNMA and HEMNMA-3D can be produced by running, in the terminal, "scipion3 tests continuousflex.tests.test_workflow_HEMNMA" and “scipion3 tests continuousflex.tests.test_workflow_HEMNMA3D”, respectively: (1) tests of the entire protocol with the flexible references coming from an atomic structure and from an EM map; and (2) test of the alignment module (test run using 5 MPI threads). The automated tests of the TomoFlow method are also available and can be run using scipion3 tests continuousflex.tests.test_workflow_TomoFlow. 
-* GENESIS is not installed by default in continuousflex. To install GENESIS, go to the plugin manager and, under continuousflex plugin, and check install GENESIS. The automated tests of GENESIS provide an example of cryo-EM flexible fitting of an atomic model into a 3D density map using NMMD for CHARMM and C-Alpha Go model. The tests can be produced by running "scipion3 tests continuousflex.tests.test_workflow_GENESIS" (you need at least 2 MPI cores for these tests).
+* The automated tests of GENESIS provide an example of cryo-EM flexible fitting of an atomic model into a 3D density map using NMMD for CHARMM and C-Alpha Go model. The tests can be produced by running "scipion3 tests continuousflex.tests.test_workflow_GENESIS" (you need at least 2 MPI cores for these tests).
 * HEMNMA additionally provides tools for synthesizing noisy and CTF-affected single particle cryo-EM images with flexible or rigid biomolecular conformations, for several types of conformational distributions, from a given atomic structure or an EM map. One part of the noise is applied on the ideal projections before and the other after the CTF, as described in [9-10].
 * HEMNMA-3D additionally provides tools for synthesizing noisy, CTF and missing wedge affected cryo-ET tomograms and single particle subtomograms with flexible or rigid biomolecular conformations, for several types of conformational distributions, from a given atomic structure or an EM map. One part of the noise is applied on the ideal projections before and the other after the CTF, as described in [9-10].
 * A reproduction of some utility codes with their corresponding licenses are contained in this plugin for subtomogram averaging, missing wedge correction, denoising and data reading. These codes are not used in the methods above, but they are made optional for data preprocessing and visualization.
@@ -98,6 +97,8 @@ References
 
 [10] Jonic S, Sorzano CO, Thevenaz P, El-Bez C, De Carlo S, Unser M: Spline-based image-to-volume registration for three-dimensional electron microscopy. Ultramicroscopy 2005, 103:303-317. `[Journal] <https://www.sciencedirect.com/science/article/pii/S0304399105000173>`__
 
+[11] Vuillemot R, Mirzaei A, Harastani M, Hamitouche I, Fréchin L, Klaholz BP, Miyashita O, Tama F, Rouiller I, Jonic S. MDSPACE: Extracting continuous conformational landscapes from cryo-EM single particle datasets using 3D-to-2D flexible fitting based on Molecular Dynamics simulation. Journal of Molecular Biology. 2023 Jan 10:167951. `[Journal] <https://www.sciencedirect.com/science/article/abs/pii/S0022283623000074>`__
+
 Citation
 ----------
 Harastani, M., Vuillemot, R., Hamitouche, I., Moghadam, N. B., & Jonic, S. (2022). ContinuousFlex: Software package for analyzing continuous conformational variability of macromolecules in cryo electron microscopy and tomography data. Journal of Structural Biology, 214(4), 107906. `[Journal] <https://doi.org/10.1016/j.jsb.2022.107906>`__

continuousflex/__init__.py

@@ -35,6 +35,7 @@
 import re
 import pyworkflow.utils as pwutils
 
+
 _logo = "logo.png"
 
 MD_NMMD_GENESIS_VERSION = "1.1"
@@ -43,7 +44,7 @@
 # Use this general activation variable when installed outside Scipion
 MODEL_CONTINUOUSFLEX_ACTIVATION_VAR = "MODEL_CONTINUOUSFLEX_ACTIVATION"
 
-__version__ = "3.3.13"
+__version__ = "3.3.14"
 
 
 class Plugin(pwem.Plugin):
@@ -135,7 +136,6 @@ def getCondaInstallation(version, txtfile):
             FFLAGS = "-fallow-argument-mismatch -ffree-line-length-none"
         else:
             FFLAGS = "-ffree-line-length-none"
-
         env.addPackage('MD-NMMD-Genesis', version=MD_NMMD_GENESIS_VERSION,
                        buildDir='MD-NMMD-Genesis', tar="void.tgz",
                        commands=[(

continuousflex/bibtex.py

@@ -126,6 +126,15 @@
 author = {Rémi Vuillemot and Osamu Miyashita and Florence Tama and Isabelle Rouiller and Slavica Jonic}
 }
 
+@article{vuillemot2023mdspace,
+  title={MDSPACE: Extracting continuous conformational landscapes from cryo-EM single particle datasets using 3D-to-2D flexible fitting based on Molecular Dynamics simulation},
+  author={Vuillemot, R{\'e}mi and Mirzaei, Alex and Harastani, Mohamad and Hamitouche, Ilyes and Fr{\'e}chin, L{\'e}o and Klaholz, Bruno P and Miyashita, Osamu and Tama, Florence and Rouiller, Isabelle and Jonic, Slavica},
+  journal={Journal of Molecular Biology},
+  pages={167951},
+  year={2023},
+  publisher={Elsevier}
+}
+
 @article{kobayashi2017genesis,
 author = {Kobayashi, Chigusa and Jung, Jaewoon and Matsunaga, Yasuhiro and Mori, Takaharu and Ando, Tadashi and Tamura, Koichi and Kamiya, Motoshi and Sugita, Yuji},
 title = {GENESIS 1.1: A hybrid-parallel molecular dynamics simulator with enhanced sampling algorithms on multiple computational platforms},

continuousflex/conda.yaml

@@ -4,13 +4,18 @@ dependencies:
   - pip
   - python=3.8
   - pip:
+      - setuptools==59.5.0
       - torch==1.10.1
+      - starfile
+      - matplotlib
+      - mrcfile
       - umap-learn
-      - scipion-em
-      - setuptools==63.4.3
       - torchvision==0.11.2
       - tensorboard==2.8.0
       - tqdm==4.64.0
       - protobuf==3.20.3
       - pycuda==2020.1
-      - farneback3d==0.1.3
+      - git+https://github.com/scipion-em/scipion-pyworkflow.git@devel
+      - scipion-em
+      - numpy==1.23.0
+      - git+https://github.com/MohamadHarastani/farneback3d.git

continuousflex/protocols.conf

@@ -106,11 +106,37 @@ MD-NMMD-Fitting = [
         {"tag": "protocol", "value": "ProtGenerateTopology", "text": "Generate topology", "icon": "bookmark.png"}
 	]},
 	{"tag": "section", "text": "4. Energy Minimization", "children": [
-	    {"tag": "protocol", "value": "ProtGenesis", "text": "MD-NMMD-Genesis", "icon": "bookmark.png"}
+	    {"tag": "protocol", "value": "FlexProtGenesis", "text": "MD-NMMD-Genesis", "icon": "bookmark.png"}
 	]},
 	{"tag": "section", "text": "5. Normal Mode Analysis (Optional)", "children": [
         {"tag": "protocol", "value": "FlexProtNMA", "text": "NMA"}
 	]},
 	{"tag": "section", "text": "6. Flexible Fitting using MD / NMMD", "children": [
-	    {"tag": "protocol", "value": "ProtGenesis", "text": "MD-NMMD-Genesis", "icon": "bookmark.png"}
+	    {"tag": "protocol", "value": "FlexProtGenesis", "text": "MD-NMMD-Genesis", "icon": "bookmark.png"}
+	]}]
+
+MDSPACE = [
+	{"tag": "section", "text": "1. Import atomic model", "children": [
+	    {"tag": "protocol", "value": "ProtImportPdb", "text": " Input PDB", "icon": "bookmark.png"}
+	]},
+		{"tag": "section", "text": "2. Import particles", "children": [
+        {"tag": "protocol", "value": "ProtImportParticles", "text": "Input particles", "icon": "bookmark.png"}
+	]},
+		{"tag": "section", "text": "3. Prepare simulation (Optional)", "children": [
+        {"tag": "protocol", "value": "ProtGenerateTopology", "text": "Generate topology", "icon": "bookmark.png"}
+	]},
+	{"tag": "section", "text": "4. Energy Minimization", "children": [
+	    {"tag": "protocol", "value": "FlexProtGenesis", "text": "MD-NMMD-Genesis", "icon": "bookmark.png"}
+	]},
+	{"tag": "section", "text": "5. Normal Mode Analysis", "children": [
+        {"tag": "protocol", "value": "FlexProtNMA", "text": "NMA"}
+	]},
+	{"tag": "section", "text": "6. MDSPACE", "children": [
+	    {"tag": "protocol", "value": "FlexProtMDSPACE", "text": "MDSPACE", "icon": "bookmark.png"}
+	]},
+	{"tag": "section", "text": "7. align output PDBs", "children": [
+	    {"tag": "protocol", "value": "FlexProtAlignPdb", "text": "PDB alignement protocol", "icon": "bookmark.png"}
+	]},
+	{"tag": "section", "text": "8. Principal Component Analysis ", "children": [
+	    {"tag": "protocol", "value": "FlexProtDimredPdb", "text": "PCA", "icon": "bookmark.png"}
 	]}]

continuousflex/protocols/__init__.py

@@ -51,7 +51,8 @@
 from .protocol_tomoflow_refine_alignment import FlexProtRefineSubtomoAlign
 from .protocol_deep_hemnma_train import FlexProtDeepHEMNMATrain
 from .protocol_deep_hemnma_infer import FlexProtDeepHEMNMAInfer
-from .protocol_genesis import ProtGenesis
-from .protocol_nmmd_refine import ProtNMMDRefine
+from .protocol_genesis import FlexProtGenesis
+from .protocol_mdspace import FlexProtMDSPACE
+from .protocol_generate_topology import ProtGenerateTopology
 from .protocol_generate_topology import ProtGenerateTopology
 from .protocol_pdb_synthesize import FlexProtSynthesizePDBs

continuousflex/protocols/protocol_align_pdbs.py

@@ -40,6 +40,9 @@
 PDB_SOURCE_OBJECT = 1
 PDB_SOURCE_TRAJECT = 2
 
+MATCHING_PDB_NONE = 0
+MATCHING_PDB_CHAIN = 1
+MATCHING_PDB_SEG = 2
 
 class FlexProtAlignPdb(ProtAnalysis3D):
     """ Protocol to perform rigid body alignement on a set of PDB files. """
@@ -48,7 +51,7 @@ class FlexProtAlignPdb(ProtAnalysis3D):
     # --------------------------- DEFINE param functions --------------------------------------------
     def _defineParams(self, form):
         form.addSection(label='Input')
-        form.addParam('pdbSource', EnumParam, default=0,
+        form.addParam('pdbSource', EnumParam, default=PDB_SOURCE_PATTERN,
                       label='Source of PDBs',
                       choices=['File pattern', 'Object', 'Trajectory Files'],
                       help='Use the file pattern as file location with /*.pdb')
@@ -83,12 +86,10 @@ def _defineParams(self, form):
                       label="Step of the trajectory",
                       help='Step to skip points in the trajectory', expertLevel=params.LEVEL_ADVANCED)
 
-
-
         form.addParam('alignRefPDB', params.PointerParam, pointerClass='AtomStruct',
                       label="Alignement Reference PDB",
                       help='Reference PDB to align the PDBs with')
-        form.addParam('matchingType', params.EnumParam, label="Match PDBs and reference PDB ?", default=0,
+        form.addParam('matchingType', params.EnumParam, label="Match PDBs and reference PDB ?", default=MATCHING_PDB_NONE,
                       choices=['All PDBs are matching', 'Match chain name + residue no',
                                'Match segment name + residue no'],
                       help="Method to find atomic coordinates correspondence between the pdb set "
@@ -130,11 +131,13 @@ def readInputFiles(self):
         # Get pdbs coordinates
         if self.pdbSource.get() == PDB_SOURCE_TRAJECT:
             pdbs_arr = dcd2numpyArr(inputFiles[0])
-            nframe, natom, _ = pdbs_arr.shape
-            pdbs_arr = pdbs_arr
+            start = self.dcd_start.get()
+            step = self.dcd_step.get()
+            end = self.dcd_end.get() if self.dcd_end.get() != -1 else pdbs_arr.shape[0]
+            pdbs_arr = pdbs_arr[start:end:step]
             for i in range(1,len(inputFiles)):
                 pdb_arr_i = dcd2numpyArr(inputFiles[i])
-                pdbs_arr = np.concatenate((pdbs_arr, pdb_arr_i), axis=0)
+                pdbs_arr = np.concatenate((pdbs_arr, pdb_arr_i[start:end:step]), axis=0)
 
         else:
             pdbs_matrix = []
@@ -154,28 +157,28 @@ def rigidBodyAlignementStep(self):
 
         # open files
         inputPDB = ContinuousFlexPDBHandler(self.getPDBRef())
+        inputPDB.write_pdb(self._getExtraPath("reference.pdb"))
         refPDB = ContinuousFlexPDBHandler(self.alignRefPDB.get().getFileName())
         arrDCD = dcd2numpyArr(self._getExtraPath("coords.dcd"))
         nframe, natom,_ =arrDCD.shape
         alignXMD = md.MetaData()
 
         # find matching index between reference and pdbs
-        if self.matchingType.get() == 1:
+        if self.matchingType.get() == MATCHING_PDB_CHAIN:
             idx_matching_atoms = inputPDB.matchPDBatoms(reference_pdb=refPDB, matchingType=0)
             refPDB.select_atoms(idx_matching_atoms[:, 1])
-        elif self.matchingType.get() == 2:
+        elif self.matchingType.get() == MATCHING_PDB_SEG:
             idx_matching_atoms = inputPDB.matchPDBatoms(reference_pdb=refPDB, matchingType=1)
             refPDB.select_atoms(idx_matching_atoms[:, 1])
         else:
             idx_matching_atoms = None
-        refPDB.write_pdb(self._getExtraPath("reference.pdb"))
 
         # loop over all pdbs
         for i in range(nframe):
             print("Aligning PDB %i ... " %i)
 
             # rotate
-            if self.matchingType.get() != 0 :
+            if self.matchingType.get() != MATCHING_PDB_NONE :
                 coord = arrDCD[i][idx_matching_atoms[:, 0]]
             else:
                 coord = arrDCD[i]
@@ -203,7 +206,7 @@ def rigidBodyAlignementStep(self):
     def createOutputStep(self):
         pdbset = self._createSetOfPDBs("outputPDBs")
         arrDCD = dcd2numpyArr(self._getExtraPath("coords.dcd"))
-        refPDB = ContinuousFlexPDBHandler(self._getExtraPath("reference.pdb"))
+        refPDB = ContinuousFlexPDBHandler(self.getPDBRef())
 
         nframe, natom,_ = arrDCD.shape
         for i in range(nframe):
@@ -238,8 +241,6 @@ def applyAlignmentStep(self):
             r2 = p2.getTransform()
             rot = r2.getRotationMatrix()
             tran = np.array(r2.getShifts()) / inputSet.getSamplingRate()
-            # middle = np.ones(3) * p1.getDim()[0]/2 * inputSet.getSamplingRate()
-            # new_tran = np.dot(middle, rot) + tran
             new_trans = np.zeros((4, 4))
             new_trans[:3, 3] = tran
             new_trans[:3, :3] = rot

continuousflex/protocols/protocol_deep_hemnma_infer.py

@@ -37,6 +37,7 @@
 from xmipp3.convert import (createItemMatrix, setXmippAttributes)
 from pyworkflow import BETA
 from continuousflex import Plugin
+from pwem.utils import runProgram
 
 OPTION_NMA = 0
 OPTION_ANGLES = 1
@@ -132,7 +133,7 @@ def createOutputStep(self):
         fn_infer = self._getExtraPath('infer.xmd')
         fn_combined = self._getExtraPath('images.xmd')
         args = '-i %(fn_train)s -o %(fn_combined)s --set union %(fn_infer)s' % locals()
-        self.runJob('xmipp_metadata_utilities', args)
+        runProgram('xmipp_metadata_utilities', args)
 
 
     #--------------------------- INFO functions --------------------------------------------