This is a collection of scripts that we use during cryoEM data collection and analysis with RELION. We find them very useful and we hope that you can find them useful too :) Enjoy!
- ice.py --> micrograph ice thickness/quality estimation
- png_out.py --> generation of png files from micrographs and ctf images
- relion_live.py --> LIVE feedback for On-The-Fly data processing with RELION (very useful to explore already processed datasets too!)
- relion_dashboard.py --> RELION metadata analysis
Nayim Gonzalez-Rodriguez, Emma Arean-Ulloa & Rafael Fernandez-Leiro
We encourage you to create a relion_dashboard conda environment that you can use to run these scripts
conda create -y -n relion_dashboard python=3.10
conda activate relion_dashboard
pip install pandas dash==2.6.2 starfile==0.4.10 pathlib2 numpy glob2 pathlib argparse seaborn regex dash-cytoscape==0.3.0 dash_bootstrap_components==1.4.0 flask==2.1.2 werkzeug==2.1.2Now, clone this repository to your favorite script folder or within the environment bin directory
git clone https://github.com/cryoEM-CNIO/CNIO_Relion_ToolsAdd the location to your PATH or provide full path when using the tools.
We provide an example of a Schemes folder for relion_it.py that includes the jobs for ice thickness estimation, and png export for relion_live.py. Copy the Schemes folder to your working directory BEFORE running relion_it.py.
The PNG scheme is set apart from the PREP scheme so it doesn't slow down preprocessing. The default relion_it.py script won't lauch it. To launch the PNG scheme please follow either of these two options:
1. Edit relion_it.py and insert the following in line 943, right below the section for the PROC scheme
if options['do_png']:
command = 'relion_schemer --scheme png --reset &'
print(' RELION_IT: executing: ', command)
os.system(command)
command = 'relion_schemer --scheme png --run --pipeline_control Schemes/png/ >> Schemes/png/run.out 2>> Schemes/png/run.err &'
print(' RELION_IT: executing: ', command)
os.system(command)
if do_gui:
command = 'relion_schemegui.py png &'
print(' RELION_IT: executing: ', command)
os.system(command)Remember to include the option 'do_png' : 'True', in your options.py file that you provide relion_it.py with.
We include the example option file CNIO_Relion_Tools_options_TUTODATA.py
relion_schemer --scheme png --reset &
relion_schemer --scheme png --run --pipeline_control Schemes/png/ >> Schemes/png/run.out 2>> Schemes/png/run.err &This script estimates ice-thickness for each micrograph in the input star file. We use the data generated by CtfFind, so using CtfFind is a requierement. The script sums the signal on the Ctf image in the 4-3.6 Å spatial frequency range.
This provides a simple and reasonable estimation of ice thickness that can be used for dataset initial cleanup and late stages data analysis (taking into account that the estimation is per-micrograph).
1. Activate conda environment BEFORE opening RELION's GUI
conda activate relion_dashboard
relion2. Run ice.py as an external job type
-
Executable: ice.py (provide full path if it is not in your PATH)
-
Input micrograph star file: micrograph_ctf.star (it should be a star file with Ctf information from a CtfFind job!)
Ice thickness estimation is stored in the rlnMicrographIceThickness label
The output micrograph_ctf_ice.star file can be used for a manual selection (Subset Selection job type) or any other downstream job that you want to perform.
This script provides LIVE feedback for On-The-Fly data processing with RELION
Execute the script from your project directory
relion_live.py
If the script is not in your PATH, use the full explicit path to the script.
It admits two arguments (quite often the default parameters are ok):
-
--port: choose the port were you want to launch the webapp (default: 8051)
-
--host: choose the IP address where you want the webapp to be hosted (default: localhost).
Example:
relion_live.py --host 01.010.101.010 --port 8050Open a web browser and access the server (localhost:8050)
In order for this to work, you need to create png files for every image and ctf in the motioncorr and ctffind folders, next to mrc/ctf files and with the same name.
To do this we use an external job type from the relion GUI to run png_out.py
If you want to include it in your relion_it.py schedule use the Schemes folder in this repository.
Same instructions as above still apply.
This script launches a webapp to analyse in detail each RELION job in a given project. The webapp is organized in 6 different tabs:
- Relion pipeline: an overview of all the nodes in the project and their connectivity.
- Analyse micrographs: an interactive scatter plot to analyze any micrograph's parameter vs any other (especially useful in combination with https://github.com/cryoEM-CNIO/CNIO_Ctf_Ice_Thickness!).
- Analyse particles: an interactive scatter plot to analyze any set of particle's parameter vs any other.
- Analyse 2D Classification: a tab dedicated to analyse or follow in real time your 2D classifications, monitoring particle distribution per class and convergence.
- Analyse 3D Classification: follow the convergence of the classification, per-class distribution of particles and alignment in each iteration of 3D classifications.
- Analyse 3D Refine: follow in real time the convergence, angular distribution and FSC in each iteration.
Note that all plots are interactive. The ones in in the Analyse particles and Analyse micrographs tabs can also be used to select a subset of micrographs/particles graphically and export them as a .star file that can then be imported into RELION:
Execute the script from your project directory
conda activate relion_dashboard
relion_dashboard.py
If the script is not in your PATH, use the full explicit path to the script.
It admits two arguments (quite often the default parameters are ok):
-
--port: choose the port were you want to launch the webapp (default: 8051)
-
--host: choose the IP address where you want the webapp to be hosted (default: localhost).
Example:
relion_dashboard.py --host 01.010.101.010 --port 8051