v3.3.1: fixes to backprojection and tilt with indices; per tomo star filtering

This is a patch release to address several bugs and issues that have come to our attention:

• adding --micrograph-files argument to filter_star to create separate output files for each _rlnMicroGraphName encountered in the file
• --ind with --encode-mode=tilt wasn’t working in the case where all particles had the same number of tilts due to dtype=object patch introduced earlier
  • fixed by storing particle→tilt index produced by TiltSeriesData.parse_particle_tilt() as a list instead of an array; this is more robust in general and all downstream cases are agnostic (see tests below)
• backproject_voxel was producing errors when trying to calculate threshold FSC values due to deprecated code used to parse FSC matrix (#371)
  • fixed by copying over code already used in commands/fsc
• train_nn and train_vae would error out if inputs were not divisible by 8 when using AMP optimization (e.g. #353)
  • a warning here suffices as AMP optimization is the default and this is frustrating for many users
• better error message when CTF file is missing from write_star inputs
• better error message when backproject_voxel output is not .mrc
• bug in ET_viz notebook when --ind not specified caused by inconsistent definition of ind0
• bug in filtering notebook caused by using ind=ind_orig when loading dataset and then trying to filter again (#363)
• ZeroDivisionError bugs in all notebooks when using small training datasets
• updating template analysis notebooks to use the given kmeans value in the copied-over notebook, similarly to out auto-updating of notebook epoch numbers

In addition to making the required fixes, we have expanded and improved our deployment tests to cover these cases and close some gaps in our testing coverage:

• adding a stand-alone test of backprojection under test_reconstruct applying both .mrcs and .star inputs
• more testing of train_nn cases with different --amp, --batch-size, --poses values
• fixing check=True issue in utils.run_command() that was allowing tests of backprojection to fail silently
• new deployment task schedule
  • the main deployment task has been split into tests and style for tests of code integrity and code linting respectively
  • run tests and style along with beta-release any time a patch version tag [0-9]+\.[0-9]+\.[0-9]+-* is pushed to any branch to trigger a verified upload to TestPyPI
    • also run tests and style for any push to develop branch to allow for testing before beta release
  • update release to only run when a stable version tag (^[0-9]+\.[0-9]+\.[0-9]+$) is pushed to main
    • tests and style run on any push to main to allow for testing prior to release

Other changes include:

• applying tmpdir_factory to improve the train_dir and AbinitioDir fixtures used in tests with more robust setup and teardowns
• CodeFactor badge and nicer TestPyPI installation command in README
• dynamic update of plotted point sizes in cryoDRGN_filtering.ipynb interactive filtering widget, useful for smaller datasets for which the default is too small for points to be seen
• using plt.close() after analyze plotting for better memory management

v3.3.0: direct traversal, improved notebooks, TestPyPI auto-deployment

New Features

• cryodrgn direct_traversal, a tool for interpolating a path in the latent conformation space connecting two points in a direct line
• making the package available for installation using the TestPyPI distribution service:
  pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ "cryodrgn<=3.3.0" --pre
  • this will let us to make both the development and stable versions of the package available for easy download using pip, as opposed to having to use git clone for the former.

Improving Existing Features

• updated interfaces for cryodrgn graph_traversal and cryodrgn pc_traversal so that the arguments, argument formats, and help docstrings between all three traversal methods are as clear and consistent as possible
  • --ind in direct_traversal is replaced with --anchors as in graph_traversal, allowing both to take a list of integers as well as files containing lists of integers
  • -o now also has a more verbose alias --outtxt in graph_traversal and direct_traversal; updating its behavior in graph to save the latent space co-ordinates and updating --outind to save path indices; similarly verbose alias --outdir in pc_traversal
  • -o now also has a default value that is used when the flag is given with no argument across all three traversal commands to mean that we want to save output but don't have a file name
  • when -o is not given, all three commands display a prettier log message to screen with traversal output
• epoch numbers are automatically updated to the epoch used in cryodrgn analyze in copied-over demo notebooks
• improving package status badges shown in GitHub README: available versions, PyPI downloads

Addressing Issues and Bugs

• adding the --datadir flag to cryodrgn abinit_homo, addressing an oversight that complicated using .star files with this command (#343)
• fixing bugs and other issues found in our demonstration Jupyter notebooks (#363)
  • analysis.plot_projections() doesn't fail if # of imgs is two or one
• makeover of GitHub deployment workflow actions to fix errors and simplify release infrastructure
  • master->main branch names
  • removing remaining errors in continuous integration testing action so that it is again a useful tool for checking pull requests and protecting our main branch, especially with the now expanded coverage of notebooks, traversal, etc.
    • last pytest bug fixed (n=tilts in eval_images)
    • switching off pyright for now as type checks are not essential
    • leftover pre-commit formatting issue in commands.filter
  • more lightweight Docs action by only releasing new Sphinx autodocs version when a new version tag is pushed — not nuking these docs for now (#350)
  • new Beta Release action for automatically deploying a release to TestPyPI whenever a new version tag is pushed
  • existing Release action still not working (needs updated credentials) but is now also only deployed automatically when a new version tag is pushed
• fixing ntilts=10 default behaviour bug in eval_images which was activating tilt mode
• officially removing support for outdated Python versions 3.7 and 3.8 (already implicitly not supported)

Testing

• renaming test_quick to test_integration and improving the coverage of the reconstruction pipeline integration tests contained therein
  • adding integration tests for Jupyter demonstration notebooks to check that they execute successfully upon running cryodrgn analyze after cryodrgn train_vae with different types of inputs and parameters
• expanded fidelity and unit tests for all three traversal commands
• adding CODEOWNERS letting @michal-g be e.g. automatically added to new issues

Version 3.2.0-beta: cleaning, half-map FSCs, mask generation, and RELION 3.1

New Features

• introducing cryodrgn_utils clean, a new tool for removing extraneous output files from completed experiments (#297)
• backproject_voxel now produces half-maps and a half-map FSC by default (#329)
• creating cryodrgn_utils fsc from the fsc analysis script for calculating Fourier shell correlations between two .mrc volume files, and likewise cryodrgn_utils plot_fsc based on plotfsc; making the latter available through the former using -p
• creating cryodrgn_utils gen_mask based on cryoem_tools.gen_mask.py, now with reparametrization in Angstroms

Addressing Issues and Bugs

• fixing #358 and improving the I/O interface in both cryodrgn_utils flip_hand and cryodrgn_utils invert_contrast so that the name of the output file and any parent directories are created automatically, with more unit tests for each
• making write_star use RELION 3.1 format by default with optics groups generated from image size, pixel size, voltage, spherical aberration, and amplitude contrast; -relion30 to use old format (#324)
• updating install setup to prevent use of Python 3.11 (#306)
• abinit_homo now saves a config.yaml with a summary of parameters used, like abinit_het, train_vae, and train_nn
• fixing filter_star to accept tilt series as well (#335)
• fixing affinity bug in analyze_landscape (#345)
• fixing beta value bug in train_vae (#356)
• removing references to scipy.ndimage.morphology which is deprecated
• fixing dtype=object warning message in TiltSeries.parse_particle_tilt()

User Interface

• cleaner implementation of command-line interface, defining both cryodrgn and cryodrgn_utils commands in one file cryodrgn/command_line.py, and removing e.g. manually defined lists of modules with commands in them
• better doc strings with some usage examples for commands (e.g. cryodrgn abinit_homo -h), with module-level doc strings being included explicitly in the automatically generated help screen

Testing

• using conftest.py to define a new setup/teardown routine for experiment output directories created by tests
• writing new tests for abinit and train methods by applying these routines
• fixing test_dataset to account for changes within make_dataloader
• updating unit tests that use argparse.ArgumentParser() directly for commands in which the __main__ method was removed
• updating tests for new and updated commands fsc, clean, gen_mask, etc.

Version 3.1.0-b: interactive filtering

We have introduced a number of small fixes and feature updates since our last release v3.0.1-beta:

• creating a new interactive command-line interface cryodrgn filter as an alternative to the buggy interface in the Jupyter filtering notebook (#323)
• making cryodrgn analyze produce a plot of the learning curve (#304)
• adding cell in cryoDRGN_filtering jupyter notebook returned by cryodrgn analyze for filtering by UMAP/PC values (#313)
• fixing bugs with deprecated signatures in plotting functions (#322) and numpy dependency versioning (#318)

Version 3.0.1-beta

This is a patch release to address #312 and #310 found in v3.0.0-beta by updating the list of dependencies and making some updates to the analysis Jupyter notebooks produced by experiments.

Version 3.0.0-beta

The official cryoDRGN-ET release for heterogeneous subtomogram analysis.

• [NEW] Heterogeneous reconstruction of subtomograms. See documentation on gitbook
• [NEW] cryodrgn direct_traversal for making movies
• Updated cryodrgn backproject_voxel for voxel-based homogeneous reconstruction
• Major refactor of dataset loading for handling large datasets

Version 2.3.0

• Model configuration files are now saved as human-readable config.yaml files (#235)

• Fix machine stamp in output .mrc files for better compatibility with downstream tools (#260)

• Better documentation of help flags in ab initio reconstruction tools (#258)

• [FIX] By default, window images in cryodrgn abinit_homo (now consistent with other reconstruction tools) (#258)

• [FIX] Reduce memory usage when using --preprocessed and --ind (#272)

• Updated functionality in cryodrgn_utils filter_star

• Upcoming in the next minor version (v2.4):

  • We are working on a major refactor of data loading for handling large datasets. This will entail an API change for the mrc.py library module

Version 2.2.0

• New ab initio reconstruction tools:

  • cryodrgn abinit_homo
  • cryodrgn abinit_het

• New utility script for writing cryoSPARC .cs files:

  • cryodrgn_utils write_cs

• Improved plotting in cryodrgn analyze

• Documentation and tutorial converted to sphinx docs: https://zhonge.github.io/cryodrgn/

• Many codebase improvements with open-source software development practices (e.g. continuous integration tests, black, flake8, pyright, logging, and PyPi packaging).

Version 1.1.2

Minor release with updated documentation (and testing PyPI packing).

Version 1.1.0

Updated default settings to larger model architecture, modified positional encoding, and accelerated training:

• Mixed precision training is now turned on by default (Use --no-amp to revert to single precision training)
• Encoder/decoder architecture is now 1024x3 by default (Use --enc-dim 256 and --dec-dim 256 to revert)
• Gaussian Fourier featurization for faster training and higher resolution density maps (Use --pe-type geom_lowf to revert)