C++ acceleration for mmCIF parsing (native "fast path" for coordinates)

[Felixburton7]

Hi all,

I've worked with large datasets (ribosomes, viral capsids) and hitting some performance limits with the current MMCIFParser. Even with FastMMCIFParser, the overhead of creating Python Atom objects for 200k+ entries is a bottleneck.

I'm interested in contributing a native C/C++ extension to speed this up.

Update suggestion and context:

This challenge was tackled in the AlphaFold 3 data pipeline (source). The biggest performance gains came from:

1. Moving tokenization out of Python.
2. The "Fast Path": Bypassing the creation of millions of Python objects when we only needed coordinates for ML models.

Proposal of changes

It could be good to implement a lightweight extension (e.g., Bio.PDB._cmmcif) using the standard Python C API—similar to the existing ccealignmodule.c.

This could would offer two modes:

1. A drop-in Replacement: Accelerates the standard get_structure() by doing tokenization in C (est. 1.5-2x speedup).
2. A NumPy "Fast Path": an additional method (e.g., parse_to_arrays()) that extracts coordinates/B-factors directly into NumPy arrays, skipping the object overhead entirely.
   • Target Speedup: ~50-100x (based on previous work).

Other things considered

• Gemmi: is an excellent library, but it follows a DOM approach (creating objects for everything) and would introduce a new dependency. I am proposing a native solution that fits Biopython's existing build system.

quick questions

1. Is there interest in a "NumPy-only" fast path for ML workflows?
2. Are there any blockers to adding a new C extension for parsing?

Note: This would be a clean-room implementation (written from scratch) to ensure full compatibility.

Happy to write the code or give more detail.

[Felixburton7]

@JoaoRodrigues / @peterjc any update or anything I can do?

[peterjc]

I would ask that you don't use generative AI (I forget now if you'd expressed an interest there), my views https://blastedbio.blogspot.com/2025/11/thoughts-on-generative-ai-contributions.html

Following the Bio.PDB._cmmcif precedent seems reasonble to me - but I've not used the structural code for some years and defer to @JoaoRodrigues, @etal, etc for reviewing here.

[JoaoRodrigues]

Looking into a faster tokenizer sounds good, since that benefits everyone. As for a new parser that returns a pure array, that's fine too. I'd probably implement that as a separate parser entirely, e.g. "MMCIFCoordinateParser". Note that if we do the tokenizer, we can deprecate the FastMMCIFParser.

[JoaoRodrigues]

Also, I agree with @peterjc regarding AI. Please make sure any contribution you submit has been fully vetted by yourself.

[Felixburton7]

Thanks @peterjc @JoaoRodrigues, I’ll look into this and follow your guidance on best practices (including AI) and the implementation plan. Thanks for the help, and I’ll update you once I’ve got an implementation.

[Augustin-Zidek]

I strongly recommend doing this, we saw a >20x speedup when we switched to our C++ CIF parser from Biopython's CIF parser. The API design is quite similar to MMCIF2Dict: https://github.com/google-deepmind/alphafold3/blob/main/src/alphafold3/parsers/cpp/cif_dict_lib.cc

[JoaoRodrigues]

@Augustin-Zidek , would you guys be interested in contributing that piece of code to Biopython? I see the license includes a NC clause, which is not compatible with our BSD-3 license. Seems silly to reinvent the wheel here :)

[Augustin-Zidek]

Agreed. Let me investigate this and get back to you (might take a while though, sorry!).