Parallel IDF

[joshchiou]

The current implementation of idf_from_chunks is sequential, which makes the IDF step a bottleneck for very large matrices. This PR adds a version which parallelizes the IDF counting using existing Rayon imports, which should improve performance on large datasets.

[kaizhang]

Hi Josh, thanks for the pull request! Have you tested if the new implementation produces the same output as the old one? And what is the performance gain for large datasets?

[joshchiou]

Hi Kai - I added a couple of tests with simulated matrices that show that the new implementation produces identical results to the old one.

I haven't had a chance to benchmark performance gain, but previously the single threaded approach was stuck on the idf step for a ~10M x 1M matrix for about a week without finishing. With the parallel implementation it finished in < 8 hours using 48 CPUs.

Results from the test:
https://github.com/joshchiou/SnapATAC2/tree/idf_from_chunks_parallel/snapatac2-python/tests/test_idf

Tolerance: 1.00e-12

Test 1: Random matrices with varying density
✓ Random (density=0.05): Vectors are mathematically identical (max diff < 1.00e-12)
✓ Random (density=0.1): Vectors are mathematically identical (max diff < 1.00e-12)
✓ Random (density=0.3): Vectors are mathematically identical (max diff < 1.00e-12)
✓ Random (density=0.7): Vectors are mathematically identical (max diff < 1.00e-12)

Test 2: Uniform matrix (all columns have same count)
✓ Uniform matrix: Vectors are mathematically identical (max diff < 1.00e-12)

Test 3: Sparse matrix (some columns have zero counts)
✓ Sparse matrix: Vectors are mathematically identical (max diff < 1.00e-12)

Test 4: Multiple chunks with different sizes
✓ Multiple chunks: Vectors are mathematically identical (max diff < 1.00e-12)

Test 5: Single large matrix
✓ Large matrix: Vectors are mathematically identical (max diff < 1.00e-12)

Both idf_from_chunks and idf_from_chunks_parallel produce identical results.