mAFQ vs. pyAFQ: Differences in streamline counts and FA

[TalOhevia]

Hi,

Our lab is in the final stages of transitioning from mAFQ to pyAFQ (3.0), and we are currently validating the new pipeline using an existing dataset previously analyzed with mAFQ to assess the comparability of the results.

The original DWI resolution is approximately 2.6–2.7 mm isotropic. During our initial tests, pyAFQ had difficulty reliably segmenting some bundles, particularly the Optic Radiation. I therefore upsampled the preprocessed DWI data to 2 mm, which substantially improved the stability of the segmentation. For the comparison with our previous MATLAB results, I tried to keep the downstream processing as similar as possible between the two pipelines. In particular, we used the same whole-brain tractography approach and matched the AFQ cleaning parameters as closely as possible.

However, we are seeing fairly large and systematic differences between the MATLAB AFQ and pyAFQ results:

• pyAFQ retains substantially fewer streamlines per bundle than mAFQ.
• At the same time, the resulting FA values are consistently higher in pyAFQ.

As an example, I am attaching results for the Arcuate, including a single-subject example illustrating the difference in the number of retained streamlines, as well as the group-level difference in FA between the two pipelines.

I realize that this is not a perfectly controlled comparison, particularly because the preprocessing/registration pipelines differ, and I would not necessarily expect mAFQ and pyAFQ to produce identical results. However, the consistency and magnitude of the differences made us wonder where they might be coming from.

Have you encountered similar differences when comparing mAFQ and pyAFQ? And do you have any recommendations for how we could minimize them, or better isolate their source, so that we can fully understand the implications of transitioning from mAFQ to pyAFQ?

Any advice on which part of the pipeline you would consider the most likely source of these differences would be greatly appreciated.

Thank you very much for your time and help, and have a great weekend!

Best,
Tal Ohevia

[arokem]

Hi Tal - thanks for sharing these results. That does seem like a rather dramatic difference.

Early on in the development of pyAFQ we made similar comparisons, and found repeatability to be pretty good (see https://pmc.ncbi.nlm.nih.gov/articles/PMC8785971/), so this is surprising.

My hunch is that this has to do with the process of generating the streamlines. That is, differences in the tractography settings used in each of these pipelines. One way to confirm this hunch would be to take the whole-brain tractography generated in mAFQ (with mrtrix?) and use that as input to the following steps in the pyAFQ. Would you mind trying that (as documented here: https://tractometry.org/pyAFQ/howto/converter.html#tractography-from-other-pipelines)?

[TalOhevia]

Hi Ariel,

Thank you very much for the suggestion, and apologies for the delayed follow-up.

I tried using the whole-brain tractography from the MATLAB pipeline as input to pyAFQ, together with the corresponding FA map. Interestingly, the resulting FA profiles were still very similar to my original pyAFQ results, including the systematically higher FA values. This suggests that tractography differences alone are probably not driving the discrepancy.

I also found that interpolation does account for part of the difference: using trilinear rather than cubic interpolation brought the pyAFQ profiles noticeably closer to MATLAB AFQ (see attached example), although a substantial difference still remains.

So I’m still trying to isolate the remaining source, but your suggestion was very helpful in narrowing it down. Thank you again for your help!

Best,
Tal

[arokem]

Thanks for testing the tractography differences. I am indeed surprised that this had little impact, but maybe that's encouraging. At least it means pyAFQ is more reliable than I thought.

As for your new results - this seems like a big effect for a small tweak, but I am not sure that I understand what I am looking at. At what point in the processing is this interpolation taking place?

[TalOhevia]

Hi Ariel,

Thank you for your reply, and apologies for the delayed update. Since then, I tested an additional dataset that had been preprocessed using a different pipeline, and observed a very similar FA discrepancy. This makes me think that the preprocessing/interpolation is probably not the main source of the difference, as I initially suspected.

I’m still trying to isolate the source, and I’d be happy to share any insights if I manage to identify it.

Thanks again for your help!
Tal

[36000]

Thanks for continuing to look into this. It may be worth varying the cleaning parameters on the pyAFQ side. It looks like, in the arcuates you posted, pyAFQ cleaned them more strictly. This would also increase FA. In pyAFQ, you can pass this to either GroupAFQ or ParticipantAFQ:

segmentation_params = {"cleaning_params": {"distance_threshold": 4, "length_threshold": 4, "clean_rounds": 5}}

Those are the defaults. Note that you do not have to re-run the entire pipeline to after updating cleaning parameters to see the difference. You just have to re-run bundle recognition. This can be done by doing:

myAFQ = ParticipantAFQ(..., segmentation_params=segmentation_params)
myAFQ.clobber(dependent_on="recog") # deletes all derivatives that depend on bundle recognition
myAFQ.export_all()

Overall, there are many differences between the pyAFQ and mAFQ pipelines. Another difference would be in the seeding strategy. You can vary seeding strategies. For example, if your T1w image is low-quality, you can revert to seeding wherever FA>0.2 following this example: https://tractometry.org/pyAFQ/howto/howto_examples/pyafq_2.html

[TalOhevia]

Thank you both for your suggestions!

I experimented with different bundle-cleaning parameters and was able to obtain cleaner, anatomically plausible bundles (see attached example). However, the FA values remained systematically higher than those obtained with mAFQ.

I also compared two approaches to diffusion tensor fitting, one using MRtrix3 and the other using pyAFQ. Both produced very similar FA values, suggesting that tensor fitting and FA map generation are unlikely to be the main sources of the discrepancy.

One detail worth mentioning is that I intentionally used clip_edges=False, as I found that this setting produced better bundle segmentation results than clip_edges=True. I then performed the clipping and FA sampling between the anatomical ROIs in a separate script. However, even the FA profiles generated directly by the attached pyAFQ script (without clipping) show values that are considerably higher than what we typically observe in our lab.

This brings me back to the possibility that I may be overlooking something in my pyAFQ configuration or analysis workflow.

I've attached the script I'm currently using. If you have a chance to take a look, I would really appreciate it if you could let me know whether anything stands out as potentially problematic or unusual.

Thank you again for your time and help!

Best,
Tal


pyafq_imported_tractography_v34.py