MRIQC Anat pipeline: differences with MRIQC v22.06

[manuegrx]

There were many differences between MRIQC implemented in MIA and MRIQC v22.06 (for anat and func pipeline).
These differences were due to workflow and metrics computation differences.
MIA pipeline has been modified to be up to date with MRIQC v22.06 (see commit 2f540b1 ).

For functional data, MRIQC v22.06 results and results from MRIQC implemented in MIA are the same (see first sheet of the attached file compa_indices_alej.xlsx).

For anat data, there are still some differences for almost all indices (but slight differences, see the second sheet of the attached file).

It seems that for some of them (cnr, qi_1, qi_2, snrd, summary_bg, tpm_overlap) , if MRIQC v22.06 is launch a second time these indices are also slightly different.
It seems to be explain by differences in the "inverse composite transform " obtained in the Registration bricks (ANTs registration) of the spatial normalization pipeline.
As this transform is slightly different each time, the airmask obtained and the template registered are slightly different and so are the indices computed using theses images.

For the other indices, the differences came from a difference in the result obtained with N4BiasFieldCorrection (ANTs) in MIA and in MRIQC 22.06. This command is used 2 time in the Skull stripping pipeline.
This issue has been investigated without success for now but it will be great to understand why the N4BiasFieldCorrection command give different result in MIA.
The file "MRIQC_anat_issues.docx" resume tests already done

If we are using output images from MRIQC v22.06 Skull stripping pipeline as input in MIA (and so skipping the skull stripping pipeline in MIA) we obtain similar results to when MRIQC v22.06 is launch a second time

TO DO : understand why N4BiasFieldCorrection command give different results in MIA.
MRIQC_anat_issues.docx

compa_indices_alej.xlsx

[servoz]

Please, can you add in your .bashrc the following lines:

export ANTS_RANDOM_SEED=1
export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1
export OMP_NUM_THREADS=1

Then open a new shell or do source .bashrc if you want to keep the current shell.
What does the comparison between host and Mia give with these environment variables? (no need to look for the docker, we don't have the hand on the procs management).

[servoz]

Without ANTS_RANDOM_SEED, ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS, OMP_NUM_THREADS

---

• on_host:

  • func: same (3 runs)
  • anat: not same (3 runs):
    • Change:
      cnr,
      qi_1, qi_2,
      snrd_csf, snrd_gm, snrd_total, snrd_wm,
      summary_bg_k, summary_bg_mean, summary_bg_n, summary_bg_p95, summary_bg_stdv,
      tpm_overlap_csf, tpm_overlap_gm, tpm_overlap_wm

• in_mia:

  • func: same (3 runs)
  • anat: not same (3 runs):
    • Change:
      same as on_host +
      histogram_qi2_ref_pdf (only in Mia)

• in_docker:

  • func: same (3 runs)
  • anat: not same (3 runs):
    • Change:
      same as on_host

=> Func, reproducibility
=> Anat: virtual reproducibility (the parameters impacted and the level of variation tend to indicate that this would be due to the variability inherent in some types of calculations implemented for anat such as those with ANTs and others see: https://github.com/ANTsX/ANTs/wiki/antsRegistration-reproducibility-issues)

• on_host Vs in_mia Vs in_docker:
  • func: Only a little change in dvars_vstd (0.0006%). in_mia same as on_host.
  • anat: not same for all IQMs

Concusion: difficult to identify the variation that will be induced by the pipelines themselves because of the variances obtained for each pipeline by running them several times. We must first fix the intra-pipeline variance = > ANTS_RANDOM_SEED, ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS, OMP_NUM_THREADS=1 environment variables should fix this first issue (I'm not sure it's mandatory to use all these variables, but this way we should be sure to handle the problem...)

with ANTS_RANDOM_SEED, ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS, OMP_NUM_THREADS

---

• on_host:

  • func: same (3 runs)
  • anat: same (3 runs) => We manage threading and variance

• in_mia:

  • func: same (3 runs)
  • anat: same (3 runs) => We manage threading and variance

• in_docker:

  • func: same (3 runs)
  • anat: not same (3 runs):
    • Change:
      same as on_host without ANTS_RANDOM_SEED, ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS, OMP_NUM_THREADS => you can't manage threading in docker with host environment variables.

=> Func, reproducibility
=> Anat: reproducibility (except for in_docker but we know why)

• on_host Vs in_mia Vs in_docker:
  • func: Only a little change in dvars_vstd (0.0006%). in_mia same as on_host. As this does not come from variance, there must be a small pipeline difference between docker and (mia - host). We could look for the reason - a fix (do we want or accept to live with this very small difference?)
  • anat: not same for all IQMs
    • host Vs docker: We see the same change (IQMs / intensity) as without ANTS_RANDOM_SEED, ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS, OMP_NUM_THREADS => the two pipelines are identical and the differences are explained by the variance not managed in the docker.
    • Mia Vs (docker - host): not same for all IQMs.

Concusion: the differences observed between mia and host are not due to variance. There is a difference between the two pipelines. I am investigating this.

withoutANTS_RANDOM_SEED-ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS-OMP_NUM_THREADS.zip

withANTS_RANDOM_SEED-ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS-OMP_NUM_THREADS.zip

[servoz]

Since the last commit, we observe:

• Threading and random_seed management is not an option (see output_iqms_without_env_var.tsv). If MRIQC is used to make group stats it is necessary to manage this point. I don't know what is better. Tell users to set environment variables (I'm not a fan of this but not completely against)? Add a "reproducibility" plug and manage it with os.environ ?
• We have now almost the same results between host and mia. There are still differences with docker, but as shown in the previous post, this is completely explainable because we do not manage threading and random_seed in the docker.

Finally there are 4 extremely small differences (<1E-05 %):

• inu_med: 4.63E-06%
• tpm_overlap_csf: 1.057E-05 %
• tpm_overlap_gm: 5.19E-07 %
• tpm_overlap_wm: 2.27E-06

and a big difference: q2.

For the report generation, I had worked in September 22 on the curve plot of q2 and I had observed that this were very strange. I think that the q2 is miscalculated and it would be worthwhile to see if there is an error in the script for q2.

But for inu_med, tpm_overlap_csf, tpm_overlap_gm and tpm_overlap_wm, since we have a lot of other things to do, I suggest to stop here.

If it's ok for you @manuegrx , I'll let you close this ticket.

output_iqms_with_env_var.tsv.zip

output_iqms_without_env_var.tsv.zip

[manuegrx]

@servoz I get exactly the same result as you in MIA using the env variable !
I took a quick look at the qi2 code and I did not see an obvious error

[servoz]

Thanks for not closing yet this ticket!
With the latest commits, tpm_overlap_csf, tpm_overlap_gm and tpm_overlap_wm are fixed.
I am investigating the last discrepancy for inu_med and q2 ...
output_iqms.tsv.zip

[servoz]

inu_med fixed.

[servoz]

qi_2 fixed.
Now we have to deal with the variance and this ticket could be closed!

[servoz]

Careful investigation shows that:

• There is intrinsic variability in the calculations used for the anat pipeline, unlike for the func pipeline.
  Without_random_seed_management.zip
• This variability can be cancelled by using the environment variable ANTS_RANDOM_SEED (the other two, ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS and OMP_NUM_THREADS seem to be unnecessary). The result is visible in Mia or for MRIQC directly on the host. The environment variable has of course no effect in the docker.
  with_random_seed_management.zip
• After handling this variability, the different commits performed so far provide the same results for the anat pipeline between host and Mia, the difference observed with docker can be explained by the variability which is not managed in the docker. On the other hand we observe two differences for the func. Concerning dvars_vstd, we won't invest time because we have the same result in Mia and in the host which means that the difference is rather on the docker side (dependencies? codes?). For tsnr it's a bit different, as the difference is between Mia and host. I think it's a good idea to fix this last issue. We will then have a version in Mia perfectly identical to MRIQC on the host. The only difference with docker will be the variability that we can't manage in docker and the dvars_vstd parameter, which will come from the docker side instead.
  docker_mia_host_random_seed_managed.zip

So, I try to fix tsnr in func pipeline, then it will be time to close this ticket!

[servoz]

tsnr fixed.