How to reproduce results?

[IlyaTrofimov]

Hello!

Dear colleagues, I'm trying to reproduce results from your paper.
I have downloaded data and executed scripts, assuming that the first one is SHAPR and the second one is SHAPR+topological loss:

python run_train_script.py -p config/red-blood-cell.json
python run_train_script.py -p config/red-blood-cell-2D.json

Training has finished, but numbers on a test set which I see in WanDB are different from numbers in your paper (https://arxiv.org/pdf/2203.01703.pdf, Table 1).
In fact the list of metrics in WanDB (IoU_Error, dice_error, volume_error) are different from the errors presented in the paper (1 − IoU, Volume, Surface area, Surface roughness).
The script "shapr/scripts/evaluation.py" only outputs figures.

Can you please give a hint how to reproduce results?

[Pseudomanifold]

Hi!

The script needs to be modified to result in tabular output. The lead author @DLMLDW also used a Jupyter notebook with which to tabulate the results—maybe you could share this notebook, Dominik? Otherwise, evaluation.py produces a data frame that can be stored; you have to change the call to make_df accordingly. Hope that helps!

[IlyaTrofimov]

I have modified evaluation.py to calculate average of 1 − IoU, Volume, Surface area, Surface roughness.
Now results match Table 1 in your paper.
Thanks.

[Pseudomanifold]

Thanks so much! If you want, we'd be happy to accept this as a pull request!