Results not matching with paper?

[samra-irshad]

Hi, I am trying to reproduce the results but could not get the results closer to those reported in paper. Did you utilize validation set for reporting the results or test set (in table 4)? Also, could you provide with the training, test and validation splits?

[sinAshish]

The reported results are from our validation set since ground truth in testing set are not available.
Also, the experiments were performed about 2 years ago and I have changed the computer, so I don’t recall the exact splitting, but tbh results should be consistent across architectures.

[samra-irshad]

Right. Actually I am not talking about the test set in competition, I am talking about the test set in split. Like you divided the data into (13: train, 2: validation and 5: test) for CHAOS dataset. I can see in code that validation performance has been saved in folders at end of each epoch and the best segmentations on validation split are used for 3D reconstruction. So the question is did you use validation split (2 cases) for reporting model's performance in paper or reported the scores after evaluating the model with best validation score on test set at end?

[sinAshish]

I'm not sure exactly about the splits, but a 5-fold cv was used to report the score on the test set.

[josedolz]

The reported results are on our test set (averaged over k-folds). The validation set is only used to pick the best model during training for each run.

[samra-irshad]

Ok thanks Jose

[samra-irshad]

My results are different for 3-fold cross-validation and i have applied the same exact method. I am clueless why there's performance difference. I have emailed you the training, validation and test lists that I am using. It would be very helpful if you could train your model instance with my sent lists and update me with the results. Or maybe train your model once again for a fresh train-valid-test splits and update me on email or here? Or perhaps provide the instance of trained model?

[sinAshish]

train your model instance with my sent lists and update me with the results. Or maybe train your model once again for a fresh train-valid-test splits

sorry, but this project is already done and we are busy with other things now.
You should try looking for any bugs in the code!