Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Epistemic uncertainty with and without effects #4

Open
vietngth opened this issue Nov 7, 2022 · 0 comments
Open

Epistemic uncertainty with and without effects #4

vietngth opened this issue Nov 7, 2022 · 0 comments

Comments

@vietngth
Copy link

vietngth commented Nov 7, 2022

Hi, I ran the experiments for ood_loc_gpn_{10,16} to reproduce the results. The metrics are logged after training as follows:

val test
accuracy 0.923077 0.889306
brier_score 0.280527 0.269254
ECE 0.10638 0.0702577
confidence_aleatoric_apr 0.966722 0.980251
confidence_epistemic_apr 0.95776 0.962611
confidence_structure_apr nan nan
confidence_aleatoric_auroc 0.706276 0.864282
confidence_epistemic_auroc 0.635031 0.772384
confidence_structure_auroc nan nan
CE 0.37979 0.339846
avg_prediction_confidence_aleatoric 0.816697 0.819579
avg_prediction_confidence_epistemic 14919.8 14410.5
avg_sample_confidence_aleatoric 0.816697 0.819579
avg_sample_confidence_epistemic 16556.9 15927.3
avg_sample_confidence_features 15835.7 16779.6
avg_sample_confidence_neighborhood nan nan
average_entropy 0.513569 0.501767
ood_detection_aleatoric_apr 0.801985 0.78415
ood_detection_aleatoric_auroc 0.833289 0.836685
ood_detection_epistemic_apr 0.80048 0.788122
ood_detection_epistemic_auroc 0.852889 0.855747
ood_detection_features_apr 0.75896 0.748484
ood_detection_features_auroc 0.81794 0.8237
ood_detection_neighborhood_apr nan nan
ood_detection_neighborhood_auroc nan nan
ood_detection_structure_apr nan nan
ood_detection_structure_auroc nan nan
ood_accuracy 0 0
ood_avg_prediction_confidence_aleatoric 0.556919 0.558297
ood_avg_prediction_confidence_epistemic 2879.92 2808.5
ood_avg_sample_confidence_aleatoric 0.556919 0.558297
ood_avg_sample_confidence_epistemic 4068.24 4109.26
ood_avg_sample_confidence_neighborhood nan nan
ood_avg_sample_confidence_features 2063.81 1759.3
ood_average_entropy 1.03177 1.03767
id_accuracy 0.923077 0.889306
id_avg_prediction_confidence_aleatoric 0.816697 0.819579
id_avg_prediction_confidence_epistemic 14919.8 14410.5
id_avg_sample_confidence_aleatoric 0.816697 0.819579
id_avg_sample_confidence_epistemic 16556.9 15927.3
id_avg_sample_confidence_features 15835.7 16779.6
id_average_entropy 0.513569 0.501767

Is this experiment exclusive for the With Effect case? I would like to reproduce 3 cases: aleatoric w/Net, Epist w/Net and Epist w/o Net as table 1 in the paper. Could you provide me with some guidance? Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant