-
Notifications
You must be signed in to change notification settings - Fork 96
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
7ea53a0
commit aecbc0d
Showing
3 changed files
with
55 additions
and
86 deletions.
There are no files selected for viewing
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,58 +1,46 @@ | ||
| \def \codeURL{https://github.com/rescience-c/template} | ||
| % DO NOT EDIT - automatically generated from metadata.yaml | ||
|
|
||
| \def \codeURL{https://github.com/valentinosPariza/Re-Label-Free-XAI} | ||
| \def \codeDOI{} | ||
| \def \codeSWH{swh:1:dir:8aa22d2a6b71c52b0863a06ab40f40ada1ec5355} | ||
| \def \dataURL{} | ||
| \def \dataDOI{} | ||
| \def \editorNAME{} | ||
| \def \editorNAME{Koustuv Sinha,\\ Maurits Bleeker,\\ Samarth Bhargav} | ||
| \def \editorORCID{} | ||
| \def \reviewerINAME{} | ||
| \def \reviewerIORCID{} | ||
| \def \reviewerIINAME{} | ||
| \def \reviewerIIORCID{} | ||
| \def \dateRECEIVED{03 February 2023} | ||
| \def \dateACCEPTED{} | ||
| \def \datePUBLISHED{} | ||
| \def \articleTITLE{Reproducibility Study of \enquote{Label-Free Explainability for Unsupervised Models}} % \centering | ||
| \def \articleTYPE{Replication / ML Reproducibility Challenge 2022} | ||
| \def \articleDOMAIN{} | ||
| \def \dateRECEIVED{04 February 2023} | ||
| \def \dateACCEPTED{19 April 2023} | ||
| \def \datePUBLISHED{20 July 2023} | ||
| \def \articleTITLE{[Re] Reproducibility Study of “Label-Free Explainability for Unsupervised Models”} | ||
| \def \articleTYPE{Replication} | ||
| \def \articleDOMAIN{ML Reproducibility Challenge 2022} | ||
| \def \articleBIBLIOGRAPHY{bibliography.bib} | ||
| \def \articleYEAR{2023} | ||
| \def \reviewURL{} | ||
| \def \articleABSTRACT{ | ||
| \subsubsection*{Scope of Reproducibility} | ||
| In this work, we evaluate the reproducibility of the paper \textit{Label-Free Explainability for Unsupervised Models} by Crabbe and van der Schaar \citep{main_paper}. Our goal is to reproduce the paper's four main claims in a label-free setting:(1) feature importance scores determine salient features of a model's input, (2) example importance scores determine salient training examples to explain a test example, (3) interpretability of saliency maps is hard for disentangled VAEs, (4) distinct pretext tasks don’t have interchangeable representations. | ||
|
|
||
| \subsubsection*{Methodology} | ||
| The authors of the paper provide an implementation in PyTorch for their proposed techniques and experiments. We reuse and extend their code for our additional experiments. Our reproducibility study comes at a total computational cost of 110 GPU hours, using an NVIDIA Titan RTX. | ||
|
|
||
| \subsubsection{Results} | ||
| We reproduced the original paper's work through our experiments. We find that the main claims of the paper largely hold. We assess the robustness and generalizability of some of the claims, through our additional experiments. In that case, we find that one claim is not generalizable and another is not reproducible for the graph dataset. | ||
|
|
||
| \subsubsection*{What was easy} | ||
| The original paper is well-structured. The code implementation is well-organized and with clear instructions on how to get started. This was helpful to understand the paper's work and begin experimenting with their proposed methods. | ||
|
|
||
| \subsubsection*{What was difficult} | ||
| We found it difficult to extrapolate some of the authors' proposed techniques to datasets other than those used by them. Also, we were not able to reproduce the results for one of the experiments. We couldn't find the exact reason for it by running explorative experiments due to time and resource constraints. | ||
|
|
||
| \subsubsection*{Communication with original authors} | ||
| We reached out to the authors once about our queries regarding one experimental setup and to understand the assumptions and contexts of some sub-claims in the paper. We received a prompt response which satisfied most of our questions. | ||
|
|
||
| } | ||
| \def \replicationCITE{} | ||
| \def \replicationBIB{} | ||
| \def \replicationURL{} | ||
| \def \replicationDOI{} | ||
| \def \contactNAME{} | ||
| \def \contactEMAIL{} | ||
| \def \articleKEYWORDS{rescience c, rescience x} | ||
| \def \reviewURL{https://openreview.net/forum?id=qP34dvJpHd} | ||
| \def \articleABSTRACT{Scope of Reproducibility : In this work, we evaluate the reproducibility of the paper Label-Free Explainability for Unsupervised Models by Crabbe and van der Schaar. Our goal is to reproduce the paper's four main claims in a label-free setting:(1) feature importance scores determine salient features of a model's input, (2) example importance scores determine salient training examples to explain a test example, (3) interpretability of saliency maps is hard for disentangled VAEs, (4) distinct pretext tasks don’t have interchangeable representations. Methodology: The authors of the paper provide an implementation in PyTorch for their proposed techniques and experiments. We reuse and extend their code for our additional experiments. Our reproducibility study comes at a total computational cost of 110 GPU hours, using an NVIDIA Titan RTX. Results: We reproduced the original paper's work through our experiments. We find that the main claims of the paper largely hold. We assess the robustness and generalizability of some of the claims, through our additional experiments. In that case, we find that one claim is not generalizable and another is not reproducible for the graph dataset. What was easy: The original paper is well-structured. The code implementation is well-organized and with clear instructions on how to get started. This was helpful to understand the paper's work and begin experimenting with their proposed methods. What was difficult: We found it difficult to extrapolate some of the authors' proposed techniques to datasets other than those used by them. Also, we were not able to reproduce the results for one of the experiments. We couldn't find the exact reason for it by running explorative experiments due to time and resource constraints. Communication with original authors: We reached out to the authors once about our queries regarding one experimental setup and to understand the assumptions and contexts of some sub-claims in the paper. We received a prompt response which satisfied most of our questions.} | ||
| \def \replicationCITE{Crabbe, Jonathan and Mihaela van der Schaar. “Label-Free Explainability for Unsupervised Models.” International Conference on Machine Learning (2022).} | ||
| \def \replicationBIB{Jonathan3:online} | ||
| \def \replicationURL{https://arxiv.org/abs/2203.01928} | ||
| \def \replicationDOI{10.48550/arXiv.2203.01928} | ||
| \def \contactNAME{Valentinos Pariza} | ||
| \def \contactEMAIL{valentinos.pariza@student.uva.nl} | ||
| \def \articleKEYWORDS{rescience c, machine learning, reproducibility, feature importance, example importance, disentangled VAEs, label-free, unsupervised, post-hoc explainability, python} | ||
| \def \journalNAME{ReScience C} | ||
| \def \journalVOLUME{9} | ||
| \def \journalISSUE{2} | ||
| \def \articleNUMBER{} | ||
| \def \articleNUMBER{11} | ||
| \def \articleDOI{} | ||
| \def \authorsFULL{Anonymous Authors} | ||
| \def \authorsABBRV{Anonymous} | ||
| \def \authorsSHORT{Anonymous} | ||
| \def \authorsFULL{Valentinos Pariza et al.} | ||
| \def \authorsABBRV{V. Pariza et al.} | ||
| \def \authorsSHORT{Pariza et al.} | ||
| \title{\articleTITLE} | ||
| \date{} | ||
| \author[1,\orcid{0000-0000-0000-0000}]{Anonymous} | ||
| \affil[1]{Anonymous Institution} | ||
| \author[1,\orcid{0009-0008-3440-9935}]{Valentinos Pariza} | ||
| \author[1,\orcid{0009-0003-1320-3819}]{Avik Pal} | ||
| \author[1,\orcid{0009-0001-1540-3318}]{Madhura Pawar} | ||
| \author[1,\orcid{0009-0002-0762-8004}]{Quim Serra Faber} | ||
| \affil[1]{University of Amsterdam, Amsterdam, The Netherlands} | ||
| \affil[1]{Equal contribution} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters