Improved the model accuracy, corrected some problems with kmain.py, added the ablation study, improved README #11
Conversation
There was a problem hiding this comment.
Thanks a lot for the PR @thevoxium! It would be great if you could please outline the changes you have made in brief and update the README with the improved accuracy. Moreover, would it be possible to also provide the model file?
|
I used google colab to train the model, using tensorflow backend. theano was not using GPU there. That's why I have not given the model file. |
|
But I have experimented with 17 model architectures and this one works the best. |
|
Hi @thevoxium, you may download the model file from colab and send it as a commit here. It is really great that you experimented with a lot of model architectures. Would you mind placing a table in the README file mentioning this ablation study, with the corresponding accuracy percentages? |
|
The model was trained with tensorflow backend, will it be compatible with the theano one? |
|
I will make the required changes in README. Thanks for the suggestion. |
If you are saving the model file with Keras, it should be compatible. Nevertheless, you may send it here and I can help you check that out. |
There was a problem hiding this comment.
@thevoxium The README looks awesome! Thanks a lot for the commit. However, it would be great to have a markdown supported table in place of an image. It would help other contributors easily add rows later.
thevoxium
changed the title
|
1. Changed the model architecture
2. Explained the code of kmain.py in the README file. 3. Explained implementation of LRP. 4. Added the ablation study table describing various architectures, train accuracy, and test accuracy. 5. Added the best model file. |
There was a problem hiding this comment.
Awesome @thevoxium! Great job 😄
Some quick fixes I would like to suggest here. In https://github.com/thevoxium/algo_bias/blob/branch_2/kmain.py#L84 it would be great if you can update it with the latest model file name. Same at https://github.com/thevoxium/algo_bias/blob/branch_2/kmain.py#L122. Accordingly, it would be nice to have the model file placed inside the checkpoints/ directory. Other than that, everything else looks fine to me.
README.md
Outdated
|
|
||
| Next, we ask user for the choice to load previous saved model. Depending on which we load the model or train from beginning. We then ask user the choice to train or test. Depending on the selection, we load a csv file containing data to train or test. We then seperate attributes with target values, namely x_train and y_train. We then compile the model using Adam optimizer and Categorical Crossentropy as loss function. We use Keras' checkpoints to define early stopping (for training) if accuracy does not improve after certain epochs. Then depending on the user choice we train or test the model. If we train the model, then the best performing weights are saved to use them to load the model next time. | ||
|
|
||
| If the user chose to test, we print the model accuracy on the test data. Next we do LRP, Layer-wise Relevance Propagation is a method for understanding deep neural networks by running a backward pass to observe how the individual layers of the program work. This technique brings brings explainability to Deep Neural Networks. More information on LRP is given [here](https://link.springer.com/chapter/10.1007/978-3-030-28954-6_10). |
There was a problem hiding this comment.
Change the line to; Next, we do LRP(Layer-wise Relevance Propagation) which is a....
README.md
Outdated
| # then we pass model inputs, that is x_train | ||
| analysis = analyzer.analyze(x_train) | ||
| ``` | ||
| Then we perform the layer wise layer analysis by blocking off the layers and then performing analysis on them. |
There was a problem hiding this comment.
Change it to -> layer wise analysis....
I have made the required final changes in the README. Please, review and merge the PR. |
|
I am unable to reproduce some of the results. Hence, it would be good to have a fixed random seed for all the experiments as outlined in #12. |
|
This PR also fixes #9. |
|
Can using different backends affect this? Like using tensorflow instead of theano? |
|
If we have all the possible randomly initialised parameter consistent across experiments, ideally there should not be any difference. |
|
I will work on this after issue #6. Thanks for the review. |
I have changed the kmain.py file to include the best working model. It achieved 81.7% accuracy on the test case.
Partially fixes #8.