Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2 errors found: TrueNegatives typos, "get_target" function does not support masking #16

Open
SimoneSartoni opened this issue Dec 12, 2021 · 0 comments

Comments

@SimoneSartoni
Copy link

Hi Luca, I am actually working on Knowledge Tracing for my thesis. I looked at your code to see if I could find a DKT model as baseline.
While trying it I found 2 errors:

  1. distraction error in metrics.py in TruePositives function, which extends tensorflow TrueNegatives (instead of truePositives) resulting in wrong ACC and AUC.
  2. using "get_target" function before computing loss function and metrics does not support the use of automatic masking.
    I know you changed all -1 s to zeros, making it work, but this way the padded values will be considered in your metrics and loss as correct anwered values, modifying real Accuracy and AUC for example. I suggest you to try incorporating "get_target" function in a final custom layer of the model instead, giving as input the target skill to the model too. This way you can use directly a custom layer (remember to define compute_mask function too) to do the reduce_sum over the skill_true array. I have done it and values of AUC, ACC and loss change a lot in the case you have very different length of the sequences as input (due to large padding).

Despite this errors thanks for the code available, I used to understand better how to implement my own DKT as baseline
Bye ;)

@SimoneSartoni SimoneSartoni changed the title 2 errors found: TrueNegatives typos, "get_target" function does not support 2 errors found: TrueNegatives typos, "get_target" function does not support masking Dec 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant