IIITBH-IITP

[Saichethan]

https://github.com/WING-NUS/scisumm-corpus/blob/master/2018-evaluation-script/program/task1_eval.py replace "annv3.csv" to ".csv"

if anyone is facing WordNet error

/scisumm-corpus/2018-evaluation-script/program/rouge/data$ rm WordNet-2.0.exc.db

/scisumm-corpus/2018-evaluation-script/program/rouge/data$ ./WordNet-2.0-Exceptions/buildExeptionDB.pl ./WordNet-2.0-Exceptions ./smart_common_words.txt ./WordNet-2.0.exc.db

3. Still cumulative Task1 B results are not present in scores.txt

[dennlinger]

According to the script program/run.sh, results for the overall tasks should be present in ./input/res/Task1/, however, there is no such folder generated currently. Am I right in assuming that this would contain the final scores used for evaluation?

[Saichethan]

@dennlinger results(here I mean scores) will be present in the output directory. Whereas ./input/res/Task1/ contains our submitted CSV files

[dennlinger]

No, ./input/ref/ contains the input files. However, if you look at line 3 in the aforementioned script, it states [...]/input/res/, which is different (s, not f). Further, script task1_eval.py, which is called indirectly via program/run.sh->program/task1.sh->program/task1_eval.py, then uses this specific input parameter to write the overall evaluation of the task (see ll. 489-501 in script task1_eval.py), right? According to my understanding, the written values consider the results across all files, instead of the currently generated output over one file only.

[Saichethan]

For a single run, these two results are written in score.txt

and

which of these should I report system description paper?

[cmkumar87]

@dennlinger @Saichethan Thanks for this discussion! I had made the change in rename.py from .txt to .csv. I won't merge this PR to an evaluation script to be fair to all participants.

[cmkumar87]

@dennlinger It is very unlikely that we have errors in how scores are computed since the script has not been changed since 2018. So please double check and confirm if you are reporting a bug (as a separate github issue). Note that we check the system output files against multiple reference / ground truth files for each reference document (3 annotators).

[Saichethan]

@cmkumar87 @dennlinger My intentions are just to cross-check if I am doing the evaluation correctly...

[cmkumar87]

@Saichethan @dennlinger Is this change required to run the script?
https://github.com/WING-NUS/scisumm-corpus/blob/master/2018-evaluation-script/program/task1_eval.py replace "annv3.csv" to ".csv"

If so I will make it to the repo; please run the script from the repo and not the modified version from your fork.

I am not merging this PR for fairness to other contestants since this is from a participant. thanks!

[cmkumar87]

@Saichethan Your PR is not going into your evaluation folder but to 2018-evaluation-scripts. I am not able to merge any pull request outside of the your evaluation folder. Also from the PR I am not able to find scores for all the runs in your submission. Please advise and raise a new PR

[Saichethan]

@cmkumar87 please check once now, if the updated version is in alignment with your requirements?
or should I close this PR and reopen?