Global WordNet Association
lexEN: a re-reviewed English WSD evaluation corpus — proposing a row in Wordnet Annotated Corpora, and a question about machine-relabelled corpora #5
Unanswered
vassiliphilippov
asked this question in
Q&A
Replies: 1 comment
|
These seem like very interesting projects. Separately, we have been trying to update SemCor for the more recent English Wordnet releases here: https://github.com/globalwordnet/semcor/ It would be great to see if we can put our heads together on this. I don't understanding the specific request as this repo doesn't have a contribution guide and is focused on documenting formats rather than describing projects. Perhaps you would like this to listed on the Global Wordnet Association website https://globalwordnet.github.io/? Feel free to make a PR to the relevant repo: https://github.com/globalwordnet/globalwordnet.github.io |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hello,
We have released two English sense-annotation resources and would value the community's view on where, if anywhere, they belong in the GWA's lists.
lexEN
lexEN is an evaluation corpus built on the Raganato et al. (2017) unified all-words framework as corrected by Maru et al. (2022). A panel of models flagged 363 items where automatic predictions disagreed with the gold label; three professional lexicographers then reviewed those items independently — blind to the source label, to which system had flagged the item, and to one another's answers. A two-of-three majority rule, fixed before adjudication, decided each case: 211 labels changed, 56 items removed as unanswerable, 4,861 polysemous instances remain.
The review record is published rather than summarised: each reviewer's per-item choice, the inter-annotator agreement, the written reviewer brief, and the complete item-level diff against the source labels.
The agreement figure is the part we found most interesting, and the reason we are posting rather than just linking. Three professional lexicographers, working from one brief on the same items, reach Fleiss κ = 0.537 at WordNet's fine granularity, rising to 0.740 when senses are grouped more coarsely. That suggests a share of what is usually reported as system error on all-words WSD is disagreement about the sense inventory rather than about the word — which is presumably not news here, but we have not often seen it quantified on the standard evaluation set with professional annotators.
The question
We would like to propose lexEN as a row in the Wordnet Annotated Corpora table, following the process in
Contributing.md. The row would be:Word count is running words in the 948 distinct annotated sentences across 23 documents; taggable and tagged are equal because the set retains only polysemous annotated targets, as with the existing Senseval row.
Separately, and this is the part we are genuinely unsure about. We have also released a version of SemCor in which all 226,036 instances were relabelled by language models rather than by people, alongside the retraining experiments it was built for — retraining BEM, ESCHER and ConSeC on those labels, with no other change, improved them by several F1 points on evaluation sets the relabelling never touched.
Every corpus currently in that table is human-annotated. We did not want to add a machine-annotated resource without asking whether it is in scope at all, and if so whether you would want it flagged as such in a column. We are entirely happy for the answer to be no — it is a different kind of artefact, and we would rather ask than assume.
One small formatting question if lexEN is welcome: no column in that table renders as a link, so the row would mention lexEN without pointing anywhere. Would you like a URL in
other_resources, or is the current terse style deliberate?Thank you for maintaining these lists — they were the first thing we checked when deciding what to release and in what form.
All reactions