Experiment working, but some parsing errors in the document display in GUI

[agolo-alan-hogue]

Hello!

Happy to say I managed to run a full eval and view it in the GUI. Overall things are looking great.

There's just one thing, which might be an issue with my data but nevertheless might be worth looking into.

Most documents display properly, but a few have things like what is shown below (text and image).

Below that I have included the corresponding line from the benchmark.jsonl file.

Any idea what is causing this? Thanks!

1
A helicopter accident in northeastern 

class="annotation gt unknown lowlight beginning annotation_id_0_1_2">Syria over the weeken
Groundtruth: [UNKNOWN]
Note: Entity not found in the knowledge base
d left 22 

American

 service member

s injured, the U.S.

 mili

n class="annotation gt unknown lowlight beginning annotation_id_0_1_7">tary said Tuesday.

The milit
Groundtruth: [UNKNOWN]
Note: Entity not found in the knowledge base
ary statement said tha

t the cause of the accident was under investigatio

n and that no enemy fire involved.“A helicopter mishap in northeastern 

Syria

 resulted in the injuries of various degrees of 22 U.S.

 service members,” US Central Command

 said. “No enemy fire was reported.”

"The service members are receiving treatment for their injuries and 10 have been evacuated to higher care facilities," Centcom

 added in a statement.

A spokesman for the U.S.

-backed Syria

n Kurdish

 forces did not immediately respond to an Associated Press

 request for comment.

{"id": 1, "title": "1", "text": "A helicopter accident in northeastern Syria over the weekend left 22 American service members injured, the U.S. military said Tuesday.\n\n\nThe military statement said that the cause of the accident was under investigation and that no enemy fire involved.\u201cA helicopter mishap in northeastern Syria resulted in the injuries of various degrees of 22 U.S. service members,\u201d US Central Command said. \u201cNo enemy fire was reported.\u201d\n\n\n\"The service members are receiving treatment for their injuries and 10 have been evacuated to higher care facilities,\" Centcom added in a statement.\n\n\nA spokesman for the U.S.-backed Syrian Kurdish forces did not immediately respond to an Associated Press request for comment.\n\n\nThere are at least 900 U.S. forces in Syria on average, along with an undisclosed number of contractors. U.S. special operations forces also move in and out of the country, but are usually in small teams and are not included in the official count.\n\n\nU.S. forces have been in Syria since 2015 to assist the Kurdish-led Syrian Forces in the fight against the militant Islamic State group. Since the extremist group was defeated in Syria in March 2019, U.S. troops have been trying to prevent any comeback by IS, which swept through Iraq and Syria in 2014, taking control of large swaths of territory.\n\n\nHowever, IS sleeper cells remain a threat. There are also about 10,000 IS fighters held in detention facilities in Syria and tens of thousands of their family members living in two refugee camps in the country's northeast.\n\n\nOver the past years, U.S. troops have been subjected to attacks carried out by IS members and Iran-backed fighters there. In late March, a drone attack on a U.S. base killed a contractor and wounded five American troops and another contractor. In retaliation, U.S. fighter jets struck several locations around the eastern province of Deir el-Zour, which borders Iraq.\n\n\nU.S. Defense Secretary Lloyd Austin said at the time that the strikes were a response to the drone attack as well as a series of recent attacks against U.S.-led coalition forces in Syria by groups affiliated with Iran\u2019s Revolutionary Guard.\n\n\nIn a related development, Syrian Kurdish-led authorities announced Saturday that hundreds of IS fighters held in prisons around the region will be put on trial after their home countries refused to repatriate them.", "evaluation_span": [0, 2357], "labels": [{"id": 0, "span": [25, 43], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [3], "optional": false, "type": "Q27096213|Q43229"}, {"id": 1, "span": [276, 294], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [16], "optional": false, "type": "Q27096213|Q43229"}, {"id": 2, "span": [43, 48], "entity_id": "Q858", "name": "Syria", "parent": 21, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 3, "span": [30, 35], "entity_id": "Q858", "name": "Syria", "parent": 0, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 4, "span": [608, 613], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 5, "span": [742, 747], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 6, "span": [979, 984], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 7, "span": [1022, 1027], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 8, "span": [107, 120], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 9, "span": [727, 738], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 10, "span": [954, 965], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 11, "span": [1154, 1165], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 12, "span": [1551, 1562], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 13, "span": [214, 218], "entity_id": "Q30", "name": "United States of America", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 14, "span": [345, 349], "entity_id": "Q30", "name": "United States of America", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 15, "span": [596, 600], "entity_id": "Q30", "name": "United States of America", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 16, "span": [282, 286], "entity_id": "Q30", "name": "United States of America", "parent": 1, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 17, "span": [809, 813], "entity_id": "Q30", "name": "United States of America", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 18, "span": [164, 168], "entity_id": "Q30", "name": "United States of America", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 19, "span": [368, 386], "entity_id": "Q1476046", "name": "United States Central Command", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 20, "span": [544, 551], "entity_id": "Q1476046", "name": "United States Central Command", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 21, "span": [38, 59], "entity_id": "Unknown", "name": "UnknownNoMapping", "parent": null, "children": [2], "optional": false, "type": "OTHER"}, {"id": 22, "span": [664, 680], "entity_id": "Q40469", "name": "Associated Press", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 23, "span": [116, 146], "entity_id": "Unknown", "name": "UnknownNoMapping", "parent": null, "children": [], "optional": false, "type": "OTHER"}, {"id": 24, "span": [67, 92], "entity_id": "Unknown", "name": "UnknownNoMapping", "parent": null, "children": [], "optional": false, "type": "OTHER"}, {"id": 25, "span": [2169, 2199], "entity_id": "Unknown", "name": "UnknownNoMapping", "parent": null, "children": [], "optional": false, "type": "OTHER"}, {"id": 26, "span": [1070, 1089], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 27, "span": [1210, 1212], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 28, "span": [1314, 1316], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 29, "span": [1376, 1378], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 30, "span": [1609, 1611], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 31, "span": [2236, 2238], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 32, "span": [1234, 1238], "entity_id": "Q796", "name": "Iraq", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 33, "span": [1892, 1896], "entity_id": "Q796", "name": "Iraq", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 34, "span": [1624, 1628], "entity_id": "Q794", "name": "Iran", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 35, "span": [1864, 1876], "entity_id": "Q239097", "name": "Deir ez-Zor", "parent": null, "children": [], "optional": false, "type": "Q27096213"}, {"id": 36, "span": [1923, 1935], "entity_id": "Q941013", "name": "Lloyd Austin", "parent": null, "children": [], "optional": false, "type": "Q215627"}, {"id": 37, "span": [2113, 2139], "entity_id": "Q271110", "name": "Islamic Revolutionary Guard Corps", "parent": null, "children": [], "optional": false, "type": "Q43229"}]}

[flackbash]

Hi, there seem to be two problems:

The first one is that there is non-nested overlapping ground truth label included in your benchmark, namely this one:
{"id": 23, "span": [116, 146], "entity_id": "Unknown", "name": "UnknownNoMapping", "parent": null, "children": [], "optional": false, "type": "OTHER"}
which overlaps with this label:
{"id": 8, "span": [107, 120], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}
Nested labels can be handled by ELEVANT (e.g. "[northeastern [Syria]]"), but not non-nested overlapping labels ("[U.S. mili{tary] said Tuesday. The milit}ary ..."). That label anyhow does seem to be a mistake in the data or in the parsing of the benchmark.

The other problem is that the benchmark labels (and label ids) are not sorted by their corresponding span start.
This should actually be automatically the case when using ELEVANT to add a benchmark in NIF format using the add_benchmark.py script (due to this line here). Is that how you added the benchmark or did you use another procedure?

I fixed these two problems and the benchmark was displayed just fine:

{"id": 1, "title": "1", "text": "A helicopter accident in northeastern Syria over the weekend left 22 American service members injured, the U.S. military said Tuesday.\n\n\nThe military statement said that the cause of the accident was under investigation and that no enemy fire involved.\u201cA helicopter mishap in northeastern Syria resulted in the injuries of various degrees of 22 U.S. service members,\u201d US Central Command said. \u201cNo enemy fire was reported.\u201d\n\n\n\"The service members are receiving treatment for their injuries and 10 have been evacuated to higher care facilities,\" Centcom added in a statement.\n\n\nA spokesman for the U.S.-backed Syrian Kurdish forces did not immediately respond to an Associated Press request for comment.\n\n\nThere are at least 900 U.S. forces in Syria on average, along with an undisclosed number of contractors. U.S. special operations forces also move in and out of the country, but are usually in small teams and are not included in the official count.\n\n\nU.S. forces have been in Syria since 2015 to assist the Kurdish-led Syrian Forces in the fight against the militant Islamic State group. Since the extremist group was defeated in Syria in March 2019, U.S. troops have been trying to prevent any comeback by IS, which swept through Iraq and Syria in 2014, taking control of large swaths of territory.\n\n\nHowever, IS sleeper cells remain a threat. There are also about 10,000 IS fighters held in detention facilities in Syria and tens of thousands of their family members living in two refugee camps in the country's northeast.\n\n\nOver the past years, U.S. troops have been subjected to attacks carried out by IS members and Iran-backed fighters there. In late March, a drone attack on a U.S. base killed a contractor and wounded five American troops and another contractor. In retaliation, U.S. fighter jets struck several locations around the eastern province of Deir el-Zour, which borders Iraq.\n\n\nU.S. Defense Secretary Lloyd Austin said at the time that the strikes were a response to the drone attack as well as a series of recent attacks against U.S.-led coalition forces in Syria by groups affiliated with Iran\u2019s Revolutionary Guard.\n\n\nIn a related development, Syrian Kurdish-led authorities announced Saturday that hundreds of IS fighters held in prisons around the region will be put on trial after their home countries refused to repatriate them.", "evaluation_span": [0, 2357], "labels": [{"id": 0, "span": [25, 43], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [1], "optional": false, "type": "Q27096213|Q43229"}, {"id": 1, "span": [30, 35], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 2, "span": [38, 59], "entity_id": "Unknown", "name": "UnknownNoMapping", "parent": null, "children": [3], "optional": false, "type": "OTHER"}, {"id": 3, "span": [43, 48], "entity_id": "Q858", "name": "Syria", "parent": 2, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 4, "span": [67, 92], "entity_id": "Unknown", "name": "UnknownNoMapping", "parent": null, "children": [], "optional": false, "type": "OTHER"}, {"id": 5, "span": [107, 120], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 7, "span": [164, 168], "entity_id": "Q30", "name": "United States of America", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 8, "span": [214, 218], "entity_id": "Q30", "name": "United States of America", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 9, "span": [276, 294], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [10], "optional": false, "type": "Q27096213|Q43229"}, {"id": 10, "span": [282, 286], "entity_id": "Q30", "name": "United States of America", "parent": 9, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 11, "span": [345, 349], "entity_id": "Q30", "name": "United States of America", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 12, "span": [368, 386], "entity_id": "Q1476046", "name": "United States Central Command", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 13, "span": [544, 551], "entity_id": "Q1476046", "name": "United States Central Command", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 14, "span": [596, 600], "entity_id": "Q30", "name": "United States of America", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 15, "span": [608, 613], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 16, "span": [664, 680], "entity_id": "Q40469", "name": "Associated Press", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 17, "span": [727, 738], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 18, "span": [742, 747], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 19, "span": [809, 813], "entity_id": "Q30", "name": "United States of America", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 20, "span": [954, 965], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 21, "span": [979, 984], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 22, "span": [1022, 1027], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 23, "span": [1070, 1089], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 24, "span": [1154, 1165], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 25, "span": [1210, 1212], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 26, "span": [1234, 1238], "entity_id": "Q796", "name": "Iraq", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 27, "span": [1314, 1316], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 28, "span": [1376, 1378], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 29, "span": [1551, 1562], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 30, "span": [1609, 1611], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 31, "span": [1624, 1628], "entity_id": "Q794", "name": "Iran", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 32, "span": [1864, 1876], "entity_id": "Q239097", "name": "Deir ez-Zor", "parent": null, "children": [], "optional": false, "type": "Q27096213"}, {"id": 33, "span": [1892, 1896], "entity_id": "Q796", "name": "Iraq", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 34, "span": [1923, 1935], "entity_id": "Q941013", "name": "Lloyd Austin", "parent": null, "children": [], "optional": false, "type": "Q215627"}, {"id": 35, "span": [2113, 2139], "entity_id": "Q271110", "name": "Islamic Revolutionary Guard Corps", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 36, "span": [2169, 2199], "entity_id": "Unknown", "name": "UnknownNoMapping", "parent": null, "children": [], "optional": false, "type": "OTHER"}, {"id": 37, "span": [2236, 2238], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}]}

I realized that other benchmark readers don't enforce this sorting, which is something I need to fix.

Thank you for reporting this in any case!

[flackbash]

The sorting of the ground truth labels for two other benchmark readers was implemented in 59f97bf. I'll close this issue. Feel free to reopen it in case your problem was not solved.