Adding sentiment to corpora

[TomazErjavec]

In ParlaCAP sentiment scores and labels will be added to sentences and utterances and this issue serves to document the needed changes:

• how to encode sentiment
• changes to the schema and documentation
• changes to the conversion programs
• changes to the registry files

Note that I have made a new milestone ParlaCAP (and assigned this issue to it) which should be used for issues pertaining to the project.
The debate here should be releveant to @matyaskopp, @katjameden, @nljubesi.

[TomazErjavec]

For SI we have already added sentiment to <u> and <s> as well as modifying the schema and conversion to vertical file + adding a new sentiment taxonomy, currently local to SI (this is a draft and might well change, esp. the description part).

In short:

• the sentiment label is encoded in u/@ana and s/@ana, and makes reference to the sentiment taxonomy and uses the extended prefix senti.
• the sentiment score is encoded in u/@n and s/@n; this is not a very good solution, as @n is a very general attribute but I currently do not have a better idea how to preserve the score in the encoding.

E.g.

ParlaMint/Samples/ParlaMint-SI/2007/ParlaMint-SI_2007-11-28-SDZ4-Izredna-30.ana.xml

Line 144 in 6dd236d

<u who="#PečeSašo" xml:id="ParlaMint-SI_2007-11-28-SDZ4-Izredna-30.ana.u1" ana="#chair senti:neupos" n="3.16">

ParlaMint/Samples/ParlaMint-SI/2007/ParlaMint-SI_2007-11-28-SDZ4-Izredna-30.ana.xml

Line 146 in 6dd236d

<s xml:id="ParlaMint-SI_2007-11-28-SDZ4-Izredna-30.ana.seg1.1" ana="senti:mixpos" n="3.68">

The new ParlMint-SI is also available via the concordancer for testing.

The documentation and other conversions (in particular, to TSV) still need to be implemented.

The problem right now is that I've added the new taxonomy to all the relevant programs that deal with taxonomies but now the CI validation complains that this taxonomy is missing from all the corpora (except SI). @matyaskopp, how best to solve this? Make (somehow) the taxonomy optional or (manually) insert the taxonomy, it's XInclude and prefixDef into all the samples? Or something else?

[TomazErjavec]

I was unhappy with using @n for the sentiment score, as @n is a very general attribute.
I am now considering an alternative, namely using <measure/> for encoding everything to do with sentiment, so we would have e.g.

<u who="#BernikJožef"
   xml:id="ParlaMint-SI_2000-10-27-SDZ3-Redna-01.ana.u31" ana="#chair">
   <measure type="sentiment" quantity="2.47" ana="senti:neuneg"/>
   <seg xml:id="ParlaMint-SI_2000-10-27-SDZ3-Redna-01.ana.seg110">
      <s xml:id="ParlaMint-SI_2000-10-27-SDZ3-Redna-01.ana.seg110.1">
         <measure type="sentiment" quantity="4.08" ana="senti:mixpos"/>
         <w xml:id="ParlaMint-SI_2000-10-27-SDZ3-Redna-01.ana.seg110.1.1"
            msd="UPosTag=NOUN|Case=Nom|Gender=Fem|Number=Sing"
            ana="mte:Ncfsn"
            lemma="hvala">Hvala</w>

If I don't get any flames for this suggestion I will redo the SI corpus, schema and scripts to take into account this new encoding.

[nljubesi]

Besides discussing how sentiment is to be encoded, I would like to touch upon which sentiment should be encoded.

Are you still of the position of encoding

• predicted sentiment on sentence level?
• length-averaged sentiment on utterance level?

You know my position that the second one is of questionable usability (it tends to group around neutral sentiment due to the nature of the speeches, most parts being neutral, some positive, some negative, the last two cancelling themselves out).

I am kind-of ok with encoding both, but am still of the nasty opinion that it might not be smart to add utterance-level sentiment to concordancers. This one might seem easier to be used, but does not encode most of the sentiment in the corpus, would therefore be much less informative for downstream analyses.

[TomazErjavec]

Yes, we definitelly want utterance level sentiment for ParlaMint-SI.
However, @katjameden did not use heuristics to determine u-level sentiment from s-level ones, rather, she used the manually annotated u-level sentiment for SI (with better agreement scores than s-level had!) to train a random forest model (your idea actually, I think), and the prediciton results are I think also better than the s-level ones. So....
(The downside of this approach is, of course, that we have u-level senti only for SI).

Also, s-level sentiment will still be encoded in the corpus and concordancers, and people are free to use it if they prefer.

[matyaskopp]

I was unhappy with using @n for the sentiment score, as @n is a very general attribute. I am now considering an alternative, namely using <measure/> for encoding everything to do with sentiment, so we would have e.g.

<u who="#BernikJožef"
   xml:id="ParlaMint-SI_2000-10-27-SDZ3-Redna-01.ana.u31" ana="#chair">
   <measure type="sentiment" quantity="2.47" ana="senti:neuneg"/>
   <seg xml:id="ParlaMint-SI_2000-10-27-SDZ3-Redna-01.ana.seg110">
      <s xml:id="ParlaMint-SI_2000-10-27-SDZ3-Redna-01.ana.seg110.1">
         <measure type="sentiment" quantity="4.08" ana="senti:mixpos"/>
         <w xml:id="ParlaMint-SI_2000-10-27-SDZ3-Redna-01.ana.seg110.1.1"
            msd="UPosTag=NOUN|Case=Nom|Gender=Fem|Number=Sing"
            ana="mte:Ncfsn"
            lemma="hvala">Hvala</w>

If I don't get any flames for this suggestion I will redo the SI corpus, schema and scripts to take into account this new encoding.

Sorry, I missed this conversation; I do not like @n either.

Using measure is better, but still not sure if it was initially meant to use it this way for annotations.
I was not able to find a better solution for annotating various elements with the category and numbers describing their affiliation with the category.
I have two notes:

• maybe add the attribute corresp that will bind strictly the annotated element
• it is not obvious the scale of the numbers quantity="2.47" means nothing to me, if it is percentage, then maybe also consider adding unit

[TomazErjavec]

Using measure is better, but still not sure if it was initially meant to use it this way for annotations. I was not able to find a better solution for annotating various elements with the category and numbers describing their affiliation with the category.

I agree that it was not really meant for this, on the other hand, I was not able to find any other solution either, so, I guess we go with measure.

I have two notes:

* maybe add the attribute `corresp` that will bind strictly the annotated element

You mean like

<measure type="sentiment" quantity="2.47" ana="senti:neuneg" corresp="#ParlaMint-SI_2000-10-27-SDZ3-Redna-01.ana.u31"/>

We of course can do that, although I don't really see the point, given that measure is included in the annotated element.

* it is not obvious the scale of the numbers `quantity="2.47"` means nothing to me, if it is percentage, then maybe also consider adding `unit`

I did, but there is no obvious unit here - it's simply a scale from 0-5. I think this should be documented in the corpus TEI header somewhere.

[matyaskopp]

We of course can do that, although I don't really see the point, given that measure is included in the annotated element.

I was thinking about a possible situation, where we want to annotate w with measure element. w does not allow measure child, so it would be necessary to place it outside, so corresp attribute would be the must.

I am saying it from my experience with anchor in audio alignment (I did not use the corresp in ParCzech, which I now regret)

I did, but there is no obvious unit here - it's simply a scale from 0-5. I think this should be documented in the corpus TEI header somewhere.

An option is to use unitRef but it would require The Unit Declaration.
However, I am unsure if I have enough courage to introduce this, as I don’t know what else might be required to manage.

[TomazErjavec]

OK, I changed the RNG + TEI schema to support measure and will also use @corresp, indeed, better safe than sorry.
Will now also change the conversion to vertical so that this change is reflected there as well.
Note that we have another issue about how to get the sentiment annotation into metadata files, cf. #897.

As for unitRef, wow, I wasn't even aware that this exists.. It is tempting to use it, but, indeed, it might be more complicated than it seems, also, introducing a whole new division into the teiHeader just for one annotation might be excessive.

I'd prefer to ponder this a bit, as, anyway, sentiment still needs to be introduced into the Guidelines, as well as metadata as above.

[nljubesi]

Yes, we definitelly want utterance level sentiment for ParlaMint-SI. However, @katjameden did not use heuristics to determine u-level sentiment from s-level ones, rather, she used the manually annotated u-level sentiment for SI (with better agreement scores than s-level had!) to train a random forest model (your idea actually, I think), and the prediciton results are I think also better than the s-level ones. So.... (The downside of this approach is, of course, that we have u-level senti only for SI).

Also, s-level sentiment will still be encoded in the corpus and concordancers, and people are free to use it if they prefer.

s-level sentiment is the only thing that I would trust. For SI, 1 out of 29 datasets, u-level sentiment is probably ok, but I still have reservations. For u-level sentiment in 28 out of 29 datasets I personally do not know how I would use it, except as a much weaker data selection method than s-level sentiment. I simply do not see a viable use case in concordancers where u-level sentiment should be preferred to s-level sentiment.

I most strongly suggest s-level sentiment is considered primary, and u-level sentiment might "still be encoded" if you are sure enough that it will not be used except if people know what they are doing with it (me still not seeing a viable use case).

@katjameden is it maybe the perfect time for you to offer a corpus-based analysis of sentiment via different criteria? It would be very interesting to see how conclusions from s-level u-level-machinelearning u-level-heuristic compare. I think this would be a very decent chapter / section in your PhD. Maybe you planned such a thing?