query length and node names

[otichy]

Hi, I love your tool and I have two ideas that would really help in my use case, where I use queries with complex regexps and a lot of OR operators.

1. Could the query character limit be changed or turned off (e.g. through settings by users)?
2. Could we have node names in the output? There is node_name in Match annotation, but that outputs the token ID, what I have in mind is if a node is defined as e.g. WORK#/work.*/ this column would return "WORK".
   Thank you!

[matthias-stemmler]

Hi @otichy, thanks for using Annimate and for the ideas! I really appreciate the feedback about practical use cases. 🙂

To address your points:

Query length

The character limit is to prevent crashes due to stack overflows in the underlying graphANNIS library when dealing with very complex queries, so it cannot just be turned off. However, I could increase the stack size and with it the maximal query length and/or use a smarter way to estimate the complexity than just the length.

Could you provide examples for

• (a) the longest query in terms of character count you can think of in any of your use cases, and
• (b) the most complex query in terms of the number of operators such as |, &, . etc., where an | within a regexp doesn't count (only between nodes)?

Node/variable name

This information is currently not available to Annimate because it's not exposed by graphANNIS. This is, in fact, the reason why for "OR" queries you can only select groups of query nodes such as #WORK ... | #2 ... and not the individual nodes.

However, if I understand you correctly, this grouped selection is actually what you need, and you're just missing the information which alternative in the query applied for each match. Or do you need the node name to later filter the list (i.e., throw out everything where the node name isn't "WORK") because you would have preferred selecting only the "WORK" node, but couldn't?

Here an example use case/query would help a lot as well.

I'll see if I can contribute a feature upstream to graphANNIS that will make the necessary info available, so Annimate can then export the node name.

Thanks!

[otichy]

Dear @matthias-stemmler,

Thanks for getting back to me!

Query length: I suspected something like that might be the reason, but could you perhaps allow users to change the limit so they could try what crashes and what does not, with a safe default (e.g. as it is now)?

My use case is perhaps quite specific, but here it is. I am using ANNIS to query the parsed versions of Helsinki Corpora of historical English, such as YCOE, PPCME2 etc. These corpora are not lemmatized and feature a lot of spelling variation. Therefore I
1, need to use long regexs to capture all possible forms, 2, would like to string a number of queries together using OR, but still be able to distinguish which of the forms captures by the regex was which.

Here is an example to query the YCOE corpus for the nouns "bishop" or "brother" with a determiner in the same phrase:

(brotherHEAD#node & brother#/Breþer|breþer|Broðar|broðar|Broð+ar|broð+ar|Broðer|broðer|Broðera|broðera|Broðor|broðor|Broðore|broðore|Broðra|broðra|Broðr+a|broðr+a|Broðran|broðran|Broðre|broðre|Broðres|broðres|Broðro|broðro|Broðron|broðron|Broðru|broðru|Broðrum|broðrum|Broður|broður|Broþer|broþer|Broþor|broþor|Broþra|broþra|Broþran|broþran|Broþron|broþron|Broþru|broþru|Broþrum|broþrum|Broþur|broþur|Bruðor|bruðor|Breðer|breðer|Breðre|breðre/ & brotherPOS#pos=/(N)|(N^.)/ & brotherNUM#pos=/NUM./
& #brother = #brotherPOS
& #brotherHEAD > #brotherPOS
& #brotherHEAD > #brotherNUM)|(bishopHEAD#node & bishop#/Bisceop|bisceop|Bisceopæ|bisceopæ|Bisceopan|bisceopan|Bisceopas|bisceopas|Bisceopæs|bisceopæs|Bisceope|bisceope|Bisceopes|bisceopes|Bisceopos|bisceopos|Bisceopum|bisceopum|Biscep|biscep|Biscepæ|biscepæ|Biscepas|biscepas|Biscepe|biscepe|Biscepes|biscepes|Biscepum|biscepum|Biscop|biscop|Biscopæ|biscopæ|Biscopan|biscopan|Biscopas|biscopas|Biscope|biscope|Biscopes|biscopes|Biscoppe|biscoppe|Biscopum|biscopum|Biscpæ|biscpæ|Biscpas|biscpas|Biscpes|biscpes|Biscpum|biscpum|Biscupe|biscupe|Bissceop|bissceop|Bisscop|bisscop|Bisscopas|bisscopas|Bisscope|bisscope|Bisscopes|bisscopes|Bysceop|bysceop|Bysceopæ|bysceopæ|Bysceopas|bysceopas|Bysceope|bysceope|Bysceopes|bysceopes|Byssceopæ|byssceopæ|Besceop|besceop|Bysscope|bysscope// & bishopPOS#pos=/(N)|(N^.)/ & bishopNUM#pos=/NUM./
& #bishop = #bishopPOS
& #bishopHEAD > #bishop
& #bishopHEAD > #bishopNUM)

Ideally, I would do this for dozens of nouns at the same time, and I would like to know which of the resulting rows refers to which noun (brother, bishop, etc.). In a way I am doing an on the fly lemmatization here.

Best!

Ondrej

[matthias-stemmler]

Hi @otichy,

now I understand your use case - that's a really clever use of named query nodes!

I think a better solution to your problem would be to do the lemmatization as a pre-processing step on the corpus data rather than during export. You can do this using the Annatto tool as follows:

1. Create a workflow file, say lemmatize.toml, like this:

[[import]]
path = "mycorpus.graphml"
format = "graphml"

[import.config]

[[graph_op]]
action = "map"

[[graph_op.config.mapping.rules]]
query = "/Breþer|breþer|Broðar|broðar|Broð+ar|broð+ar|Broðer|broðer|Broðera|broðera|Broðor|broðor|Broðore|broðore|Broðra|broðra|Broðr+a|broðr+a|Broðran|broðran|Broðre|broðre|Broðres|broðres|Broðro|broðro|Broðron|broðron|Broðru|broðru|Broðrum|broðrum|Broður|broður|Broþer|broþer|Broþor|broþor|Broþra|broþra|Broþran|broþran|Broþron|broþron|Broþru|broþru|Broþrum|broþrum|Broþur|broþur|Bruðor|bruðor|Breðer|breðer|Breðre|breðre/"
target = 1
anno = "lemma"
value = "brother"

[[graph_op.config.mapping.rules]]
query = "/Bisceop|bisceop|Bisceopæ|bisceopæ|Bisceopan|bisceopan|Bisceopas|bisceopas|Bisceopæs|bisceopæs|Bisceope|bisceope|Bisceopes|bisceopes|Bisceopos|bisceopos|Bisceopum|bisceopum|Biscep|biscep|Biscepæ|biscepæ|Biscepas|biscepas|Biscepe|biscepe|Biscepes|biscepes|Biscepum|biscepum|Biscop|biscop|Biscopæ|biscopæ|Biscopan|biscopan|Biscopas|biscopas|Biscope|biscope|Biscopes|biscopes|Biscoppe|biscoppe|Biscopum|biscopum|Biscpæ|biscpæ|Biscpas|biscpas|Biscpes|biscpes|Biscpum|biscpum|Biscupe|biscupe|Bissceop|bissceop|Bisscop|bisscop|Bisscopas|bisscopas|Bisscope|bisscope|Bisscopes|bisscopes|Bysceop|bysceop|Bysceopæ|bysceopæ|Bysceopas|bysceopas|Bysceope|bysceope|Bysceopes|bysceopes|Byssceopæ|byssceopæ|Besceop|besceop|Bysscope|bysscope/"
target = 1
anno = "lemma"
value = "bishop"

[[export]]
path = "mycorpus.lemmatized.graphml"
format = "graphml"

[export.config]

Insert the correct path to your corpus file, and add more rules as needed.
2. Run Annatto by calling

annatto run lemmatize.toml

This will produce a new corpus file mycorpus.lemmatized.toml, which you can then import into Annimate.

In your original query, you can then replace the complex

brother#/Breþer|breþer|.../

node with just

brother#lemma="brother"

and produce a column telling you which alternative matched by configuring a "Match annotation" column for the lemma annotation.

This way, you avoid putting the long list of alternative spellings in each query and you can even save the lemmatized corpus and share it with others.

Would that solve your use case?

Best,
Matthias

[otichy]

Dear Matthias,

Thanks for the Annato tutorial, I still haven't gotten to use that, but I definitely will in the future, looks easy to use!

However, I am not sure if it solves this particular issue:

1. even if I shorten query by switching from the token regex to a single lemma, the query length strikes back soon - my ideal query would have ca 40-60 (rather than just two) of those blocks of 3 nodes with one lemma each.
2. for most of the historical corpora I work with, lemmatization is not this straightforward, the token forms specified through regex are often ambiguous and may capture instances of more than one lemma. I usually construct them for max recall and weed them out after export. Therefore, I can't easily first lemmatize and then search.
3. "configuring a "Match annotation" column for the lemma annotation" will output the lemma itself, not the name of the node, right?

Would you consider adding a setting (or a command line parameter) that would allow users to disable the length check with the risk of crashing the app? If not, I can of course try a fork, but I still think this might be useful for others as well.

Thanks!

Ondrej

[matthias-stemmler]

Hi Ondrej,

now I understand that what you're doing is not exactly lemmatization because of the ambiguity you mentioned.

I'll see what I can do to address both of your points. Regarding query length, I'll probably first try to make it "just work" for everyone before I introduce a setting (the fewer knobs users need to understand, the better; we can still introduce a setting as the next step). This should be possible by replacing the length check with an estimate of query complexity by counting operators (meaning you get long regexps for free) while also increasing the limit via the stack size.

Ad 3. You're right, I had assumed that you always use the lemma as the node name, so the two would indeed be the same (as they are in the Annatto example I gave). However, if the token forms are ambiguous and may correspond to more than one lemma, this no longer works.

I'll come back to you when I have something ready. The possibility to export node names will probably take a bit longer because of the graphANNIS change.

Best,
Matthias

[matthias-stemmler]

@otichy I just released version 1.8.1 with improved support for longer queries:

• I increased the stack size and changed the validation so it checks for the number of operators (not counting "OR" within regexps) instead of characters. You can now have up to 4096 operators in a query.
• I also improved the display of query nodes under "Match annotation", "Edge annotation" and "Match in context" so they are easier to read in case of long variable names/query fragments or many alternatives.

This should solve your first point. Please test it and let me know if it works for you.

The second point (exporting query node names) is still open.