Error analysis: visualization of AM dependency trees

When you want to perform error analysis on AM dependency trees, this can be achieved with the following steps:

cd to analyzers/ and call:

python prepare_visualize.py system_output.amconll gold_file.amconll some/directory/

Where system_output.amconll are the parser outputs, gold_file.amconll contains gold AM dependency trees and some/directory/ directory will serve as storage of intermediate files (conll-like files that MaltEval can read). This command intersects the two amconll files and sorts them by sentence length so you can look at short examples first.

If we have placed the MaltEval.jar file inside the analyzers folder, we can call MaltEval.jar to visuzalize the trees and explore in the errors as follows:

java -jar analyzers/MaltEval.jar -g some/directory/gold_file.amconll -s some/directory/system_output.amconll -v 1

If you point your cursor on a word, a little label will pop up, telling you its graph fragment and type. Alternatively, you can change the View Mode to Inline Scroll (default is 'Inline Zoom') in the settings. This View Mode will display the graph fragment and type below each word.

MaltEval is quite powerful, see here for a documentation. There's also a paper with some usage examples from Nilsson and Nivre 2008.

Full default comparison + visualization

cd to analyzers/ and call:

bash am_comparison_full.sh file1 file2 folder/

where file1 and file2 are the original amconll files you want to compare (this script calls the preparation python script for you) and folder is an empty, existing folder where the results will be put.

The bash script is documented/self-explanatory. It prints f-score and LAS/UAS, and will open the visualization tool. More detailed f-scores per dependency labels are printed to a file in the folder.

Where to find the AM-CoNLL files

On the Coli servers, amconll files that are appropriate for the comparative visualization are in /proj/irtg/sempardata/mrp/amconll/PSD (replace PSD by DM or EDS for the other graphbanks). Note that not all frameworks have the same corpus as train and dev set.

Non-comparative visualization

If you don't want to compare with a gold standard but just want to have a look at AM dependency trees, nearly the same procedure can be used but condensed into a bash script (again, cd to analyzers/):

bash visualize_am_trees.sh your_file.amconll

Another visualization script for comparing two amconll files

Limitations of the previously described approach:

• graph fragments are shown in penman notation, which can be hard to read
• comparison fails if corresponding sentences in the two amconll files have different ids, or no ids at all (e.g. AMR vs SDP ids)

The script analyzers/compare_amconll.py can help here. It is not as elaborate as the approach using the MaltEval tool however. This is work in progress. Specifically, a GUI might be used in the future instead of Imagemagick.

Limitations of the this approach:

• if sentence equality is not determined by id but string equality, not all equivalent sentences might be detected as such due to tokenisation or special character handling differences
• no f1 score computation
• no elaborate GUI: No colour-coded diff on the dependency trees, no filter sentences based on several criteria
• visualization size is fixed and quality is low (hard to read for longer sentences).

You need to have graphviz, pdflatex and Imagemagick installed (similar to the compare_mrp.py script). An example command is shown below:

./compare_amconll.py --viz ./example_compare_amconll/system_1.amconll ./example_compare_amconll/system_2.amconll

You can add the --useid option if you know the two amconll files have comparable ids (e.g. comparing SDP amconll files). If ids aren't used to determine equality of sentences, then string equality is used. Note that this doesn't work in all cases. A very basic handling of special characters like quotation marks, parenthesis and hypens was introduced for this purpose.

Furthermore there is a --maxshow INTEGER option which controls how many sentences are shown before the program terminates. Only one sentence is shown in the default case (no --maxshow specified). Sentences are shown in random order (you can edit the random seed in the script).

Get PDF with graph for specific sentence

I've found an interesting sentence with MaltEval, how do I get the corresponding graph?

1. prerequisite: you have to have python, graphviz and mtool installed. Moreover you need to have access to the MRP file containing this sentence (internal note: on coli server /proj/irtg/sempardata/mrp/LDC2019E45/2019/training).
2. get the ID of that sentence (not the one MaltEval shows in the bottom table, as the sentences there are sorted by length and numbered accordingly):
   2.1 You could get the ID from the amconll file: search for the sentence (or some rare token that occurred in the sentence), the line below the sentence should start with #id: followed by the ID. (e.g. the line is #id:22000001)
   2.2 You could also get the ID by searching for the sentence in the original mrp file. Each graph is a json-object with an id as top level key, as well as an input key whose value is the raw sentence. Make sure to choose the id-value on the topmost level of the json object: this id has quotation marks around it, unlike the ids of the individual nodes which happen to be integers.
   2.3 Note 1: hopefully all sentences occur only once in the corpus or duplicates have the same graph. Note 2: In the future, there might be some searchable tsv file or a script for this. TODO make this step easier.
3. then ask mtool to read the mrp file and output the graph with the given ID as dot:

   python /PATH/TO/main.py --strings --id ID_OF_SENTENCE --read mrp --write dot /PATH/TO/MRP/INPUT/mrpfile.mrp /PATH/TO/DOT/OUTPUT/dotfile.dot
   

   You could also use the same command without the --strings option if you prefer character-offsets instead of the string itself.
4. next, call dot with the newly created dot-file as input and specify the output file name:

   dot -Tpdf /PATH/TO/dotfile.dot -o /PATH/TO/outfile.pdf

5. example:

   python ~/Schreibtisch/HiwiAK/mtool/main.py --id 20001001 --read mrp --write dot ./amr/ex_wsj.mrp ./20001001.dot
   dot -Tpdf ./20001001.dot -o ./20001001.pdf