Error analysis: visualization of AM dependency trees

When you want to perform error analysis on AM dependency trees, this can be achieved with the following steps:

cd to analyzers/ and call:

python prepare_visualize.py system_output.amconll gold_file.amconll some/directory/

Where system_output.amconll are the parser outputs, gold_file.amconll contains gold AM dependency trees and some/directory/ directory will serve as storage of intermediate files (conll-like files that MaltEval can read). This command intersects the two amconll files and sorts them by sentence length so you can look at short examples first.

If we have placed the MaltEval.jar file inside the analyzers folder, we can call MaltEval.jar to visuzalize the trees and explore in the errors as follows:

java -jar analyzers/MaltEval.jar -g some/directory/gold_file.amconll -s some/directory/system_output.amconll -v 1

If you point your cursor on a word, a little label will pop up, telling you its graph fragment and type. Alternatively, you can change the View Mode to Inline Scroll (default is 'Inline Zoom') in the settings. This View Mode will display the graph fragment and type below each word.

MaltEval is quite powerful, see here for a documentation. There's also a paper with some usage examples from Nilsson and Nivre 2008.

Full default comparison + visualization

cd to analyzers/ and call:

bash am_comparison_full.sh file1 file2 folder/

where file1 and file2 are the original amconll files you want to compare (this script calls the preparation python script for you) and folder is an empty, existing folder where the results will be put.

The bash script is documented/self-explanatory. It prints f-score and LAS/UAS, and will open the visualization tool. More detailed f-scores per dependency labels are printed to a file in the folder.

Where to find the AM-CoNLL files

On the Coli servers, amconll files that are appropriate for the comparative visualization are in /proj/irtg/sempardata/mrp/amconll/PSD (replace PSD by DM or EDS for the other graphbanks). Note that not all frameworks have the same corpus as train and dev set.

Non-comparative visualization

If you don't want to compare with a gold standard but just want to have a look at AM dependency trees, nearly the same procedure can be used but condensed into a bash script (again, cd to analyzers/):

bash visualize_am_trees.sh your_file.amconll

Another visualization script for comparing two amconll files

Limitations of the previously described approach:

• graph fragments are shown in penman notation, which can be hard to read
• comparison fails if corresponding sentences in the two amconll files have different ids, or no ids at all (e.g. AMR vs SDP ids)

The script analyzers/compare_amconll.py (and also analyzers/compare_amconll_qtgui.py) can help here. It is not as elaborate as the approach using the MaltEval tool however.

Limitations of the this approach:

• if sentence equality is not determined by id but string equality, not all equivalent sentences might be detected as such due to tokenisation or special character handling differences
• no f1 score computation
• no elaborate GUI: No colour-coded diff on the dependency trees, no filter sentences based on several criteria
• for long sentences, graph constants might be too small to read (would need some sort of zoom)
• visualization not possible if graph_dependency_parser.components.dataset_readers.amconll_tools.AmSentence.to_tex_svg fails to produce an output for some reason

You need to have graphviz, pdflatex, Inkscape and PyQt5 (pip install pyqt5) installed.

An example command is shown below:

./compare_amconll.py ./example_compare_amconll/system_1.amconll ./example_compare_amconll/system_2.amconll

You can add the --useid option if you know the two amconll files have comparable ids (e.g. comparing SDP amconll files). If ids aren't used to determine equality of sentences, then sentence string equality is used. Note that this doesn't work in all cases. A very basic handling of special characters like quotation marks, parenthesis and hypens was introduced for this purpose. Sentences are shown in random order (you can edit the random seed in the script).

The script will first of all compute the intersection of both amconll files, so compute the set of sentences that appear in both files (id-based or sentence string-based, see above). A filter (defined by the function is_interesting) is then applied on the intersection. By default, this filter excludes sentences which are either too short (up to 5 tokens) or too long (15 or more tokens), but feel free to change this.

If there are still sentence left, the sentences are ordered and then a window (GUI) will appear with the following content:

• top left: current sentence number and total sentences (after filtering)
• top center: the sentence string
• top right: buttons to move to previous or next sentence
• middle and bottom: AM dependency trees of the current sentence according to the first and second file.
• On the right hand side, for each of the two images a button will allow you to maximize this image (in a separate window which will block the main window from responding until the single-image view is closed)

Note: there is also a variant similar to the compare_mrp.py script which can be envoked using --viz plus --nogui option, then you need Imagemagick (PyQt is not required then) to be installed. In this case, you can add the --maxshow INTEGER option which controls how many sentences are shown before the program terminates. Only one sentence is shown in the default case (no --maxshow specified). If you don't want any visualization, call the script with --nogui and a suitable value for --maxshow, then it will just print sentences (sentence string) to stdout.

Get PDF with graph for specific sentence

I've found an interesting sentence with MaltEval, how do I get the corresponding graph?

1. prerequisite: you have to have python, graphviz and mtool installed. Moreover you need to have access to the MRP file containing this sentence (internal note: on coli server /proj/irtg/sempardata/mrp/LDC2019E45/2019/training).
2. get the ID of that sentence (not the one MaltEval shows in the bottom table, as the sentences there are sorted by length and numbered accordingly):
   2.1 You could get the ID from the amconll file: search for the sentence (or some rare token that occurred in the sentence), the line below the sentence should start with #id: followed by the ID. (e.g. the line is #id:22000001)
   2.2 You could also get the ID by searching for the sentence in the original mrp file. Each graph is a json-object with an id as top level key, as well as an input key whose value is the raw sentence. Make sure to choose the id-value on the topmost level of the json object: this id has quotation marks around it, unlike the ids of the individual nodes which happen to be integers.
   2.3 Note 1: hopefully all sentences occur only once in the corpus or duplicates have the same graph. Note 2: In the future, there might be some searchable tsv file or a script for this. TODO make this step easier.
3. then ask mtool to read the mrp file and output the graph with the given ID as dot:

   python /PATH/TO/main.py --strings --id ID_OF_SENTENCE --read mrp --write dot /PATH/TO/MRP/INPUT/mrpfile.mrp /PATH/TO/DOT/OUTPUT/dotfile.dot
   

   You could also use the same command without the --strings option if you prefer character-offsets instead of the string itself.
4. next, call dot with the newly created dot-file as input and specify the output file name:

   dot -Tpdf /PATH/TO/dotfile.dot -o /PATH/TO/outfile.pdf

5. example:

   python ~/Schreibtisch/HiwiAK/mtool/main.py --id 20001001 --read mrp --write dot ./amr/ex_wsj.mrp ./20001001.dot
   dot -Tpdf ./20001001.dot -o ./20001001.pdf