Error analysis: visualization of AM dependency trees

When you want to perform error analysis on AM dependency trees, here are some scripts you can use (all in am-parser/analyzers) and some further remarks:

Table of contents:

1. The MaltEval approach: am_comparison_full.sh and visualize_am_trees.sh:
   for comparative or single framework analysis, using the MaltEval tool.
   Pro: F1 score computation and colorized diff, elaborate GUI.
   Con: for comparative case need corpora with matching IDs per sentence, graph constants only in Penman notation.
2. The own GUI approach: compare_amconll.py (compare_amconll_qtgui.py):
   for comparative (not single) framework analysis, using PyQt (and Pdflatex, Graphviz and Inkscape for drawing). A much more simplistic version using ImageMagick instead of PyQt is also available.
   Pro: don't need matching IDs, graph constants drawn using graphviz.
   Con: no single framework view, not many options (less elaborate GUI compared to MaltEval), no F1 score computation or colorized diff.
3. One AM dependency tree to LaTeX: am_dep_to_latex.py:
   based on the sentence ID, produces the LaTeX and dot files for a specified AM dependency tree.
4. Get PDF with graph for one sentence: step by step description about how to visualize the graph itself (not its AM dependency tree). Prerequisite: appropriate MRP file exists.
   See also Visualizing graphbanks for visualizing all graphs in a certain file (not only for MRP files) and the compare_mrp.py script.
   Note would be useful to evaluate an AM dependency tree stored in an AMCoNLL file to the final graph: maybe have a look at evaluate the AM-CoNLL trees to graphs or here.
5. Internal note: Location on server where to find AMCoNLL files on the coli servers

1. The MaltEval approach

First, cd to analyzers/ and call:

python prepare_visualize.py system_output.amconll gold_file.amconll some/directory/

Where system_output.amconll are the parser outputs, gold_file.amconll contains gold AM dependency trees and some/directory/ directory will serve as storage of intermediate files (conll-like files that MaltEval can read). This command intersects the two amconll files and sorts them by sentence length so you can look at short examples first.

If we have placed the MaltEval.jar file inside the analyzers folder, we can call MaltEval.jar to visuzalize the trees and explore in the errors as follows:

java -jar analyzers/MaltEval.jar -g some/directory/gold_file.amconll -s some/directory/system_output.amconll -v 1

If you point your cursor on a word, a little label will pop up, telling you its graph fragment and type. Alternatively, you can change the View Mode to Inline Scroll (default is 'Inline Zoom') in the settings. This View Mode will display the graph fragment and type below each word.

MaltEval is quite powerful, see here for a documentation. There's also a paper with some usage examples from Nilsson and Nivre 2008.

Full default comparison + visualization

cd to analyzers/ and call:

bash am_comparison_full.sh file1 file2 folder/

where file1 and file2 are the original amconll files you want to compare (this script calls the preparation python script for you) and folder is an empty, existing folder where the results will be put.

The bash script is documented/self-explanatory. It prints f-score and LAS/UAS, and will open the visualization tool. More detailed f-scores per dependency labels are printed to a file in the folder.

Example call (assuming you have placed MaltEval.jar in analyzers):

cd analyzers
mkdir tmp
bash am_comparison_full.sh ./example_compare_amconll/system_1.amconll ./example_compare_amconll/system_2.amconll tmp/

Non-comparative visualization

If you don't want to compare with a gold standard but just want to have a look at AM dependency trees, nearly the same procedure can be used but condensed into a bash script (again, cd to analyzers/):

bash visualize_am_trees.sh your_file.amconll

Example call (assuming you have placed MaltEval.jar in analyzers):

cd analyzers
bash visualize_am_trees.sh ./example_compare_amconll/system_1.amconll

2. The own GUI approach

Limitations of the previously described approach:

• graph fragments are shown in penman notation, which can be hard to read
• comparison fails if corresponding sentences in the two amconll files have different ids, or no ids at all (e.g. AMR vs SDP ids)

The script analyzers/compare_amconll.py (and also analyzers/compare_amconll_qtgui.py) can help here. It is not as elaborate as the approach using the MaltEval tool however.

Limitations of this approach:

• if sentence equality is not determined by ID but string equality, not all equivalent sentences might be detected as such due to tokenisation or special character handling differences
• no F1 score computation
• no elaborate GUI: No colour-coded diff on the dependency trees, no filter sentences based on several criteria
• for long sentences, graph constants might be too small to read (would need some sort of zoom)
• visualization not possible if graph_dependency_parser.components.dataset_readers.amconll_tools.AmSentence.to_tex_svg fails to produce an output for some reason

You need to have graphviz, pdflatex, Inkscape, command line cat and tr tools and PyQt5 (pip install pyqt5) installed. All, except for the last requirements are due to the to_tex_svg function call. (If you don't already have the parser itself running, you may also need penman, allennlp, tqdm, cython, and https://github.com/andersjo/dependency_decoding which you can pip install with pip install git+https://github.com/andersjo/dependency_decoding.)

An example command is shown below:

cd analyzers/
python ./compare_amconll.py ./example_compare_amconll/system_1.amconll ./example_compare_amconll/system_2.amconll

You can add the --useid option if you know the two amconll files have comparable ids (e.g. comparing SDP amconll files). If ids aren't used to determine equality of sentences, then sentence string equality is used. Note that this doesn't work in all cases. A very basic handling of special characters like quotation marks, parenthesis and hypens was introduced for this purpose. Sentences are shown in random order (you can edit the random seed in the script).

The script will first of all compute the intersection of both amconll files, so compute the set of sentences that appear in both files (id-based or sentence string-based, see above). A filter (defined by the function is_interesting) is then applied on the intersection. By default, this filter excludes sentences which are either too short (up to 5 tokens) or too long (15 or more tokens), but feel free to change this.

If there are still sentence left, the sentences are ordered and then a window (GUI) will appear with the following content:

• top row: the sentence (left) and the current sentence number and number of total sentences (right, after filtering done by is_interesting)
• second row: AM dependency tree from the first file (left), buttons to maximize or save the image as SVG (right)
• third row: AM dependency tree from the second file (left), buttons to maximize or save the image as SVG (right)
• last row: scrollable list of sentences (left, can click on a sentence), previous and next sentence buttons (right)

Note that changing the sentence takes some time, so please be patient ;) Also, if you maximize an image, the main window will stop responding until you close the maximized-image window.

You can't use PyQt?
there is also a variant similar to the compare_mrp.py script which can be envoked using --viz plus --nogui option, then you need Imagemagick (PyQt is not required then) to be installed. In this case, you can add the --maxshow INTEGER option which controls how many sentences are shown before the program terminates. Only one sentence is shown in the default case (no --maxshow specified).

You don't want any visualization?
Call the script with --nogui and a suitable value for --maxshow, then it will just print sentences (sentence string) to the standard output.

3. One AM dependency tree to LaTeX

If you have found a specific sentence in your AMCoNLL file and would like have some visualization you can directly insert into your paper, you can call am_dep_to_latex.py. Below are two example calls, but remember to first change your directory to the analyzers folder:

cd analyzers/
python am_dep_to_latex.py file direc --id 12345   ## select sentence with ID 12345
python am_dep_to_latex.py file direc --i 42       ## select the 42nd sentence in the file

If this was successfull, the direc you specified will contain the necessary files including sentence.tex and sentence.pdf (compiled using pdflatex).

Note: On my (PW) latop, there appeared some warnings, so I had to manually call pdflatex afterwards:

cd analyzers/
mkdir ./tmp     ## tmp is our direc here: will contain tex output and final pdf
python am_dep_to_latex.py ./example_compare_amconll/system_1.amconll ./tmp/
cd ./tmp
pdflatex -interaction=batchmode sentence.tex > /dev/null 2>&1  ## the piping with > will hide all the text normally written to the console
evince sentence.pdf  ## use evince or any other PDF viewer

NB: The PyQt GUI as described in the previous section allows you to save an SVG file ('Save' button) of an AM dependency tree.

4. Get PDF with graph for one sentence

I've found an interesting sentence with MaltEval, how do I get the corresponding graph?

1. prerequisite: you have to have python, graphviz and mtool installed. Moreover you need to have access to the MRP file containing this sentence (internal note: on coli server /proj/irtg/sempardata/mrp/LDC2019E45/2019/training).
2. get the ID of that sentence (not the one MaltEval shows in the bottom table, as the sentences there are sorted by length and numbered accordingly):
   2.1 You could get the ID from the amconll file: search for the sentence (or some rare token that occurred in the sentence), the line below the sentence should start with #id: followed by the ID. (e.g. the line is #id:22000001)
   2.2 You could also get the ID by searching for the sentence in the original mrp file. Each graph is a json-object with an id as top level key, as well as an input key whose value is the raw sentence. Make sure to choose the id-value on the topmost level of the json object: this id has quotation marks around it, unlike the ids of the individual nodes which happen to be integers.
   2.3 Note 1: hopefully all sentences occur only once in the corpus or duplicates have the same graph. Note 2: In the future, there might be some searchable tsv file or a script for this. TODO make this step easier.
3. then ask mtool to read the mrp file and output the graph with the given ID as dot:

   python /PATH/TO/main.py --strings --id ID_OF_SENTENCE --read mrp --write dot /PATH/TO/MRP/INPUT/mrpfile.mrp /PATH/TO/DOT/OUTPUT/dotfile.dot
   

   You could also use the same command without the --strings option if you prefer character-offsets instead of the string itself.
4. next, call dot with the newly created dot-file as input and specify the output file name:

   dot -Tpdf /PATH/TO/DOT/OUTPUT/dotfile.dot -o /PATH/TO/outfile.pdf

5. example:

   python ~/Schreibtisch/HiwiAK/mtool/main.py --id 20001001 --read mrp --write dot ./amr/ex_wsj.mrp ./20001001.dot
   dot -Tpdf ./20001001.dot -o ./20001001.pdf

See also Visualizing graphbanks for visualizing all graphs in a certain file (not only for MRP files) and the compare_mrp.py script.
Note would be useful to evaluate an AM dependency tree stored in an AMCoNLL file to the final graph: maybe have a look at evaluate the AM-CoNLL trees to graphs or here.

5. Internal note: Location on server

Where to find the AM-CoNLL files

On the Coli servers, amconll files that are appropriate for the comparative visualization are in /proj/irtg/sempardata/mrp/amconll/PSD (replace PSD by DM or EDS for the other graphbanks). Note that not all frameworks have the same corpus as train and dev set.

There are also other folders containing amconll files for other non-MRP experiments.