Skip to content

AM parser on COGS

Jump to bottom
weissenh edited this page Jul 12, 2022 · 4 revisions

AM parser on COGS

under construction (July 2022) missing:

  • describe old setup (not using docker)
  • more detailed reproduction steps (training scripts and options, explain log file format etc)
  • port content from old readme in scripts folder

TL;DR Code documentation for the AM parser experiments on the COGS dataset of Kim and Linzen (2020).

This site describes how the AM parser (based on the version presented in Groschwitz et al. 2021) was adapted to the COGS dataset of Kim and Linzen (2020) targeting compositional generalization. The COGS corpus contains neo-Davidsonian-inspired logical forms as meaning representation which needed to be converted to graphs first.

Corresponding papers

  • Pia Weißenhorn, Yuekun Yao, Lucia Donatelli, Alexander Koller (2022) Compositional Generalization Requires Compositional Parsers arXiv:2202.11937 [cs.CL]
  • Pia Weißenhorn, Lucia Donatelli, Alexander Koller (2022) Compositional generalization with a broad-coverage semantic parser To be presented at *SEM2022, co-located with NAACL2022.
  • Pia Weißenhorn (2022) Investigations on what makes compositional generalization challenging. Master's Thesis, Saarland University, Germany. (Supervisors: Alexander Koller and Lucia Donatelli)

Note: Extending the AM parser to COGS has been pursued as part of Pia Weißenhorn's master thesis. In the process of writing this thesis, two papers were published (on arXiv and at *SEM) describing some of the thesis' findings.

Reproduction steps

The ability of the AM parser to handle COGS meaning representation is only available on certain branches of the two main repositories:

  • am-tools: this repo is used for the pre- and postprocessing including the on-the-fly graph conversion. You need to compile an am-tools.jar from the cogs branch cogs_new_decomp to perform the preprocessing.

  • am-parser: The neural model for the COGS AM parser is also found on a specific branch of the am-parser repo, namely the cogs_unsupervised branch

Both cogs branches (am-tools, am-parser) were branched off from Groschwitz et al.'s 2021 'unsupervised parser' code at an earlier stage of development.

Furthermore an am-tools.jar from the main branch is needed, as this contains the most recent version of the Astar decoder (including a bug fix important for COGS specifically: punctuation should be ignored). For more details on code versions, see below.

Please see the main readme (note: no longer valid, this happened before we started with docker) and sites for installing necessary dependencies for the repositories.

Previously, parts of the documentation for AM parser's COGS experiments have been written here, and need to be copied and revised to this wiki page. to do: add more documentation on how to reproduce results and what files are important for this.

Versions used for AM parser experiments reported in Weißenhorn et al. (2022)

The MIT-licensed COGS dataset was used as-is in its commit version 6f66383 from 2nd April 2021).

Name Version/Commit
am-tools, cogs_new_decomp branch commit bdbb03e on Sep 29, 2021
am-tools, master branch (for A* decoding) commit 78fac84 on Jul 8, 2021
am-parser, cogs_unsupervised branch commit b57ffad on Sep 30, 2021
Java OpenJDK RE 12.0.2
Gradle 6.7
NVIDIA CUDA 10.2
Python 3.7.3
AllenNLP 0.8.4

Notes on graph conversion

Graphs not printed. As the graphs are just an intermediary step for the AM parser, there was no need to print them to a file somewhere. The AM parser operates on AM dependency trees which are obtained by converting logical forms to graphs which in turn are directly decomposed (into tree automata summarizing different AM dependency trees before training , after training only one AM dependency tree is left) and on the post-processing side predicted AM dependency trees are evaluated into graphs and then directly converted into logical forms again.

Verbal primitives. There is an additional caveat with respect to 'verbal primitives': in training sets some samples contain a word in isolation plus its meaning (these samples are called 'primitives'). Beside nouns and their meaning, some primitives contain verbs like 'crawl' or 'touch'. Notice that they ask for arguments like agent or theme, which the graph conversion translates into unfilled/open sources. As sources are learned in Groschwitz et al.'s (2021) setup, there is no unique graph for such examples, because the decomposition did not specify which source exactly should be there: does the agent edge point to a node annotated with source S0 ? or should it be source S1 instead ? Or S2? (Similarly for theme edges). Notice that verbal primitives are needed for the prim_to_inf_arg generalization type (Primitive verb to infinitival argument: 'crawl' to 'The baby planned to crawl').

Alignments. Another caveat revolves around the necessity of alignments: COGS logical forms contain variables such as x_1 or x_4 to represent alignment to the first or fourth input token (0-indexed). Interestingly, terms are also sorted based on these token positions (and proper noun positions which are not visible in the logical form). As COGS main evaluation metric is exact match accuracy, finding the right alignments and x_i variables is therefore crucial. So, although it is called 'graph conversion', just a graph is not a sufficient input for the conversion back to logical form. Instead, graphs (more specifically nodes) have to be aligned to input tokens.

Preposition reification. For a more elegant decomposition with the AM algebra it was decided to 'reify' preposition meaning to a node. Instead of treating noun1.nmod.on(x_0, x_3) as a nmod.on edge between two nouns, the preposition token 'on' (same for 'in', 'beside') yields a respective node with outgoing nmod.op1/2 edges to the two nouns respectively. E.g. "cat on a mat" without preposition reification would yield (x_0 / cat :nmod.on (x_3 / mat)), with preposition reification we have (x_0 / cat :nmod.op1-of (x_1 / on :nmod.op2 (x_3 / mat))). Watch out for the respective option when envoking the AM parser (especially am-tools)

Comparison to the graph conversion of Jambór and Bahdanau (2021). Note that a similar graph conversion was developed concurrently to our work by Jambór and Bahdanau (2021).

  • They label the 'determiner node' with * , whereas we label it with the. Additionally, the outgoing edge is labelled article by them, but iota in our conversion.
  • We 'reify' prepositions (see above), whereas they use an edge only ( cat -nmod.on-> mat) representation for PP modifcation.
  • They do not need a designated root node.
  • We don't know how they dealt with primitive samples.

Notes on reading output

  • AMCONLL files contain supertags. These supertags are printed in PENMAN notation with node as nodename / nodelabel. Although node names start with x_, they should not be mistaken for actual alignments to input token positions. These supertags were learnded during training to be reusable and node names are not part of the evaluation, unlike labels. For ease of presentation example graphs in published papers (Weißenhorn et al. 2022, Weißenhorn 2022) do use node names to visualize alignments, but the code does not: in the code, alignments are given by virtue of supertags being predicted for specific input tokens.
  • it can happen that the AM parser predicts a graph for which the conversion procedure refrains to output a logical form. So called 'ill-formed' logical forms appear if the AM parser would create a predicate such as Ava.agent, so a proper name as part of a predicate: this is not allowed.
  • for reading AMCONLL files see the specific page in the AM parser wiki. If you find reading such amconll files tedious, you might want to have a look at the wiki page on visualizing AM dependency trees for error analysis

Things to play around with

  • training input (am-parser/scripts/cogs2021/get_train_dev.sh):
    • lex label replacement (helps for nouns, but deteriorates verb): node name is replaced by $FORM$ if it matches the word form of the aligned token: so one can effectively copy tokens from the input into node names. However, inflection can get in the way, e.g. for verbs.
    • preposition reification or not
    • use train or train100 corpus for training (need to preprocess respective files in advance of course)
    • how many sources (tried 3, with 2 couldn't do distransitive verbs, 4 or more would be possible but might increase runtime)
  • during training (am-parser/jsonnets/cogs2021/COGS_*.jsonnet):
    • edge model:
      • kg_edges Kiperwasser and Goldberg-style dependency model
      • dm_edges Dozat and Manning-style dependency model (not tested with COGS)
      • kg_rel_edges , same as kg but enhanced with relative distance encodings for the edge existence (but not label) decision. These distance encodings helped specifically for PP recursion to resolve the attachment ambiguity to always low-attachment correctly (in paper signified with +dist)
    • all automaton or supervised loss (supervised loss for dependency edge existence and lexical labels for supertags or have all the loss flow through tree automata?)
    • embeddings: pretrained BERT or embeddings learned from COGS data only, embedding size
    • layers and dropout, learning rate, dimensionalities
  • after training (am-parser/scripts/cogs2021/unsupervised_predict.sh):
    • decoding with fixed-tree or projective Astar decoder (-p option enables projective Astar instead of fixed-tree decoder, but then need am-tools.jar from master branch for most recent Astar implementation)

Tips on conducting further error analysis

  • you can look at the average token-level edit distances (with or without exact match cases) to get a first idea of what's going on (per generalization type output provided by am-tools ToCOGSCorpus run with --verbose option : will print this to stdout and all non-exact match cases to stderr): to do how to read stderr 'No exact match found'
  • you can of course manually compare logical forms for individual samples of gold and prediction
  • you can use analyzers/compare_amconll.py (see am-parser wiki page for more info: 'own GUI option') to display AM dependency trees : to do example call
  • if predictions were generated using the projective Astar decoder (instead of the fixed-tree decoder), you can inspect the trained model's scores for supertags and dependency edges: scores.zip is output part of unsupervised predict but can also be generated separately using dump_scores.py, then cogs branch am-parser/analyzers/scores_prettify.py, see also the am-parser wiki page on computing scores to do: describe usage: heatmaps, output files, filtering
  • for fixed-tree decoder one can ask how many non-projective AM dependency trees were predicted. This script is not properly tested but might give a first hint at the degree of non-projectivity: cogs branch am_dep_projectivity.py. This is especially relevant for COGS, since it is assumed that all gold dependency trees are projective.

References and further links

Clone this wiki locally