Skip to content

AM parser on COGS

Jump to bottom
weissenh edited this page Apr 12, 2022 · 4 revisions

AM parser on COGS

under construction (April 2022)

TL;DR Code documentation for the AM parser experiments in Weißenhorn et al. (2022).

This site describes how the AM parser (based on the version presented in Groschwitz et al. 2021) was adapted to the COGS dataset of Kim and Linzen (2020) targeting compositional generalization. The COGS corpus contains neo-Davidsonian-inspired logical forms as meaning representation which needed to be converted to graphs first.

Corresponding paper

  • Pia Weißenhorn, Yuekun Yao, Lucia Donatelli, Alexander Koller (2022) Compositional Generalization Requires Compositional Parsers arXiv:2202.11937 [cs.CL]

Code bases

The ability of the AM parser to handle COGS meaning representation is only available on certain branches of the two main repositories:

  • am-tools: this repo is used for the pre- and postprocessing including the on-the-fly graph conversion. You need to compile an am-tools.jar from the cogs branch cogs_new_decomp to perform the preprocessing.

  • am-parser: The neural model for the COGS AM parser is also found on a specific branch of the am-parser repo, namely the cogs_unsupervised branch

Both cogs branches (am-tools, am-parser) are based on the branches for Groschwitz et al. 2021 'unsupervised parser' code.

Furthermore an am-tools.jar from the main branch is needed, as this contains the most recent version of the Astar decoder (including a bug fix important for COGS specifically: punctuation should be ignored).

Reproduction steps

Please see the main readme and sides for installing necessary dependencies for the repositories.

Main documentation for the COGS experiments is currently found in a readme here

to do: add more documentation on how to reproduce results and what files are important for this.

Versions used for AM parser experiments reported in Weißenhorn et al. (2022)

The MIT-licensed COGS dataset was used as-is in its commit version 6f66383 from 2nd April 2021).

Name Version/Commit
am-tools, cogs_new_decomp branch commit bdbb03e on Sep 29, 2021
am-tools, master branch (for A* decoding) commit 78fac84 on Jul 8, 2021
am-parser, cogs_unsupervised branch commit b57ffad on Sep 30, 2021
Java OpenJDK RE 12.0.2
Gradle 6.7
NVIDIA CUDA 10.2
Python 3.7.3
AllenNLP 0.8.4

Notes on graph conversion

As the graphs are just an intermediary step for the AM parser, there was no need to print them to a file somewhere. The AM parser operates on AM dependency trees which are obtained by converting logical forms to graphs which in turn are directly decomposed (into tree automata summarizing different AM dependency trees before training , after training only one AM dependency tree is left) and on the post processing site predicted AM dependency trees are evaluated into fgraphs and then directly converted into logical forms again.

There is an additional caveat with respect to 'verbal primitives': in training sets some samples contain a word in isolation plus its meaning (sees samples are called 'primitives'). Beside nouns and their meaning, some primitives contain verbs like 'crawl' or 'touch'. Notice that they ask for arguments like agent or theme, which the grpah conversion translates into open sources beyond the root node. As sources are learned in Groschwitz et al.'s (2021) setup, there is no unique graph for such examples, because the decomposition did not specify which source exactly should be there: does the agent edge point to a node annotated with source S0 ? or should it be S1? Or ? S2? (Similarly for theme edges).

Another caveat revolves around the necessity of alignments: COGS logical forms contain variables such as x_1 or x_4 to represent alignment to the first or fourth input token (0-indexed). Interestingly terms are also sorted based on these token positions (and proper noun positions whihc are not visibile in the logical form). As COGS main evaluation metric is exact match, finding the right alignments and x_i variables is therefore crucial. So, alhtough it is called 'graph conversion', just a graph is not a sufficient input for the conversion back to logical form. Instead graphs (more specifically nodes) have to be aligned to input tokens.

Note that a smiliar graph conversion was developed concurrently to our work by Jambór and Bahdanau (2021).

Notes on reading output

  • amconll files contain supertags. These supertags are printed in PENMAN notation with node as nodename / nodelabel. Although node names start with x_, they should not be mistaken for actual alignments to input token positions. These supertags were learnded during training to be reusable and node names are not part of the evaluation, unlike labels. For ease of presentation example graphs in published papers (Weißenhorn et al. 2022, Weißenhorn 2022) do use node names to visualize alignments, but the code does not: in the code, alignments are given by virtue of supertags being predicted for specific input tokens.
  • it can happen that the AM parser predicts a graph for which the conversion procedure refrains to output a logical form. So called 'ill-formed' logical forms appear if the AM parser would create a predicate such as Ava.agent, so a proper name as part of a predicate: this is not allowed.
  • for reading AMCONLL files see the specific page in the AM parser wiki. If you find reading such amconll files tedious, you might want to have a look at the wiki page on visualizing AM dependency trees for error analysis

Things to play around with

  • after training:
    • decoding with fixed-tree or astar decoder
  • during training:
    • edge model: kg, dm, kgrel (that is Kiperwasser and Goldberg, Dozat Manning, or Kiperwasser Goldberg enhanced with relative distance encodings)
    • all automaton or supervised loss (supervised loss for dependency edge existence and lexical labels for supertags or have all the loss flow through tree automata?)
    • embeddings (pretrained BERT or not, embedding size)
    • layers and dropout, learning rate, dimensionalities
  • training input:
    • lex label replacement (helps for nouns, but deteriorates verb)
    • prep reification or not
    • use train or train100 corpus for training (need to preprocess respective files in advance of course)
    • how many sources (tried 3, with 2 couldn't do distransitive verbs, 4 or more would be possible but might increase runtime)

Tips on conducting further error analysis

  • you can look at the average token-level edit distances (with or without exact match cases) to get a first idea of what's going on (per generalization type output provided by am-tools ToCOGSCorpus run with --verbose option : will print this to stdout and all non-exact match cases to stderr): to do how to read stderr 'No exatc match found'
  • you can of course manually compare logical forms for individual samples of gold and prediction
  • you can use analyzers/compare_amconll.py (see am-parser wiki page for more info: 'own GUI option') to display AM dependency trees : to do example call
  • if predictions were generated using the projective Astar decoder (instead of the fixed-tree decoder), you can inspect the trained model's scores for supertags and dependency edges: scores.zip is output part of unsupervised predict but can also be generated separately using dump_scores.py, then cogs branch am-parser/analyzers/scores_prettify.py, see also the am-parser wiki page on computing scores to do: describe usage: heatmaps, output files, filtering
  • for fixed-tree decoder one can ask how many non-projective AM dependency trees were predicted. This script is not properly tested but might give a first hint at the degree of non-projectivity: cogs branch am_dep_projectivity.py This is especially relevant for COGS since it is assumed that all gold dependency trees are projective.

References and further links

Clone this wiki locally