Skip to content

AM parser on COGS

Jump to bottom
weissenh edited this page Mar 1, 2022 · 4 revisions

AM parser on COGS

under construction (March 2022)

TL;DR Code documentation for the AM parser experiments in Weißenhorn et al. (2022).

This site describes how the AM parser (based on the version presented in Groschwitz et al. 2021) was adapted to the COGS dataset of Kim and Linzen (2020) targeting compositional generalization. The COGS corpus contains neo-Davidsonian-inspired logical forms as meaning representation which needed to be converted to graphs first.

Corresponding paper

  • Pia Weißenhorn, Yuekun Yao, Lucia Donatelli, Alexander Koller (2022) Compositional Generalization Requires Compositional Parsers arXiv:2202.11937 [cs.CL]

Code bases

The ability of the AM parser to handle COGS meaning representation is only available on certain branches of the two main repositories:

  • am-tools: this repo is used for the pre- and postprocessing including the on-the-fly graph conversion. You need to compile an am-tools.jar from the cogs branch cogs_new_decomp to perform the preprocessing.

  • am-parser: The neural model for the COGS AM parser is also found on a specific branch of the am-parser repo, namely the cogs_unsupervised branch

Both cogs branches (am-tools, am-parser) are based on the branches for Groschwitz et al. 2021 'unsupervised parser' code.

Furthermore an am-tools.jar from the main branch is needed, as this contains the most recent version of the Astar decoder (including a bug fix important for COGS specifically: punctuation should be ignored).

Reproduction steps

Please see the main readme and sides for installing necessary dependencies for the repositories.

Main documentation for the COGS experiments is currently found in a readme here

to do: commit versions, java and python versions, including cogs version

Things to play around with

  • after training:
    • decoding with fixed-tree or astar decoder
  • during training:
    • edge model: kg, dm, kgrel
    • all automaton or supervised loss
    • embeddings (pretrained BERT or not, embedding size)
    • layers and dropout, learning rate, dimensionalities
  • training input:
    • lex label replacement (helps for nouns, but deteriorates verb)
    • prep reification or not
    • train or train100
    • how many sources (tried 3, with 2 couldn't do distransitive verbs, 4 or more would be possible)

Tips on conducting further error analysis

  • you can look at the average token-level edit distances (with or without exact match cases) to get a first idea of what's going on (per generalization type output provided by am-tools ToCOGSCorpus run with --verbose option : will print this to stdout and all non-exact match cases to stderr): to do how to read stderr 'No exatc match found'
  • you can of course manually compare logical forms for individual samples of gold and prediction
  • you can use analyzers/compare_amconll.py (see am-parser wiki page for more info: 'own GUI option') to display AM dependency trees : to do example call
  • if predictions were generated using the projective Astar decoder (instead of the fixed-tree decoder), you can inspect the trained model's scores for supertags and dependency edges: scores.zip is output part of unsupervised predict but can also be generated separately using dump_scores.py, then cogs branch am-parser/analyzers/scores_prettify.py, see also the am-parser wiki page on computing scores to do: describe usage: heatmaps, output files, filtering
  • for fixed-tree decoder one can ask how many non-projective AM dependency trees were predicted. This script is not properly tested but might give a first hint at the degree of non-projectivity: cogs branch am_dep_projectivity.py This is especially relevant for COGS since it is assumed that all gold dependency trees are projective.

References

Clone this wiki locally