nl2pl

nl2pl is a tool for training neural semantic parsers that predict program code from natural language descriptions. It uses a grammar specification and a LALR(1) parser to enforce syntactically valid programs during inference. Check out the live demo (hosted on a Heroku free dyno, may take few seconds to boot the app and return the page) to test some example models.

Grammar-Constrained Neural Semantic Parsing with LR Parsers.
Artur Baranowski and Nico Hochgeschwender
In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1275–1279, Online.
Project page / Paper 

Requirements

As of now, Python3 is supported. nl2pl depends on lark for parsing, pytorch for building neural network models. Set up a new virtual environment and install all requirements:

pip install -r requirements.txt

Quickstart

The tool employs a parser-guided decoder for predicting tokens. Accordingly, each model you build will be language-specific. The parser is generated using lark. In order to preprocess your datasets, you have to provide a .lark file containing your grammar specification. The lark grammar reference contains detailed documentation in this regard.

1) Preprocessing the data

python3 preprocess.py \
    --grammar your_grammar.lark \
    --src_train path/to/src_train.txt \
    --tgt_train path/to/tgt_train.txt \
    --src_dev path/to/src_dev.txt \
    --tgt_dev path/to/tgt_dev.txt \
    --src_test path/to/src_test.txt \
    --tgt_test path/to/tgt_test.txt \
    --save_data data_name

The preprocessing script expects at least a grammar, the dataset files containing training sources (natural language descriptions) and targets (corresponding program code) and a name under which to store the preprocessed data. Optionally, development and test splits can be provided for validating training runs and inference. The dataset source and target files are expected to be parallel, containing one example per line. The script will yield the following files:

• data_name.train.pt
• data_name.dev.pt
• data_name.test.pt
• data_name.lang.pt

Note that during preprocessing, target examples that cannot be parsed according to the provided grammar will be discarded. You can validate your target examples and spot faulty programs in your dataset:

python3 preprocess.py \
    --grammar your_grammar.lark \
    --tgt_train path/to/tgt_train.txt \
    --tgt_dev path/to/tgt_dev.txt \
    --tgt_test path/to/tgt_test.txt \
    --check

2) Training the model

python3 train.py --data /path/to/data/data_name --save model_name --validate

The training script defaults to a simple sequence-to-sequence model with an unidirectional one-layer encoder and decoder. By adding the --attention flag Bahdanau attention will be used for decoding. By adding the --copy flag a pointer-generator network will be used for copying tokens from the input sentence. See train.py for a list of all possible configuration options. If the --validate flag is set, the script will validate training results on the development data and save the model with best performance on the development data.

The script will yield a model ready for inference:

• model_name.model.pt

3) Evaluating the model

python3 translate.py --model model_name.model.pt --eval path/to/data/data_name.test.pt --out out_file.txt

This will evaluate the model on the test split of the dataset and print the statistics to out_file.txt.

4) Running a translation server for inference

python3 translate.py --model model_name.model.pt --host <host_ip> --port <port>

This will load your model as a translation service. It expects data in the following format:

[{"src": "your input sentence to translate", "id": 0}]

Test the translation service by sending a POST request:

curl -i -X POST -H "Content-Type: application/json"  \
    -d '[{"src": "your input sentence to translate", "id": 0}]' \
    http://<host_ip>:<port>/

You can run multiple models by setting a configuration in config.json and running server.py. See config.json for an example.

Citation

@inproceedings{baranowski-hochgeschwender-2021-grammar,
    title = "Grammar-Constrained Neural Semantic Parsing with {LR} Parsers",
    author = "Baranowski, Artur  and
      Hochgeschwender, Nico",
    booktitle = "Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021",
    month = aug,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.findings-acl.108",
    doi = "10.18653/v1/2021.findings-acl.108",
    pages = "1275--1279",
}

Acknowledgements

• The parser implementation is based on lark
• The tool design is loosely based on OpenNMT-py