NLR

Code for the EMNLP 2022 paper "Can Transformers Reason in Fragments of Natural Language?".

Setup

Setup the environment correspondingly, e.g.

conda create -n nlr python=3.8 anaconda
pip install -f requirements.txt

Experiments referred to in the paper are in the folder ./experiments.

To replicate our experiments, simply run

PYTHONPATH='.' python scripts/classify.py experiments/$EXPERIMENT_TO_REPLICATE

Note that the evaluation experiments (e.g. in folder ./experiments/eval-across) rely on the existence of trained models. In this case, the corresponding models need to be trained first (e.g. by replicating ./experiments/final).

Generation script for random problems

The script to run is scripts/verify_vampire.py -- it generates a problem set then classifies it with Vampire before turning it to a JSON file. Also generates some statistics. NOTE: requires a Vampire executable called vampire in the main project directory.

Parameters:
- na number of atoms
- nb number of binary atoms
- nf number of formulae
- pr probability of relational (set to 0 to get problems in S)
- pus probability of universal in s-formula
- puf probability of universal in r-formula
- pue probability of universal in e-term
- pns probability of negated subject (set to 0 to get problems in R)
- pno probability of negated object (set to 0 to get problems in R)
- pnp probability of negated predicate (set to 0 to get problems in R)
- e true for random generation, false to employ removal of trivial absurdities and universal chain blowup
- ch length of chain blowup in hard generation
- s how many problems to generate
- sd seed
- nf_min min number of formulae if generating a range
- nf_max max number of formulae if generating a range
- af_ratio ratio of atoms to formulae (can be used instead of na, when generating range)
- bf_ratio ratio of binary atoms to formulae (can be used instead when generating range)

The script comp_subj_verify_vampire.py is used for generating SRel and SRelNeg fragments, and generate.py and generate_all_all.py are used to generate constructed S+/R problems, respectively. Refer to scripts for more details.

Data and models

You can get the data from here and the pretrained models from here.

schlevik/nlr

ErrorLooks like something went wrong!

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Languages

Error
Looks like something went wrong!

Packages