README.md

Neural Baseline Models for DSTC11 Track 5

This directory contains the official baseline codes for DSTC11 Track 5.

Getting started

• Clone this repository into your working directory.

$ git clone https://github.com/alexa/dstc11-track5.git
$ cd dstc11-track5

• Install the required python packages.

$ pip3 install -r requirements.txt
$ python -m nltk.downloader 'punkt'
$ python -m nltk.downloader 'wordnet

• Train the baseline models.

$ ./bin/run_baseline_training.sh

• Run the baseline models.

$ ./bin/run_baseline_eval.sh

• Validate the structure and contents of the tracker output.

$ python -m scripts.check_results --dataset val --dataroot data/ --outfile pred/val/baseline.rg.bart-base.json
Found no errors, output file is valid

• Evaluate the output.

$ python -m scripts.scores --dataset val --dataroot data/ --outfile pred/val/baseline.rg.bart-base.json --scorefile pred/val/baseline.rg.bart-base.score.json

• Print out the scores.

$ cat pred/val/baseline.rg.bart-base.score.json | jq
{
  "detection": {
    "prec": 1,
    "rec": 0.9990605918271489,
    "f1": 0.9995300751879699
  },
  "selection": {
    "prec": 0.7950632648828044,
    "rec": 0.8843003806667435,
    "f1": 0.8373109060127791,
    "em_acc": 0.40488492249882574
  },
  "generation": {
    "bleu": 0.10419026648096004,
    "meteor": 0.1810290467362776,
    "rouge_1": 0.36509971645462036,
    "rouge_2": 0.15059917821384866,
    "rouge_l": 0.2874932052481134
  }
}