Skip to content
Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

FEVER Shared Task 2018

The First Workshop on Fact Extraction and Verification

To reproduce our FEVER shared task results:

Initial steps

  1. switch to the takuma-dev branch
  2. run bash (This will download several files and take some time.)
  3. move to jack directory and install dependencies according to the README. (i.e., python3 -m pip install -e .[tf])
  4. move to fever-baselines directory and install dependencies (i.e., pip install -r requirements.txt)

After step 2, fever, jack, fever-baselines directory should be at the same level (these should be in the same directory).

Reproduce our result

python3 --config configs/submission_config.json --model [arbitrary name]

Output files will be generated under results/[arbitrary name]. submission.json and test_submission.json correspond to the output for development and test set for each. (Note that the score displayed after running this particular model is not valid, since we included development set for the training even though the score is calculated based on the development set.)

Train a model with new data

Use the same wiki data

  1. create new configuration file configs/config.json and reflect your directory structure
  2. run python3 --config configs/config.json --model [arbitrary name]

Use a new/different wiki-pages data

  1. remove index files in data directory
  2. run python3 to create a document index and retrieval model
  3. run python3 to create a line index and retrieval model

Configuration files

Configuration files can have a parent, which is specified by parent_config attribute.

Attributes that are not specified in the child config file are inherited from the parent. Usually you do not need to modify the parent config.

Note that currently a child cannot have its child. (Having a grand-parent or grand-child is not supported)


Information Retrieval module.


Data format conversion module (to let jack handle our data).


Recognizing Textual Entailment module (training).


Recognizing Textual Entailment module (inference).


Aggregation module


Reranking module


Evaluation module

Original Paper

UCL Machine Reading Group: Four Factor Framework For Fact Finding (HexaF)


FEVER Workshop Shared-Task



No releases published


No packages published