Skip to content
FEVER Workshop Shared-Task
Branch: master
Clone or download
Pull request Compare This branch is even with uclmr:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

FEVER Shared Task 2018

The First Workshop on Fact Extraction and Verification

To reproduce our FEVER shared task results:

Initial steps

  1. switch to the takuma-dev branch
  2. run bash (This will download several files and take some time.)
  3. move to jack directory and install dependencies according to the README. (i.e., python3 -m pip install -e .[tf])
  4. move to fever-baselines directory and install dependencies (i.e., pip install -r requirements.txt)

After step 2, fever, jack, fever-baselines directory should be at the same level (these should be in the same directory).

Reproduce our result

python3 --config configs/submission_config.json --model [arbitrary name]

Output files will be generated under results/[arbitrary name]. submission.json and test_submission.json correspond to the output for development and test set for each. (Note that the score displayed after running this particular model is not valid, since we included development set for the training even though the score is calculated based on the development set.)

Train a model with new data

Use the same wiki data

  1. create new configuration file configs/config.json and reflect your directory structure
  2. run python3 --config configs/config.json --model [arbitrary name]

Use a new/different wiki-pages data

  1. remove index files in data directory
  2. run python3 to create a document index and retrieval model
  3. run python3 to create a line index and retrieval model

Configuration files

Configuration files can have a parent, which is specified by parent_config attribute.

Attributes that are not specified in the child config file are inherited from the parent. Usually you do not need to modify the parent config.

Note that currently a child cannot have its child. (Having a grand-parent or grand-child is not supported)


Information Retrieval module.


Data format conversion module (to let jack handle our data).


Recognizing Textual Entailment module (training).


Recognizing Textual Entailment module (inference).


Aggregation module


Reranking module


Evaluation module

Original Paper

UCL Machine Reading Group: Four Factor Framework For Fact Finding (HexaF)

You can’t perform that action at this time.