Skip to content
No description, website, or topics provided.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
src
.gitignore
.travis.yml
Dockerfile
LICENSE
MANIFEST.in
readme.md
requirements.txt
setup.py
version.sh

readme.md

FEVER2.0 builder API

The FEVER2.0 shared task requires builders to submit Docker images (via dockerhub) as part of the competition to allow for adversarial evaluation. Images will host a web server using this package.

Installation

pip install fever-api

Example

See the fever2-sample repo

Entrypoint

The submission must run a flask web server. In our application, the entrypoint is a function called my_sample_fever in the module sample_application (see sample_application.py). The my_sample_fever function is a factory that returns a fever_web_api object.

from fever.api.web_server import fever_web_api

def my_sample_fever():
    # Set up and initialize model
    ...
    
    # A prediction function that is called by the API
    def baseline_predict(instances):
        predictions = []
        for instance in instances:
            predictions.append(...prediction for instance...)
        return predictions

    return fever_web_api(baseline_predict)

Your dockerfile can then use the flask run method as the entrypoint, setting any valid factory as the FLASK_APP

ENV FLASK_APP sample_application:my_sample_fever
ENTRYPOINT ["flask","run"]

Web Server

The web server is managed by the fever-api package. No setup or modification is required by participants. We use the default flask port of 5000 and host a single endpoint on /predict. We recommend using a client such as Postman to test your application.

POST /predict HTTP/1.1
Host: localhost:5000
Content-Type: application/json

{
	"instances":[
	    {"id":0,"claim":"this is a test claim"}, 
	    {"id":1,"claim":"this is another test claim"}, 
	]
}

API

In our sample submission, we present a simple method baseline_predict method.

   def baseline_predict(instances):
        predictions = []
        for instance in instances:
            ...prediction for instance...
            predictions.append({"predicted_label":"SUPPORTS", 
                                "predicted_evidence": [(Paris,0),(Paris,5)]})
            
        return predictions

Inputs:

  • instances - a list of dictionaries containing a claim

Outputs:

  • A list of dictionaries containing predicted_label (string in SUPPORTS/REFUTES/NOT ENOUGH INFO) and predicted_evidence (list of (page_name,line_number) pairs as defined in fever-scorer.
You can’t perform that action at this time.