New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

local human agent #110

Merged
merged 2 commits into from May 30, 2017

Conversation

Projects
None yet
3 participants
@jaseweston
Contributor

jaseweston commented May 30, 2017

Behaviour:


bash-3.2$ python examples/eval_model.py -m local_human -t babi:Task1k:1 -dt valid
[task:babi:Task1k:1]
[download_path:/Users/jase/src/ParlAI/downloads]
[datatype:valid]
[numthreads:1]
[no_images:False]
[batchsize:1]
[datapath:/Users/jase/src/ParlAI/data]
[model:local_human]
[model_params:]
[model_file:]
[num_examples:1000]
[display_examples:False]
[parlai_home:/Users/jase/src/ParlAI]
LocalHumanAgent
[creating task(s): babi:Task1k:1]
[DialogTeacher initializing.]
[loading fbdialog data:/Users/jase/src/ParlAI/data/bAbI/tasks_1-20_v1-2/en-valid-nosf/qa1_valid.txt]
[babi:Task1k:1]: Daniel journeyed to the kitchen.
Daniel journeyed to the bedroom.
Where is Daniel?
[cands: kitchen|garden|bedroom|bathroom|office|hallway]
Enter Your Reply:bedroom
---
{'total': 1, 'accuracy': 1.0, 'f1': 1.0, 'hits@k': {1: 1.0, 5: 1.0, 10: 1.0, 50: 1.0, 100: 1.0}}
[babi:Task1k:1]: Sandra moved to the garden.
Sandra went back to the kitchen.
Where is Sandra?
[cands: kitchen|garden|bedroom|bathroom|office|hallway]
Enter Your Reply:bathroom
---
{'total': 2, 'accuracy': 0.5, 'f1': 0.5, 'hits@k': {1: 0.5, 5: 0.5, 10: 0.5, 50: 0.5, 100: 0.5}}
[babi:Task1k:1]: Daniel travelled to the office.
Sandra went back to the hallway.
Where is Sandra?
[cands: kitchen|garden|bedroom|bathroom|office|hallway]
Enter Your Reply:hallway
---
{'total': 3, 'accuracy': 0.6666666666666666, 'f1': 0.6666666666666666, 'hits@k': {1: 0.6666666666666666, 5: 0.6666666666666666, 10: 0.6666666666666666, 50: 0.6666666666666666, 100: 0.6666666666666666}}
[babi:Task1k:1]: Daniel journeyed to the hallway.
Daniel went to the bathroom.
Where is Sandra?
[cands: kitchen|garden|bedroom|bathroom|office|hallway]
Enter Your Reply:

@jaseweston jaseweston merged commit 3023f0c into master May 30, 2017

@alexholdenmiller alexholdenmiller deleted the human branch May 31, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment