Simtester is an open-source toolkit for evaluating user simulator of task-oriented dialogue system(TOD). It is developed based on Python and PyTorch. You can easily construct agents(system variants) with Simtester using either your own implemented model or our trained model and construct tester with different combinations of agents. In tester-based evaluation, the user simulator you implemented interacts with agents in the tester, ranks them, and tester calculates the Exact Distinct score of the user simulator. [paper]
Simtester works with the following operating systems：
- Linux
- Windows 10
- macOS X
Simtester requires Python version 3.6 or later.
You can install from pip(the current version is the old version and we will update it soon):
pip install simtester
Or you can install Simtester from source:
git clone https://github.com/Superbooming/simtester && cd simtester
pip install -e .When you construct a agent using your own implemented model, you should fill your model instance your_model
and your model interact function your_model_interact_fn which input the dialogue context and output the response.
When you construct a agent using own trained model, the model archive file and model state file will be downloaded
automate and you just need fill the model name. The remaining configuration will be loaded according to default config
file(in simtester/config/multiwoz/). Or you can fill your own config file(.yaml format) path and the config file
should include the model name and the responding model directory.See the config file example
in simtester/config/multiwoz/soloist/soloist-base.yaml.
# construct a agent using your own implemented model
from simtester import Agent
agent = Agent(your_model, your_model_interact_fn)
# construct a gent using our trained model
from simtester import SoloistAgent
agent = SoloistAgent('multiwoz-soloist-base') # fill model name
agent = SoloistAgent(config='simtester/config/multiwoz/soloist-base.yaml') # fill config path
# interact with agent and get the response
resopnse = agent.interact('I want to book a resturant in the center of city.')
# get dialogue context and start a new dialogue
agent.get_context()
agent.start_dialogue()When you construct a tester using your own agents, you should fill your agent list which includes your own agent instance and the ground truth agent rank where a smaller number indicates a higher rank. If rank is not filled, the number is default to incremented.
When you construct a agent with strategy we provided, you can just fill the strategy name.
When you interact with tester, you can either input the utterance list, or the rank your simulator predicts. If the utterance list is input, tester will return the response list which includes response responding to each agent and the dialogue state list which indicates whether each agent ends the dialog. You can set the end_token of tester and when the agent receives response which includes the end_token, the agent's dialogue will be ended. If the rank your simulator predicts is input, tester will compare the ground truth rank with input rank and return the result. Notice that the input of the rank represents the end of the dialogue for all agents, and a new round of dialogue will start.
from simtester import Tester
# construct tester with your own agents
tester = Tester([agentA, agentB, agentC], [2, 1, 3])
# construct tester with strategy we provided
tester = Tester('multiwoz-context-tester')
# interact with tester
tester.interact(['I want to book a resturant.',
'I want to book a hotel.',
'I want to find a attraction.'])
tester.interact(rank=[2, 3, 1])
# get current tester information and exact distinct score
tester.get_info()
tester.get_score()
# reset tester
tester.reset()Please let us know if you encounter a bug or have any suggestions by filing an issue.
We welcome all contributions from bug fixes to new features and extensions.
We expect all contributions discussed in the issue tracker and going through PRs.
If you find Simtester useful for your research or development, please cite our Paper:
@article{sun2022metaphorical,
title={Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems},
author={Sun, Weiwei and Guo, Shuyu and Zhang, Shuo and Ren, Pengjie and Chen, Zhumin and de Rijke, Maarten and Ren, Zhaochun},
journal={arXiv preprint arXiv:2204.00763},
year={2022}
}
Simtester was developed and maintained by Shandong University IR Lab.
Simtester uses MIT License.