Doc|Model|Dataset|Paper

MWPToolkit is a PyTorch-based toolkit for Math Word Problem (MWP) solving. It is a comprehensive framework for research purpose that integrates popular MWP benchmark datasets and typical deep learning-based MWP algorithms.

Our framework has the following architecture. You could utilize our toolkit to evaluate the build-in datasets, apply it to process your raw data copies or develop your own models.

Figure: The Overall Framework of MWP Toolkit

News

• Fix some bugs.

Characteristics

• Unification and Modularization. We decouple solvers with different model architectures into highly modularized, reusable components and integrate them in a unified framework, which includes data, model, evaluation modules. It is convenient for you to study MWPs at a conceptual level and compare different models fairly.
• Comprehensiveness and Standardization. MWPToolkit has deployed the popular benchmark datasets and models for MWPs solving, covering Seq2Seq, Seq2Tree, Graph2Tree, and Pre-trained Language Models. Moreover, some tricks like hyper-parameter tuning used in general NLP tasks are integrated. As all models can be implemented with a same experimental configuration, the evaluation of different models is standardized.
• Extensibility and Usability. MWPToolkit provides user-friendly interfaces for various functions or modules. And the components in the pipeline architecture are modeled as exchangeable modules. You can try different combinations of modules via simply changing the configuration file or command line. You can also easily develop your own models by replacing or extending corresponding modules with your proposed ones.

Installation

Development environment:

python >= 3.6.0
pytorch >= 1.5.0
pyltp >= 0.2.1 (optional)

Method 1: Install from pip

pip install mwptoolkit

Method 2: Install from source

# Clone current repo
git clone https://github.com/LYH-YF/MWPToolkit.git && cd MWPToolkit

# Requirements
pip install -r requirements.txt

Quick Start

Evaluate a build-in dataset with a model

To have an initial trial of our toolkit, you can use the provided cmd script:

python run_mwptoolkit.py --model=GTS --dataset=math23k --task_type=single_equation --equation_fix=prefix --k_fold=5 --test_step=5 --gpu_id=0

Above script will run GTS model on Math23K dataset with 5 cross-validation. It will take around xx minutes to train 5 GTS models independently and output the average scores of equation accuracy and value accuracy. The training log can be found in the log file.

If you would like to change the parameters, such as dataset and model, please refer to the following instruction:

• model: The model name you specify to apply. You can see all options in Section Model.
• dataset: The dataset name you specify to evaluate. You can see all options in Section Dataset.
• task_type: The type of generated equation. It should be chosen from options [single_equation | multi_equation]. Usually, it's up to the datasets. You can refer to dataset. The single-equation dataset corresponds to 'single_equation' in code and multiple-equation dataset corresponds 'multi_equation' in code.
• equation_fix: The type of equation generation order. It should be chosen from options [infix | postfix | prefix]. Please note some models require a specific type of equation generation order. Usually, the corresponding paper for model will mention which order it takes. You can look up the reference paper in Section Model.
• k_fold: The fold number of cross-validation. It could be either NA value or an integer. If it is NA value, it will run train-valid-test split procedure.
• test_step: The epoch number of training after which conducts the evaluation on test. It should be an interger.
• gpu_id: The GPU ID for training the model. It should be an integer based on your GPU configuration. Please note that we haven't tested the framework with multiple GPUs yet.

Please note model, dataset and task_type are the required. We also provide the interface where you can config your experiments by clicking options and we automatically generate corresponding cmd lines.

Evaluate a new dataset

Our supported datasets are all saved under the folder 'dataset'. Besides trying our code with these build-in datasets, we also provide the option for you to run models on your own data copies, you can follow the steps below:

Step 1: Organize your dataset. Your dataset folder (same as the dataset name) should include three json files for train, validation and test, respectively:

dataset_name
    |----trainset.json
    |----validset.json
    |----testset.json

Move your dataset folder under path 'dataset' of our framework, the file structure would be like:

dataset
    |----dataset_name
            |----trainset.json
            |----validset.json
            |----testset.json

Step 2: Setup your dataset configuration. The dataset configuration files are saved under path 'mwptoolkit/properties/dataset/'. You can write your own dataset configuration and save a JSON file under the path. The path to your JSON file should be mwptoolkit/properties/dataset/dataset_name.json

Step 3: Run the code!

python run_mwptoolkit.py --model=[model_name] --dataset=[dataset_name] --task_type=[single_equation|multi_equation] --equation_fix=[infix|postfix|prefix] --k_fold=[5|None] --gpu_id=0

Instead of moving your dataset folder and dataset configuration file to the above folders, the following parameters can be set directly.

• dataset_path: The path to dataset folder. The default value is 'dataset/dataset_name', you can change it to your own dataset path via appending --dataset_path=[your_dataset] to cmd script.
• dataset_config_path: The path to dataset configuration file. The default value is 'mwptoolkit/properties/dataset/dataset_name.json', you can change it to your own dataset configuration path via appending --dataset_config_path=[your_dataset_configuration] to cmd script.

Run hyper-parameters search

Our toolkit also provides the option to do hyper-parameters search, which could facilitate users to obtain optimal hyper-parameters efficiently. We integrated hyper-parameter search in our framework via ray.tune. Due to the search procedure, it will take longer time to train a model.

You can run the cmd script template below:

python run_hyper_search.py --model=[model_name] --dataset=[dataset_name] --task_type=[single_equation|multi_equation] --equation_fix=[infix|postfix|prefix] --k_fold=[5|None] --cpu_per_trial=2 --gpu_per_trial=0.5 --samples=1 --search_file=search_file.json --gpu_id=0

• cpu_per_trial: The CPU resources to allocate per trial.
• gpu_per_trial: The GPU resources to allocate per trial.
• samples: The number of sampling times from the search space.
• search_file: A json file including search parameter name and space. For example:["embedding_size=[64,128,256]","hidden_size=[256,512]","learning_rate=(1e-4, 1e-2)"]
• search_parameter: If you don't have the search file, you can set this parameter in command line to specify the search space. For example:--search_parameter=hidden_size=[256,512] --search_parameter=embedding_size=[64,128,256] --search_parameter=learning_rate='(1e-4, 1e-2).

Architecture

We have shown the overall architecture of our toolkit in the above Figure. The configuration is specified via command line, external config files and internal config dictionaries. Multiple processors and dataloaders are integrated to process different forms of data. Models and evaluators take charge of doing the training and evaluation. Therefore, input datasets will get prepared and trained based on the specified configuration. You can refer to documentation for more information.

Dataset

We have deployed 8 popular MWP datasets in our toolkit. These datasets are divided into two categories, Single-equation dataset and Multiple-equation dataset, which can be found in the table below. We will keep updating more datasets like ape200k(Zhao et al., 2020), dolphin1878(Shi et al., 2015) and dolphin18k(Huang et al., 2016).

task	dataset	reference
Single-equation dataset	math23k	(Wang et al., 2017)
	asdiv-a	(Miao et al., 2020)
	mawps-single	(Kedziorski et al., 2016)
	mawps_asdiv-a_svamp	(Patel et al., 2021)
Multiple-equation dataset	alg514	(Kushman et al., 2014)
	draw	(Upadhyay et al., 2017)
	mawps	(Kedziorski et al., 2016)
	hmwp	(Qin et al., 2020)

Model

We have deployed 18 deep learning MWP models in our toolkit. Based on the featured generation procedure, we categorize them into Sequence-to-sequence, Sequence-to-tree, Graph-to-tree, VAE and Pre-trained models. Please note Pre-trained models are simple implementation of pretrained language models on MWP solving task. The table is displayed as follows:

type	model	reference
Seq2Seq	DNS	(Wang et al., 2017)
	MathEN	(Wang et al., 2018)
	Saligned	(Chiang et al., 2019)
	GroupATT	(Li et al., 2019)
	RNN	(Sutskever et al., 2014)
	RNNVAE	(Zhang et al., 2016)
	Transformer	(Vaswani et al., 2017)
Seq2Tree	TRNN	(Wang et al., 2019)
	AST-Dec	(Liu et al., 2019)
	GTS	(Xie et al., 2019)
	SAUSolver	(Qin et al., 2020)
	TSN	(Zhang et al., 2020)
Graph2Tree	Graph2Tree	(Zhang et al., 2020)
	MultiE&D	(Shen et al., 2020)
Pre-trained	BertGen	(Devlin et al., 2018)
	RobertaGen	(Liu et al., 2019)
	GPT-2	(Radford et al., 2019)
Updating	KA-S2T	(Wu et al., 2020)
	HMS	(Lin et al., 2021)
	NUM-S2T	(Wu et al., 2021)

Evaluation metric

We have implemented 2 evaluation metrics to measure the effect of MWP models.

measurement	note
Equ acc	The predicted equation is exactly match the correct equation
Val acc	The predicted answer is match the correct answer

Experiment Results

We have implemented the models on the datasets that are integrated within our toolkit. All the implementation follows the build-in configurations. All the experiments are conducted with 5 cross-validation. The experiment results(Equ acc|Val acc) are displayed in the following table.

Single-equation Task Results

model	Dataset
	math23k		mawps-single		asdiv-a		mawps_asdiv-a_svamp
	Equ. Acc	Ans. Acc	Equ. Acc	Ans. Acc	Equ. Acc	Ans. Acc	Equ. Acc	Ans. Acc
DNS	57.1	67.5	78.9	86.3	63.0	66.2	22.1	24.2
MathEN	66.7	69.5	85.9	86.4	64.3	64.7	21.8	25.0
Saligned	59.1	69.0	86.0	86.3	66.0	67.9	23.9	26.1
GroupATT	56.7	66.6	84.7	85.3	59.5	61.0	19.2	21.5
AttSeq	57.1	68.7	79.4	87.0	64.2	68.3	23.0	25.4
LSTMVAE	59.0	70.0	79.8	88.2	64.0	68.7	23.2	25.9
Transformer	52.3	61.5	77.9	85.6	57.2	59.3	18.4	20.7
TRNN	65.0	68.1	86.0	86.5	68.9	69.3	22.6	26.1
AST-Dec	57.5	67.7	84.1	84.8	54.5	56.0	21.9	24.7
GTS	63.4	74.2	83.5	84.1	67.7	69.9	25.6	29.1
SAU-Solver	64.6	75.1	83.4	84.0	68.5	71.2	27.1	29.7
TSN	63.8	74.4	84.0	84.7	68.5	71.0	25.7	29.0
Graph2Tree	64.9	75.3	84.9	85.6	72.4	75.3	31.6	35.0
MultiE&D	65.5	76.5	83.2	84.1	70.5	72.6	29.3	32.4
BERTGen	64.8	76.6	79.0	86.9	68.7	71.5	22.2	24.8
RoBERTaGen	65.2	76.9	80.8	88.4	68.7	72.1	27.9	30.3
GPT-2	63.8	74.3	75.4	75.9	59.9	61.4	22.5	25.7

Multiple-equation Task Result

model	Dataset
	draw		hmwp
	Equ. Acc	Ans. Acc	Equ. Acc	Ans. Acc
DNS	35.8	36.8	24.0	32.7
MathEN	38.2	39.5	32.4	43.7
Saligned	36.7	37.8	31.0	41.8
GroupATT	30.4	31.4	25.2	33.2
AttSeq	39.7	41.2	32.9	44.7
LSTMVAE	40.9	42.3	33.6	45.9
Transformer	27.1	28.3	24.4	32.4
TRNN	27.4	28.9	27.2	36.8
AST-Dec	26.0	26.7	24.9	32.0
GTS	38.6	39.9	33.7	44.6
SAU-Solver	38.4	39.2	33.1	43.7
TSN	39.3	40.4	34.3	44.9
Graph2Tree	39.8	41.0	34.4	45.1
MultiE&D	38.1	39.2	34.6	45.3
BERTGen	33.9	35.0	29.2	39.5
RoBERTaGen	34.2	34.9	30.6	41.0
GPT-2	30.7	31.5	36.3	49.0

## Contributing

We will keep updating and maintaining this repository. You are welcome to contribute to this repository through giving us suggestions and developing extensions! If you have any questions or encounter a bug, please fill an issue.

Cite

If you find MWP toolkit is useful for your research, please cite:

@article{lan2021mwptoolkit,
    title={MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solvers},
    author={Yihuai Lan and Lei Wang and Qiyuan Zhang and Yunshi Lan and Bing Tian Dai and Yan Wang and Dongxiang Zhang and Ee-Peng Lim},
    journal={arXiv preprint arXiv:2109.00799},
    year={2021}
}

License

MWPToolkit uses MIT License.