The repo contains code for our AAAI-2020 paper
Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning
Wenhe Zhang, Chi Zhang, Yixin Zhu, Song-Chun Zhu
The 34th AAAI Conferences on Artificial Intelligence, Februrary 7-12, New York (Oral Presentation).
webpage | dataset
As a comprehensive indicator of mathematical thinking and intelligence, the number sense bridges the induction of symbolic concepts and the competence of problem-solving. To endow such a crucial cognitive ability to machine intelligence, we propose a dataset, Machine Number Sense (MNS), consisting of visual arithmetic problems automatically generated using a grammar model---AOG.
These visual arithmetic problems are in the form of geometric figures: each problem has a set of geometric shapes as its context and embedded number symbols. Solving such problems is not trivial; the machine not only has to recognize the number, but also to interpret the number with its contexts, shapes, and relations (eg, symmetry) together with proper operations.
Our comprehensive experiments show that current neural-network-based models still struggle to understand number concepts and relational operations. A simple brute-force search algorithm could work out some of the problems without context information, but taking geometric context into account by an additional perception module would provide a sharp performance gain with fewer search steps. Altogether, we call for attention in fusing the classic search-based algorithms with modern neural networks to discover the essential number concepts in future research.
The dataset is generated by by parsing and sampling an And-Or Graph (AOG) model. Each problem has an internal hierarchical tree structure composed of And-nodes and Or-nodes; an And-node denotes a decomposition of a larger entity in the grammar, and an Or-node denotes an alternative decomposition.
- We design three problem types: (a) Combination, (b) Composition, and (c) Partition, each of which has a distinctive layout.
- Each problem contains two important components:
- Layout component serves as the problem context. The attributes vary with different problem types.
- Algebra component serves as the problem content.
A crucial attribute is the styles of interpretation, including holistic view and analytic view.
- From a holistic perspective, all the numbers in a panel are involved in the same calculation process together as a whole.
- From an analytic perspective, the numbers are grouped as several parts, and each part undergoes an individual calculation process.
We benchmark the proposed MNS dataset using four predominant neural network models and two search-based algorithms. Additionally, human performance on the dataset has also been collected. For more details, please refer to our paper.
| Method | Mean Acc | Combination | Composition | Partition | |||
| Holistic | Analytic | Holistic | Analytic | Holistic | Analytic | ||
| Pure Symbolic Search | 52.15% | 62.98% | 56.83% | 22.17% | 53.73% | 51.29% | 71.60% |
| Context-guided Search | 56.70% | 64.38% | 56.08% | 29.81% | 61.84% | 59.70% | 67.59% |
| CNN | 22.71% | 25.25% | 19.65% | 22.53% | 20.07% | 24.44% | 23.25% |
| LSTM | 22.16% | 24.57% | 21.10% | 22.21% | 20.12% | 23.36% | 23.83% |
| RN | 22.96% | 27.05% | 20.47% | 22.93% | 20.27% | 25.81% | 23.64% |
| ResNet | 25.29% | 27.90% | 24.22% | 23.42% | 23.73% | 26.61% | 27.78% |
| Human | 77.58% | 66.82% | 93.64% | 61.36% | 78.18% | 77.27% | 88.18% |
Important
- Python 2.7
- OpenCV
- PyTorch
- CUDA and cuDNN expected
See requirements.txt for a detailed list of packages required
Code for dataset generation resides in src/dataset folder.
To generate a dataset, run main.py with adjustable sample amount.
Code to benchmark the dataset lives in src/model folder.
Run the following command to check performance.
python main.py train --model-name <model>
If find our work helpful, please cite us.
@inproceedings{zhang2020number,
author={Zhang, Wenhe and Zhang, Chi and Zhu, Yixin and Zhu, Song-Chun},
title={Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning},
booktitle={AAAI Conference on Artificial Intelligence (AAAI)},
year={2020}
}