This repo consists of the data used for acl 2017
Learning Character-level Compositionality with Visual Features
This is an implementation for crawling the title and its corresponding categories of an Wikipedia page.
The crawler will crawl the categories according to category_list_lang.txt
The dataset is already crawled in the folder acl2017_data
with the following command:
python crawl.py -l zh -n 100000
python crawl.py -l ja -n 100000
python crawl.py -l ko -n 100000
Running the crawler again will cover the data in the folder.
python crawl.py [-h] -l LANG -n NUM
optional arguments:
-h, --help show this help message and exit
-l LANG, --lang LANG which language to crawl [zh, ja, ko]
-n NUM, --num NUM the minimun number of pages you wish to crawl for each category (may not be able to reach this number)
The data split and the code for the paper can be found in this repo