Add new datasets for text classification.

[zhangguanheng66]

Add a few supervised learning datasets, including
- AG_NEWS
- SogouNews
- DBpedia
- YelpReviewPolarity
- YelpReviewFull
- YahooAnswers
- AmazonReviewPolarity
- AmazonReviewFull

[cpuhrsch]

I think you can split this out as well. It seems useful in its own right. But instead of taking a filepath, it can take a string and return a string (or a list of strings).

[zhangguanheng66]

Sure thing. Will do it.

[cpuhrsch]

for filename in os.listdir('dirname):
    with open(filename, 'r') as a, open(filename + '.processed', 'w') as b:
        b.write(text_normalize(a.readline())

[cpuhrsch]

I'd delete this and expect the user to know how to do this

[cpuhrsch]

Maybe make it an example / add it to the tutorial

[zhangguanheng66]

I think we could keep it as internal function.
I would like to support the one-line data loading for the supervised learning dataset so we have to minimize the work on user side.

[cpuhrsch]

Maybe "basic_normalization_english" or such? Is there a reference here (maybe within a paper)? Can be immediately moved into c++. It's very useful in general and we can indicate that.

[zhangguanheng66]

the supervised learning paper didn't mention how they did pre-processing. fastText just did this pre-processing in the bash script.
I think we could talk with some NLP guys and implement a basic pre-processor in C++.

[cpuhrsch]

extract_archive(
    download_from_url(
        torchtext.datasets.supervised.urls['AG_NEWS']) #returns a path in cache, 
        "path/to/AG_NEWS/folder"
    )
)

[cpuhrsch]

I don't this necessary needs to be its own function

[zhangguanheng66]

minor change. Put download and extract_archive together.

[cpuhrsch]

Let's add batching separately if we see that we need it.