#1387: Add TextDirectoryCorpus and refactor TextCorpus

[macks22]

... to make modifying preprocessing steps more modular. Consolidate tests in test_corpora. Resolves #1387. This replaces PR #1388.

[menshikh-iv]

Thank you @macks22, you are very active in the last month:1st_place_medal:

[menshikh-iv]

@macks22 please create short PR with fix for this problem (assert equal for pathes on windows), check this SO thread.

======================================================================
FAIL: Test with non-trivial directory structure, shown below:
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\gensim\test\test_corpora.py", line 528, in test_non_trivial_structure
    self.assertEqual(expected, base_names)
AssertionError: Lists differ: [u'0.txt', u'a_folder/1.txt', ... != ['0.txt', 'a_folder\\1.txt', '...
First differing element 1:
u'a_folder/1.txt'
'a_folder\\1.txt'
- [u'0.txt',
?  -
+ ['0.txt',
-  u'a_folder/1.txt',
?  -         ^
+  'a_folder\\1.txt',
?           ^^
-  u'b_folder/2.txt',
?  -         ^
+  'b_folder\\2.txt',
?           ^^
-  u'b_folder/3.txt',
?  -         ^
+  'b_folder\\3.txt',
?           ^^
-  u'b_folder/c_folder/4.txt']
?  -         ^        ^
+  'b_folder\\c_folder\\4.txt']
?           ^^        ^^
-------------------- >> begin captured logging << --------------------
gensim.corpora.textcorpus: INFO: Initializing dictionary
gensim.corpora.dictionary: INFO: adding document #0 to Dictionary(0 unique tokens: [])
gensim.corpora.dictionary: INFO: built Dictionary(2 unique tokens: [u'content', u'document']) from 5 documents (total 10 corpus positions)
--------------------- >> end captured logging << ---------------------