Types are missing in blocked tokenization

[koheiw]

When there are more than 10000 documents, types in the earlier blocks are ignored.

[koheiw]

> texts(data_corpus_guardian)[[100000]]
[1] "Government lawyers are seeking to block compensation payments to people who developed the devastating sleep disorder, narcolepsy, as a result of a faulty swine flu vaccine. | The Pandemrix vaccine made by GlaxoSmithKline (GSK) was given to 6 million people in Britain and millions more across Europe during the 2009-10 swine flu pandemic, but was withdrawn when doctors noticed a rise in narcolepsy cases among those who received the jab. | In June, a 12-year-old boy was awarded 120,000 by a court that ruled he had been left severely disabled by narcolepsy caused by Pandemrix. The win ended a three-year battle with the government that argued his illness was not serious enough to warrant compensation. | Narcolepsy is a permanent condition that can cause people to fall asleep dozens of times a day, even when they are in mid-conversation. Some suffer from night terrors and a problem with muscular control called cataplexy that can lead them to collapse on the spot. | Related: Narcolepsy a... <truncated>
> head(toks[[100000]], 30)
 [1] "counter"       "housebuilding" "disordered"    "finished"      "and"           "secretary"     "firearms"     
 [8] "predominately" "and"           "real"          "remarkably"    "God"           "challenges"    "killed"       
[15] "hoodies"       "hugged"        "to"            "Making"        "to"            "sin"           "because"      
[22] "conclusion"    "the"           "because"       "remainder"     "messiah"       "Odysseus"      "soul"         
[29] "."             "this"        

[koheiw]

We need to add a test for this using package options.