Preprocessing for the OneBillion dataset

[ahmedmagdiosman]

In the README there's no mention of how you generate the train.txt and test.txt for the OneB dataset. Could you clarify how do you do that?

Thanks!

[jakezhaojb]

i took a preprocessed version from colleagues. Basically, the train.txt and text.txt are just tokenized sentences. I could paste an example if you need

[ahmedmagdiosman]

Thanks that would be great!

[jakezhaojb]

so head train.txt gives me:

The U.S. Centers for Disease Control and Prevention initially advised school systems to close if outbreaks occurred , then reversed itself , saying the apparent mildness of the virus meant most schools and day care centers should stay open , even if they had confirmed cases of swine flu .
When Ms. Winfrey invited Suzanne Somers to share her controversial views about bio-identical hormone treatment on her syndicated show in 2009 , it won Ms. Winfrey a rare dollop of unflattering press , including a Newsweek cover story titled " Crazy Talk : Oprah , Wacky Cures & You . "
Elk calling -- a skill that hunters perfected long ago to lure game with the promise of a little romance -- is now its own sport .
Don 't !
Fish , ranked 98th in the world , fired 22 aces en route to a 6-3 , 6-7 ( 5 / 7 ) , 7-6 ( 7 / 4 ) win over seventh-seeded Argentinian David Nalbandian .
Why does everything have to become such a big issue ?
AMMAN ( Reuters ) - King Abdullah of Jordan will meet U.S. President Barack Obama in Washington on April 21 to lobby on behalf of Arab states for a stronger U.S. role in Middle East peacemaking , palace officials said on Sunday .
To help keep traffic flowing the Congestion Charge will remain in operation through-out the strike and TfL will be suspending road works on major London roads wherever possible .
If no candidate wins an absolute majority , there will be a runoff between the top two contenders , most likely in mid-October .
Authorities previously served search warrants at Murray 's Las Vegas home and his businesses in Las Vegas and Houston .

[ahmedmagdiosman]

Great thanks!