NLP Example doesn't work

[jcampbell05]

The NLP example provided in the README currently causes markovify to spit out aload of Nones instead of sentances.

[jsvine]

Thanks for flagging, James. Could you provide the exact code, and any associated text files, that trigger this problem?

[jcampbell05]

Brexiteer.zip

Yeah here is the whole code, it just is using tweets as a source.

[jsvine]

Strange: When I click the "Uploading Brexiteer.zip..." link in your comment, I just get redirected back to this thread, but with no attachment.

[jcampbell05]

@jsvine I've fixed it try now :)

[jsvine]

Thanks, James. Here's what's happening:

• markovify.Text expects the sentences in a corpus to be separated by periods.
• In your strip_all_entities function, you're removing all periods.
• Hence, markovify.Text thinks you've just fed it one giant sentence and, hence, has trouble producing a novel new sentence.

Fortunately, fixing is easy and simply involves using markovify.NewlineText instead. Just change this line in your script:

class POSifiedText(markovify.Text):

... to this:

class POSifiedText(markovify.NewlineText):