Should I put fastai_contrib subdir under ulmfit dir?

[alexwwang]

By default, it would report an error that "the fastai_contrib module is not found" when I execute postprocess_wikitext.py.

So I have to move it under ulmift dir and make it work properly.

I wonder, if this is a necessary step or do you meet this little slit?

[tpietruszka]

I have stumbled upon the same problem. Another possible solution - calling from the main directory:

python -m ulmfit.postprocess_wikitext --path="data/wiki/wikitext-2" --lang="en"

I am currently trying to reproduce all the steps - starting with training on wikitext - writing down all the commands on the way.

Not sure what is the intended approach though.

[PiotrCzapla]

The way @tpietruszka showed is what I'm using.
There are .sh scripts that should deal with this. But Guys there is a bug in postprocess_wikitext as it removes information about articles split in wikipedia which basically breaks the downstream tasks. So if you want to give it a try, try to fix the postprocess_wikitext.py so that it includes a way to split the wikipedia text by articles. see #25. This is an easy fix but I'm away Today so if you want to start training implement the fix. If you aren't in hurry just wait with your experiments until #25 is fixed