Allow text-dict style vocab on LM Dataset

[JackTemaki]

The default output vocabulary of the https://github.com/rwth-i6/subword-nmt repo is a python formatted dictionary in text form. The OggZipDataset can use it, but the LMDataset did not yet, so I added that capability.

[albertz]

We should never check just using such generic exception logic. You should instead use some heuristic, eg check the first few non-space characters, that it starts with {"...": ..., ...}.

[JackTemaki]

Ok I used a regex now.

[albertz]

The regex should check the non whitespace characters, from the beginning on (^), and then check that the first non-white space char is {, and then the next is ", and then it matches a pattern like you did.

You should not read the whole file at that point but just the first 1024 bytes or so.

I think it's then easier and clearer when you filter out the whitespace chars in that header, and then do the regex on the filtered header.

You also should not use re.compile as this is not needed here but just re.match.

You also should use our efficient literal_eval (I think in utils, see other related code) instead of eval.

[JackTemaki]

Ok! makes sense.

[albertz]

.+ and [0-9]+ instead of the *.

[albertz]

Otherwise ok.