Improved language detection

[andrenatal]

We should have a more expanded language detection heuristic in terms of content extraction and analysis to determine the page's language instead of traversing just through divs.

[andrenatal]

Maybe we could use this https://fasttext.cc/docs/en/language-identification.html

[kpu]

According to #103 (comment) the code no is coming out of CLD2 instead of nb and nn. This is a tell that the small version of the CLD2 classifier is being used. The full version has more languages and better accuracy (including for languages other than Norwegian). See bitextor/warc2text@e2543e4 for an example of using the full CLD2 classifier.
Or use fasttext.