lingua.Unknown is not handled appropriately if included in the set of input languages

[marians]

I have started testing the library a few days ago and just saw a first nil pointer panic like this:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4314fd8]

goroutine 22414 [running]:
github.com/pemistahl/lingua-go.loadJson(0xa834340, 0x45537a0)
	/Users/marian/go/pkg/mod/github.com/pemistahl/lingua-go@v1.0.3/json.go:37 +0x178
github.com/pemistahl/lingua-go.languageDetector.loadLanguageModels({{0xc01622a700, 0x1f, 0x1f}, 0x0, {0xc017283f80, 0xc, 0x10}, 0xc01312f320, 0xa834340, 0xa834240, ...}, ...)
	/Users/marian/go/pkg/mod/github.com/pemistahl/lingua-go@v1.0.3/detector.go:612 +0x8d
github.com/pemistahl/lingua-go.languageDetector.lookUpNgramProbability({{0xc01622a700, 0x1f, 0x1f}, 0x0, {0xc017283f80, 0xc, 0x10}, 0xc01312f320, 0xa834340, 0xa834240, ...}, ...)
	/Users/marian/go/pkg/mod/github.com/pemistahl/lingua-go@v1.0.3/detector.go:552 +0x146
github.com/pemistahl/lingua-go.languageDetector.computeSumOfNgramProbabilities({{0xc01622a700, 0x1f, 0x1f}, 0x0, {0xc017283f80, 0xc, 0x10}, 0xc01312f320, 0xa834340, 0xa834240, ...}, ...)
	/Users/marian/go/pkg/mod/github.com/pemistahl/lingua-go@v1.0.3/detector.go:526 +0x145
github.com/pemistahl/lingua-go.languageDetector.computeLanguageProbabilities({{0xc01622a700, 0x1f, 0x1f}, 0x0, {0xc017283f80, 0xc, 0x10}, 0xc01312f320, 0xa834340, 0xa834240, ...}, ...)
	/Users/marian/go/pkg/mod/github.com/pemistahl/lingua-go@v1.0.3/detector.go:484 +0xcb
github.com/pemistahl/lingua-go.languageDetector.lookUpLanguageModels({{0xc01622a700, 0x1f, 0x1f}, 0x0, {0xc017283f80, 0xc, 0x10}, 0xc01312f320, 0xa834340, 0xa834240, ...}, ...)
	/Users/marian/go/pkg/mod/github.com/pemistahl/lingua-go@v1.0.3/detector.go:452 +0xb7
created by github.com/pemistahl/lingua-go.languageDetector.ComputeLanguageConfidenceValues
	/Users/marian/go/pkg/mod/github.com/pemistahl/lingua-go@v1.0.3/detector.go:176 +0x455

Seems as if even reading from an embed File can fail at some point.

I'm using go version go1.17.2 darwin/amd64.

[pemistahl]

Thank you for reporting this @marians. Can you please show me your code which calls the library? When exactly did this error occur? I need more context to find out what's going on.

[marians]

The code is not public currently. I'll copy a reduced version of my invocation here.

languages := []lingua.Language{
	lingua.Albanian,
	lingua.Basque,
	lingua.Bosnian,
	lingua.Bulgarian,
	lingua.Chinese,
	lingua.Catalan,
	lingua.Croatian,
	lingua.Czech,
	lingua.Danish,
	lingua.Dutch,
	lingua.English,
	lingua.Estonian,
	lingua.Finnish,
	lingua.French,
	lingua.German,
	lingua.Greek,
	lingua.Hungarian,
	lingua.Italian,
	lingua.Japanese,
	lingua.Latvian,
	lingua.Lithuanian,
	lingua.Macedonian,
	lingua.Polish,
	lingua.Portuguese,
	lingua.Romanian,
	lingua.Serbian,
	lingua.Slovak,
	lingua.Slovene,
	lingua.Spanish,
	lingua.Swedish,
	lingua.Unknown,
}

detector := lingua.NewLanguageDetectorBuilder().
	FromLanguages(languages...).
	Build()

// This part is called within a loop for many different text values.
confidenceValues := detector.ComputeLanguageConfidenceValues(text)
if len(confidenceValues) > 0 {
	result := confidenceValues[0].Language().IsoCode639_1().String()
}

[pemistahl]

Ah, I think I know what's wrong. You are trying to build the language detector by including lingua.Unknown. This does not make sense because this is not a real language. The function loadJson tries to find language models for lingua.Unknown but it cannot find any, of course. I've forgotten to check for lingua.Unknown and handle this case appropriately, so your issue is still valid. I'm going to fix it soon.

For the time being, please remove lingua.Unknown from your set of languages and you should be fine.

[marians]

Thanks for the info!

My expectation (without reading up in this) was that having Unknown in the list, the classifier would be able to give "unknown" as the most likely result if the language was in fact not classified as one of the other ones. If that makes sense.

[pemistahl]

Well, one could guess so. But no, if the language cannot be reliably detected, lingua.Unknown is returned no matter if it has been included in the set of languages or not. I will make this more clear in the documentation. Thanks again for making me aware of this.