New lexical backend MLLM

[osma]

This (draft) PR adds a new Annif backend called MLLM, which stands for "Maui-like Lexical Matching". This is a Python reimplementation of the Maui algorithm (the parts related to matching using a controlled vocabulary), but with some important differences. For example, the initial matching between subject labels and input text is done using sets of tokens, one sentence at a time, instead of n-grams as Maui does. The intent is that this backend would eventually replace the use of Maui, since it's a lot easier to set up and use a built-in algoritm instead of an external Maui Server instance.

The code is in an early draft state and a number of things need to be improved before this can be merged, but I'm opening the PR now to get feedback from automated tools. Here are some TODO items:

• support --cached option for training with previously preprocessed training data
• refactor by moving some classes to a separate module (e.g. annif.lexical)
• add support for semantic features based on vocabulary structure (broader, narrower, related)
• add support for collection membership feature
  • verify that this works also without SKOS Collections in the vocabulary
• support hyperparameter optimization using the annif hyperopt command
• complete unit tests
• fix QA tool warnings
• wiki documentation

[codecov]

Codecov Report

Merging #462 (57a2a38) into master (b8e556c) will increase coverage by 0.02%.
The diff coverage is 100.00%.

❗ Current head 57a2a38 differs from pull request most recent head aa2c287. Consider uploading reports for the commit aa2c287 to get more accurate results

@@            Coverage Diff             @@
##           master     #462      +/-   ##
==========================================
+ Coverage   99.44%   99.46%   +0.02%     
==========================================
  Files          67       73       +6     
  Lines        4838     5280     +442     
==========================================
+ Hits         4811     5252     +441     
- Misses         27       28       +1     

Impacted Files	Coverage Δ
annif/backend/__init__.py	100.00% <100.00%> (ø)
annif/backend/mllm.py	100.00% <100.00%> (ø)
annif/corpus/subject.py	100.00% <100.00%> (ø)
annif/lexical/mllm.py	100.00% <100.00%> (ø)
annif/lexical/tokenset.py	100.00% <100.00%> (ø)
annif/suggestion.py	100.00% <100.00%> (ø)
tests/conftest.py	100.00% <100.00%> (ø)
tests/test_backend_mllm.py	100.00% <100.00%> (ø)
tests/test_backend_omikuji.py	100.00% <100.00%> (ø)
tests/test_lexical_mllm.py	100.00% <100.00%> (ø)
... and 15 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7f42c96...aa2c287. Read the comment docs.

[osma]

Rebased and force-pushed.

[osma]

I've now tackled some QA tool issues, though complaints still remain. Anyway I think this is ready for wider testing now.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
3 Code Smells

No Coverage information
0.0% Duplication

[osma]

I wrote some wiki documentation: https://github.com/NatLibFi/Annif/wiki/Backend:-MLLM
This page has not yet been linked to from the Home page or the sidebar. The links should be added when a version of Annif with MLLM is released.

[osma]

There are still some QA tool warnings, but it's not easy to fix them. I would like to merge this pretty soon, unless there are objections.

[juhoinkinen]

I looked through the code, I could comment only about n_jobs parameter, and probably that would not be very useful (at least before candidate generation is parallelized, which is stated as TODO).

Overall it is great to have MLLM in Annif! :)