Skip to content

Annif 1.5

Latest

Choose a tag to compare

@juhoinkinen juhoinkinen released this 12 Nov 11:22
· 1 commit to main since this release
Immutable release. Only release title and notes can be modified.
v1.5.0
5b8c8aa

This release includes notable backend enhancements: new EBM and dm_rerank backends are added and the existing NN ensemble backend is reimplemented using PyTorch.

The EBM stands for embedding-based matching. The approach uses a sentence transformer model to vectorize both the labels in the subject vocabulary and input texts to numerical vectors. The vectors of input texts are compared to those of the vocabulary to give candidate subjects, which are then ranked utilizing a model fit to the features of the training documents to give the final suggestions. The backend has been developed by the German National Library (DNB) and it uses the ebm4subjects library.

The dm_rerank is an ensemble style backend that uses a decision model service such as Jev to re-score/re-rank the candidate subjects proposed by the source backends. It supports any decision model with a /v1/systemone API endpoint implemented in recent versions of llama.cpp

Other enhancements include parallelization in the MLLM backend and in the optimization of limit and threshold values, better handling of vocabularies in ensemble projects and adding Pareto-front output to the optimize CLI command. The project configurations can now contain environment variables, which is intended for possibly private information like URLs of external services.

The project's security posture has been improved by switching to use (PyPI) Trusted Publishing, dependency cooldowns and network-traffic monitoring of CI/CD runs. We have also transferred the PyPI project under NatLibFi organization account and enabled immutability of the GitHub releases.

Maintenance changes include switching from Poetry to uv for dependency management. As always, also some bugs have been fixed.

For this release we received many community contributions (some assisted by AI tools). Many thanks to @mfakaehler, @RietdorfC, @mjsuhonos and @avalyset!

Supported Python versions:

  • 3.11, 3.12, 3.13 and 3.14

Backward compatibility:

  • ⚠️ NN ensemble projects trained with Annif 1.4 or older need to be retrained.
  • For other projects, the warnings by SciKit-learn are harmless.
  • ⚠️ This is very likely the last Annif minor release to support the current fasttext backend, because the original fastText library is no longer maintained and there are compatibility issues with other libraries. We are looking for alternative implementations of fasttext.

What's Changed

  • [StepSecurity] ci: Harden GitHub Actions by @juhoinkinen in #898
  • Bump the github-actions group across 2 directories with 5 updates by @dependabot[bot] in #909
  • Configure cooldown of 5 days for Dependabot updates of GH Actions by @juhoinkinen in #917
  • Bump the github-actions group across 2 directories with 4 updates by @dependabot[bot] in #921
  • Use uv for managing dependencies by @osma in #923
  • Fix documentation for usage of uv sync by @mfakaehler in #935
  • Mock requests in test for uploading project to nonexistent HF repo by @juhoinkinen in #939
  • Reimplement NN ensemble using PyTorch by @osma in #926
  • Configure cool-down period for uv-handled dependencies by @juhoinkinen in #944
  • Upgrade to Schemathesis 4 and Pytest 9 by @juhoinkinen in #879
  • Upgrade to schemathesis~=4.25.2 by @juhoinkinen in #953
  • Remove unused environment variables from CICD by @juhoinkinen in #954
  • Bump the github-actions group across 2 directories with 12 updates by @dependabot[bot] in #947
  • Remove support for Python 3.10 by @juhoinkinen in #956
  • Update dependencies for version 1.5 by @juhoinkinen in #955
  • Bump the github-actions group across 2 directories with 7 updates by @dependabot[bot] in #965
  • Update Docker base image to use Python 3.13 and Debian Trixie by @juhoinkinen in #961
  • Re-enable schemathesis functions by @juhoinkinen in #968
  • Update CICD by @juhoinkinen in #971
  • Enable support for Python 3.14 / Force 'fork' strategy for child process by @juhoinkinen in #958
  • Fix slow-down of CLI startup by @juhoinkinen in #963
  • Fix MultiPartException for field-sizes over 1024KB by @juhoinkinen in #975
  • Add parallel candidate generation for MLLM hyperopt by @mjsuhonos in #952
  • Bump the github-actions group across 2 directories with 8 updates by @dependabot[bot] in #977
  • CLI optimize enhancements: parallel evaluation, results file output, Pareto front calculation by @mjsuhonos in #967
  • Align ensemble source vocabularies by @mjsuhonos in #974
  • Fix HTTP backend crash for suggestions with URIs not in local vocabulary by @juhoinkinen in #978
  • Decision model reranking backend by @osma in #966
  • Environment variable interpolation for config files by @osma in #979
  • Add ebm backend by @mfakaehler in #914
  • Raise a readable error when a vocabulary file fails to parse by @avalyset in #943
  • Treat non-positive and non-finite ensemble source weights as small by @avalyset in #980
  • Expose project metadata fields via REST API by @avalyset in #946
  • Update dependencies for version 1.5 - round 2 by @juhoinkinen in #976

New Contributors

Full Changelog: v1.4.0...v1.5.0