Skip to content

v0.4.4

Choose a tag to compare

@aborruso aborruso released this 20 Apr 08:06
· 651 commits to main since this release

Fix

Semantic search was producing essentially random rankings, especially on short queries. Two underlying issues:

  1. Missing task prefixes for nomic-embed-text-v2-moe — the model requires search_document: on documents and search_query: on queries; without them embeddings carry little semantic signal.
  2. Embedded text was too sparse — for ISTAT/Eurostat dataflows whose only description is a short title (e.g. Prezzi al consumo), there's not enough context for the model.

Changes

  • Add the required search_document: / search_query: prefixes
  • Prepend the df_id to the embedding text (semantic tokens like CPI, POP give extra signal)
  • When the category cache is populated (opensdmx tree was run before), enrich each embedding with the dataflow's scheme/category names. Pure cache lookup — never triggers a live fetch from embed.
  • Diagnostic message: embed now reports how many descriptions got enriched and suggests opensdmx tree when the cache is empty

Impact

ISTAT query inflazione, top 5: from 2/5 relevant → 5/5 relevant (all IPCA / IPAB datasets).
Eurostat query inflation, top 5: all HICP / inflation datasets.

Action required

After upgrading, rebuild your embeddings:

```bash
opensdmx tree # populate category cache (if you haven't yet)
opensdmx embed # rebuild for default provider
opensdmx embed -p istat # rebuild for ISTAT
```