v0.4.4
Fix
Semantic search was producing essentially random rankings, especially on short queries. Two underlying issues:
- Missing task prefixes for
nomic-embed-text-v2-moe— the model requiressearch_document:on documents andsearch_query:on queries; without them embeddings carry little semantic signal. - Embedded text was too sparse — for ISTAT/Eurostat dataflows whose only description is a short title (e.g.
Prezzi al consumo), there's not enough context for the model.
Changes
- Add the required
search_document:/search_query:prefixes - Prepend the
df_idto the embedding text (semantic tokens likeCPI,POPgive extra signal) - When the category cache is populated (
opensdmx treewas run before), enrich each embedding with the dataflow's scheme/category names. Pure cache lookup — never triggers a live fetch fromembed. - Diagnostic message:
embednow reports how many descriptions got enriched and suggestsopensdmx treewhen the cache is empty
Impact
ISTAT query inflazione, top 5: from 2/5 relevant → 5/5 relevant (all IPCA / IPAB datasets).
Eurostat query inflation, top 5: all HICP / inflation datasets.
Action required
After upgrading, rebuild your embeddings:
```bash
opensdmx tree # populate category cache (if you haven't yet)
opensdmx embed # rebuild for default provider
opensdmx embed -p istat # rebuild for ISTAT
```