Language-tagged literals with identical `@value` get deduplicated on insert, losing language variants

[acassaignemondeca]

Versions

• fluree-server built from main @ commit e40f5560c (PR rollup: Cumulative work from #1258, #1260, #1261, #1262, #1263 (mis-merged to solo/latest) #1264, 2026-06-01)
• Linux x86_64, local single-node, default file storage

Summary

When a subject has multiple skos:prefLabel (or any langString predicate) triples whose values are identical strings but language tags differ, only one survives the insert. Per the docs (api/endpoints.md:371 mentions a 6th element for the language tag), these should be distinct flakes.

Reproducer

# 1. Create ledger
curl -X POST http://localhost:8090/v1/fluree/create \
  -H 'Content-Type: application/json' \
  -d '{"ledger":"langbug:main"}'

# 2. Insert one concept with 3 prefLabels — en and fr share the value "animal"
curl -X POST http://localhost:8090/v1/fluree/insert/langbug:main \
  -H 'Content-Type: application/json' \
  -d '{
    "@context": {"ex":"http://ex/","skos":"http://www.w3.org/2004/02/skos/core#"},
    "@id":"ex:animal", "@type":"skos:Concept",
    "skos:prefLabel":[
      {"@language":"en","@value":"animal"},
      {"@language":"es","@value":"animales"},
      {"@language":"fr","@value":"animal"}
    ]
  }'

# 3. Read back
curl -s -X POST http://localhost:8090/v1/fluree/query/langbug:main \
  -H 'Content-Type: application/sparql-query' \
  -H 'Accept: application/sparql-results+json' \
  -d 'PREFIX skos:<http://www.w3.org/2004/02/skos/core#>
SELECT ?l (LANG(?l) AS ?lang) WHERE { ?s skos:prefLabel ?l }'

Expected

Three rows: ("animal","en"), ("animales","es"), ("animal","fr").

Actual

Two rows: ("animales","es"), ("animal","fr"). The en triple is silently dropped.

Real-world impact

Ingesting the full GEMET thesaurus (GEMET_en_es_fr.jsonld, multilingual SKOS, eionet.europa.eu):

Lang	In source	After insert	Loss
en	5609	5321	−288
es	5388	5359	−29
fr	5387	5387	0

EN bears the brunt because EN and FR share many literal forms (e.g. "animal", "lichen", many proper nouns and Latin loanwords). FR is fully preserved.

Observed precedence

When (s, p, "v") exists with multiple @language tags, the surviving lang appears to follow a deterministic order — fr > es > en in our runs. This suggests dedup is performed before the language tag is taken into account, with insertion order or alphabetical order on lang code deciding the winner.

Suspected area

Flake encoding / dictionary lookup for langString literals. The 6th element (language tag) documented at docs/api/endpoints.md:371 should be part of the flake key, not metadata appended after dedup.