Basic use case has poor results

[johannespn]

Hey, first of all, thanks for starting this project!

Problem

I successfully loaded sample product data into the local qdrant db. In total, there are only 42 products in the json_with_embeddings/*.json, three of which are <$20.

Then I queried for products below 20 dollars. The top 2 results are >$20 and only the third is <$20.

Also it hallucinated on the description:

while

{
	"@context": "https://schema.org",
	"@type": "Product",
	"name": "Choker with Gold Pendant",
	"offers": {
		"@type": "Offer",
		"priceCurrency": "USD",
		"price": "29.99",
		"availability": "https://schema.org/InStock"
	}
}

Also other queries like "I am looking for jewelry for men" failed to find the one product in 42 that was categorized as male in description and PeopleAudience.suggestedGender.

I tried all the queries multiple times, the system never got it correct.

Expectation

I would expect the out of the box set up to perform better on these very simple queries and set up.

Using text-embedding-3-small, gpt-4o-mini and qdrant_local. And this is my data set
products_jsonld.json

[genericrobot77]

I'm pretty sure you need to configure the db loader to get (specifically target) the Schema.org "Types" you are using. Out of the box it is configured for movie data base, sci-fi, real estate, etc.

You should also reduce the "Temperature" of the LLM, it is default 0.7. I had a similar issue, reduced it to 0.2 and that stopped hallucinations.
/NLWeb/code/llm/openai.py

You could also add prompts relevant to your "Site" and "Type". https://github.com/microsoft/NLWeb/blob/main/code/prompts/site_type.xml

[rvguha]

Thank you for raising this issue. I will look into it. guha

…

On Thu, Jun 12, 2025 at 3:17 PM Johannes ***@***.***> wrote: *johannespn* created an issue (microsoft/NLWeb#198) <#198> Hey, first of all, thanks for starting this project! Problem I successfully loaded sample product data into the local qdrant db. Then I queried for products below 20 dollars. In total, there are only 42 products in the json_with_embeddings/*.json, three of which are <$20. The top 2 results are >$20 and only the third is <$20. Also it hallucinated on the description: Screenshot.2025-06-12.at.22.40.42.png (view on web) <https://github.com/user-attachments/assets/ee739a56-6ed9-47da-b4b1-a5c63a36df06> while { ***@***.***": "https://schema.org", ***@***.***": "Product", "name": "Choker with Gold Pendant", "offers": { ***@***.***": "Offer", "priceCurrency": "USD", "price": "29.99", "availability": "https://schema.org/InStock" } } Also other queries like "I am looking for jewelry for men" failed to find the one product in 42 that was categorized as male in description and PeopleAudience.suggestedGender. I tried all the queries multiple times, the system never got it correct. Expectation I would expect the out of the box set up to perform better on these very simple queries and set up. Using text-embedding-3-small, gpt-4o-mini and qdrant_local. And this is my data set products_jsonld.json <https://github.com/user-attachments/files/20716908/products_jsonld.json> — Reply to this email directly, view it on GitHub <#198>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABICKCX3NH37MAEM64JQOZL3DH363AVCNFSM6AAAAAB7GSRIDCVHI2DSMVQWIX3LMV43ASLTON2WKOZTGE2DCNRZGA4DGNY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[johannespn]

You should also reduce the "Temperature" of the LLM

Thanks! I reduced it to 0.2, but it still doesn't get the price right. Interestingly, switching to

models:
      high: gpt-4.1
      low: gpt-4.1-mini

with temperature=0.2 yields no results at all for the price query, but the bracelet for men query works fine. jewelry for men still doesn't show the bracelet on the top.

I also checked regarding Schema.org types and found the skip_types logic, but the relevant PeopleAudience.suggestedGender is not included here, so it should be embedded properly. All of my data entries are of Schema.org type Product, so they are also included.

To me, these seem like the most basic use cases that should be working even without LLM support. So when throwing in an LLM, this has to work out of the box to be adopted for production use.

[genericrobot77]

I see your point. and yes, not much use for you if it can't answer a simple query when your content is marked up with one of the most common schema.org Types. There may be an opportunity for an NLWeb support if Microsoft would like people to adopt this and not have to fumble their way through. More documentation would be great.