v0.1.7

What's Changed

• minor: fix llm expansion in docs by @glenacota in #847
• Update llama_index imports by @sestinj in #849
• feat: Change parsers and add custom critic model by @shahules786 in #838
• fix: change documentation to reflect accurate method name by @dbtreasure in #837
• Update answer_relevance.md by @carllapierre in #832
• fix ci: some linting issues by @jjmachan in #852

New Contributors

• @glenacota made their first contribution in #847
• @sestinj made their first contribution in #849
• @dbtreasure made their first contribution in #837
• @carllapierre made their first contribution in #832

Full Changelog: v0.1.6...v0.1.7

v0.1.6

What's Changed

• Fix: json format inconsistency in context recall (prompt for statemen… by @jokokojote in #776
• fix: Explanaitions typo on index.md by @dcamejo1 in #801
• feat(integrations): add support for langchain and langsmith by @jjmachan in #795
• Docs: fix confusion regarding getting started by @shahules786 in #815
• docs: add openlayer integration by @shahules786 in #827
• fix: Eventloop unclosed by @yunfwe in #796
• More robust JSON prompting and parsing by @mrtj in #807
• fix(integration): langchain integration and CI fixes by @jjmachan in #830
• Added more examples to the translation prompts by @baswenneker in #826
• fix: use aparse in all metrics by @shahules786 in #831

New Contributors

• @dcamejo1 made their first contribution in #801
• @yunfwe made their first contribution in #796
• @mrtj made their first contribution in #807
• @baswenneker made their first contribution in #826

Full Changelog: v0.1.5...v0.1.6

v0.1.5

What's Changed

• fix: handle string score in node filter by @shahules786 in #755
• Add ipython notebook demonstrating the ragas-haystack integration. by @annthurium in #754
• Update Bring Your Own LLM Docs Link by @pmbaumgartner in #765
• fix: edge case in answer_correctness by @shahules786 in #778
• Added metadata field to datarows by @rnbokade in #782
• fix: is_async not passed for llm generation in context recall by @jokokojote in #777
• fix: Nan in metrics/test set gen by @shahules786 in #786
• Fix adapt issue for keyphrase extractor by @smodlich in #766
• updated context_scoring_prompt and provide extra examples. by @nofrish in #725
• fix: faithfulness by @shahules786 in #787

New Contributors

• @annthurium made their first contribution in #754
• @rnbokade made their first contribution in #782
• @smodlich made their first contribution in #766
• @nofrish made their first contribution in #725

Full Changelog: v0.1.4...v0.1.5

v0.1.4

What's Changed

• docs: Add docs for new entity based metric by @sky-2002 in #674
• Fix typos that cause fatal assertion errors in prompts by @joy13975 in #676
• fix: edge case in faithfulness by @shahules786 in #682
• docs: Fix the precision calculation in context-precision metric doc by @amit-timalsina in #685
• fix: improving reproducability of evals when running everything together by @jjmachan in #671
• added octlane docs by @jjmachan in #695
• Docs: Fix content precision description by @jokokojote in #693
• Docs: Created own section for Calculation (instead of Dropdown) by @GeroVanMi in #696
• Fix: Enable Critique Prompt Customization in AspectCritique by @AaronKim218 in #700
• chore: improve bug/question template by @shahules786 in #702
• docs: clean and smooth by @shahules786 in #703
• Docs: Fix and supplement answer relevancy description by @jokokojote in #705
• fix: remove unwanted quotes from questions by @shahules786 in #708
• Readme Formatting by @gkamradt in #710
• docs: customising embeddings and llms by @shahules786 in #712
• chore: add copyright by @shahules786 in #716
• feat: test set generation improvements by @shahules786 in #719
• Adding Tonic Validate integration notebook by @akamor in #707
• fix: edge case in CR by @shahules786 in #728
• fix(ragas): remove mutable defaults by @mspronesti in #684
• fix: remove experimental filters by @shahules786 in #743
• Correct Faithfulness Verdict Map by @pmbaumgartner in #739
• feat(generator): extend construction to any langchain LLM and Embeddings by @mspronesti in #670
• docs: minor corrections by @shahules786 in #747

New Contributors

• @amit-timalsina made their first contribution in #685
• @jokokojote made their first contribution in #693
• @GeroVanMi made their first contribution in #696
• @AaronKim218 made their first contribution in #700
• @gkamradt made their first contribution in #710
• @akamor made their first contribution in #707
• @mspronesti made their first contribution in #684
• @pmbaumgartner made their first contribution in #739

Full Changelog: v0.1.3...v0.1.4

v0.1.3

What's Changed

• feat: improved node selection and question fixing by @shahules786 in #644
• docs: add explanation to metrics by @shahules786 in #658
• fix: edge cases in metrics by @shahules786 in #661
• Update evolution by @cuent in #666
• feat: Add new "entity-based context recall" metric by @sky-2002 in #667
• fix: filter and context selection by @shahules786 in #672
• Add support for optional max concurrency by @joy13975 in #643

New Contributors

• @cuent made their first contribution in #666
• @sky-2002 made their first contribution in #667

Full Changelog: v0.1.2...v0.1.3

v0.1.2

What's Changed

• docs: arize/ ragas notebook by @axiomofjoy in #594
• fix: prompt adaptation type checks by @shahules786 in #615
• fix: node indexing in testset generator by @shahules786 in #617
• Update ragas-arize.ipynb by @PubliusAu in #618
• docs: getting started by @jjmachan in #589
• Added testset generation for bedrock by @VpkPrasanna in #626
• Made embeddings and LLMs dependent on metric in evaluate function by @peanutshawny in #628
• docs: bring your own llms and embeddings by @jjmachan in #631
• docs: fix import in the example code: ContextRelevance -> ContextRelevancy by @izikeros in #646
• Fix typo: Teset -> Testset by @joy13975 in #638
• Update ragas-arize.ipynb by @PubliusAu in #637
• renaming ground_truths to ground_truth. by @alexjyong in #640
• fix: added types so you can pass langchain llms and embeddings too by @jjmachan in #652

New Contributors

• @axiomofjoy made their first contribution in #594
• @PubliusAu made their first contribution in #618
• @VpkPrasanna made their first contribution in #626
• @peanutshawny made their first contribution in #628
• @izikeros made their first contribution in #646
• @joy13975 made their first contribution in #638
• @alexjyong made their first contribution in #640

Full Changelog: v0.1.1...v0.1.2

v0.1.1

What's Changed

• Fixing typos, vertextai to vertex ai by @mrchtr in #572
• fix typo by @Kyeongpil in #570
• fix assertions and quality in prompt adaptation by @shahules786 in #574
• fix: ensure dict type by @shahules786 in #578
• style: remove additional comments in prompt.py by @leehanchung in #577
• Add stepkurniawan community contribution by @stepkurniawan in #585
• Parse JSON string in adapt() by @mitmul in #584
• fix: replace string formating by dict by @shahules786 in #588
• [patch] Reduce NaN Occurrences by Simple Prompt Modification for JSON Output for context_precision by @i-w-a in #581
• ci: migrate to latest llama-index version by @shahules786 in #603
• Fix: evolution flows for test generation by @shahules786 in #602
• fix: llamaIndex latest version by @jjmachan in #604
• Chore: add language to event tracking by @shahules786 in #592
• feat(llms.json_load): Recursively load json lists by @pberger514 in #593

New Contributors

• @mrchtr made their first contribution in #572
• @Kyeongpil made their first contribution in #570
• @leehanchung made their first contribution in #577
• @stepkurniawan made their first contribution in #585
• @mitmul made their first contribution in #584
• @i-w-a made their first contribution in #581
• @pberger514 made their first contribution in #593

Full Changelog: v0.1.0...v0.1.1

v0.1.0

What's Changed

• add initial embeddings.encode_kwargs by @lucasiscovici in #371
• feat(testset_generator): support error catching in generation process by @yuukidach in #368
• added class for FastEmbed by @tinomaxthayil in #379
• Updated NLI Verdicts to Binary Format by @lucasiscovici in #384
• added docs for ragas embeddings by @tinomaxthayil in #386
• fix context in _answer_correctness.py by @lucasiscovici in #385
• docs: AI copilot implementation by @nickscamara in #294
• fix: faithfulness for evasive/reject answers by @shahules786 in #392
• Added prompt class by @tinomaxthayil in #388
• Context precision with ground truth by @shahules786 in #391
• Added note to answer_relevance by @shahules786 in #399
• Import evaluate function before patching RagasLLM, fixes #301 by @HySoaKa in #377
• Fix: Added ground_truths arg type-hint info in the eval function by @MANISH007700 in #404
• Typo and phrasing correction by @TruscaPetre in #408
• Small improvements to documentation custom LLMs by @TruscaPetre in #410
• feat(metrics): skip embedding process when the corresponding weight is zero by @yuukidach in #409
• added prompt objects to metrics by @tinomaxthayil in #401
• feat: Automatic Prompt adaptation [language] by @shahules786 in #407
• Update _faithfulness.py by @Robs1999 in #416
• fix(metrics): avoid dividing by zero in correctness by @yuukidach in #421
• Feature/doc update ragas score by @prufrock349 in #417
• Docs: Prompts and prompt adaptation by @shahules786 in #419
• fix(metrics): wrong usage of prompt by @yuukidach in #422
• fix: context_utilization name override by @jjmachan in #424
• feat/executor: make ragas faster and more robust to run by @jjmachan in #390
• fix: Import PrompValue by @shahules786 in #427
• fix: add uuid to analytics by @shahules786 in #428
• feat: added async version for json_load by @jjmachan in #430
• fix: comment error in analytics by @jjmachan in #433
• fix: payload in analytics by @shahules786 in #434
• fix: tests for analytics by @jjmachan in #435
• Docs: tracing and logging by @shahules786 in #436
• feat: global prompt adaptation by @shahules786 in #432
• chore: remove mac artifacts from store by @jjmachan in #438
• fix: vary temperature based on strictness by @shahules786 in #437
• docs: custom llms by @shahules786 in #442
• feat: improving testset generation by @jjmachan in #381
• Fix: context_recall AttributeError by @0ENZO in #453
• Update faithfulness.md by @francesco-defalco in #451
• Fixing typo in faithfulness by @jamesbraza in #450
• fix: context_recall example by @shahules786 in #458
• fix: answer relevancy parsing by @shahules786 in #461
• feat: MVP for the new TestsetGenerator - SimpleEvolution by @jjmachan in #464
• fix: clean up embeddings for ragas and add docs for azure embeddings by @jjmachan in #477
• fix: polishing raise_exception and max_workers by @jjmachan in #486
• Convert prompts in testset generation by @shahules786 in #484
• feat(testset generation): implement new TestsetGenerator by @jjmachan in #487
• doc: add community page by @shahules786 in #488
• Fix ValueError: diag requires an array of at least two dimensions #474 by @herman159 in #475
• doc: modify prompt adaptation by @shahules786 in #490
• docs: theming community by @jjmachan in #491
• fix: CP example by @shahules786 in #489
• feat(testset): ported simple, multi_context and reasoning evolutions by @jjmachan in #496
• fix: cleanup old testset generator by @jjmachan in #500
• feat: added tracking of minimal stuff in testset generation by @jjmachan in #509
• feat: added generate_from_langchain function by @jjmachan in #511
• docs: updated testset generation by @shahules786 in #510
• fix: answer_correctness embedding by @jjmachan in #513
• added docs for customisation by @tinomaxthayil in #514
• fix: added rate_limter to langchain wrapper by @jjmachan in #517
• Update testset_generation.md by @Uzarel in #516
• fix: reshape matrix by @shahules786 in #519
• fix (metrics): changed ground_truths -> ground_truth by @jjmachan in #521
• docs: fix spelling by @shahules786 in #522
• fix: removed old prompt by @shahules786 in #523
• feat(testset): added conditional evolution by @jjmachan in #524
• fix: derive answer from relevant context by @shahules786 in #525
• feat(executor): remove the need to have both sync and async versions of the function by @jjmachan in #527
• feat: seed questions using keyphrases by @shahules786 in #526
• docs: updated contributors by @jjmachan in #532
• feat: configure retries and timeouts with evaluations and testset generators by @jjmachan in #534
• Fix case for few shots of context precision metric by @HuskyDanny in #541
• Updated imports, added dotenv by @aevedis in #539
• feat(executor): run executor in different thread to keep things clean by @jjmachan in #544
• add normalization factor to answer similarity score calculation by @enestydarealmc in #538
• Feat: prompt adaptation for test data generation by @shahules786 in #530
• fix:handle non dict output by @shahules786 in #547
• docs: adaptation for test set generator by @shahules786 in #548
• Fix: ensure quality of adapted prompts by @shahules786 in #546
• fix: raise warning for invalid JSON by @shahules786 in #550
• docs: adding code highlighting by @jjmachan in #552
• fix: wrap langchain embeddings automatically by @jjmachan in #553
• Added documentation page: Using Ragas with Athina for production performance metrics by @shivsak in #558
• fix: proper error message and fixed hf embedding error by @jjmachan in #561
• fix: answer_correctness doesn't reset stuff properly by @jjmachan in #562
• docs: fix issue template by @jjmachan in #565
• docs: fix issue template by @jjmachan in #566
• fix: handle non dict cases by @shahules786 in #564
• fix wrong type anno...

v0.1.0rc1

What's Changed

• add initial embeddings.encode_kwargs by @lucasiscovici in #371
• feat(testset_generator): support error catching in generation process by @yuukidach in #368
• added class for FastEmbed by @tinomaxthayil in #379
• Updated NLI Verdicts to Binary Format by @lucasiscovici in #384
• added docs for ragas embeddings by @tinomaxthayil in #386
• fix context in _answer_correctness.py by @lucasiscovici in #385
• docs: AI copilot implementation by @nickscamara in #294
• fix: faithfulness for evasive/reject answers by @shahules786 in #392
• Added prompt class by @tinomaxthayil in #388
• Context precision with ground truth by @shahules786 in #391
• Added note to answer_relevance by @shahules786 in #399
• Import evaluate function before patching RagasLLM, fixes #301 by @HySoaKa in #377
• Fix: Added ground_truths arg type-hint info in the eval function by @MANISH007700 in #404
• Typo and phrasing correction by @TruscaPetre in #408
• Small improvements to documentation custom LLMs by @TruscaPetre in #410
• feat(metrics): skip embedding process when the corresponding weight is zero by @yuukidach in #409
• added prompt objects to metrics by @tinomaxthayil in #401
• feat: Automatic Prompt adaptation [language] by @shahules786 in #407
• Update _faithfulness.py by @Robs1999 in #416
• fix(metrics): avoid dividing by zero in correctness by @yuukidach in #421
• Feature/doc update ragas score by @prufrock349 in #417
• Docs: Prompts and prompt adaptation by @shahules786 in #419
• fix(metrics): wrong usage of prompt by @yuukidach in #422
• fix: context_utilization name override by @jjmachan in #424
• feat/executor: make ragas faster and more robust to run by @jjmachan in #390
• fix: Import PrompValue by @shahules786 in #427
• fix: add uuid to analytics by @shahules786 in #428
• feat: added async version for json_load by @jjmachan in #430
• fix: comment error in analytics by @jjmachan in #433
• fix: payload in analytics by @shahules786 in #434
• fix: tests for analytics by @jjmachan in #435
• Docs: tracing and logging by @shahules786 in #436
• feat: global prompt adaptation by @shahules786 in #432
• chore: remove mac artifacts from store by @jjmachan in #438
• fix: vary temperature based on strictness by @shahules786 in #437
• docs: custom llms by @shahules786 in #442
• feat: improving testset generation by @jjmachan in #381
• Fix: context_recall AttributeError by @0ENZO in #453
• Update faithfulness.md by @francesco-defalco in #451
• Fixing typo in faithfulness by @jamesbraza in #450
• fix: context_recall example by @shahules786 in #458
• fix: answer relevancy parsing by @shahules786 in #461
• feat: MVP for the new TestsetGenerator - SimpleEvolution by @jjmachan in #464
• fix: clean up embeddings for ragas and add docs for azure embeddings by @jjmachan in #477
• fix: polishing raise_exception and max_workers by @jjmachan in #486
• Convert prompts in testset generation by @shahules786 in #484
• feat(testset generation): implement new TestsetGenerator by @jjmachan in #487
• doc: add community page by @shahules786 in #488
• Fix ValueError: diag requires an array of at least two dimensions #474 by @herman159 in #475
• doc: modify prompt adaptation by @shahules786 in #490
• docs: theming community by @jjmachan in #491
• fix: CP example by @shahules786 in #489
• feat(testset): ported simple, multi_context and reasoning evolutions by @jjmachan in #496
• fix: cleanup old testset generator by @jjmachan in #500
• feat: added tracking of minimal stuff in testset generation by @jjmachan in #509
• feat: added generate_from_langchain function by @jjmachan in #511
• docs: updated testset generation by @shahules786 in #510
• fix: answer_correctness embedding by @jjmachan in #513

New Contributors

• @lucasiscovici made their first contribution in #371
• @nickscamara made their first contribution in #294
• @HySoaKa made their first contribution in #377
• @TruscaPetre made their first contribution in #408
• @Robs1999 made their first contribution in #416
• @prufrock349 made their first contribution in #417
• @0ENZO made their first contribution in #453
• @francesco-defalco made their first contribution in #451
• @jamesbraza made their first contribution in #450
• @herman159 made their first contribution in #475

Full Changelog: v0.0.22...v0.1.0rc1

v0.0.22

What's Changed

• docs: comparing llms using ragas evaluations by @tinomaxthayil in #314
• convert context precision to json format by @shahules786 in #322
• fix: faithfulness.md by @kaiyikang in #328
• fix: structure faithfulness output by @shahules786 in #333
• fix: Column remap fix for Dataset evaluation by @tinomaxthayil in #331
• Fix OpenAIEmbeddings reference error in answer_relevancy by @ajinkyavbhandare in #329
• Fix: minor improvements across metrics by @shahules786 in #337
• fix wrong number by @Yongtae723 in #338
• fix: faithfulness for unrelated answering by @shahules786 in #340
• fix: load response as json by @shahules786 in #341
• docs: remove answer_relevancy from llm-docs by @tinomaxthayil in #343
• fix: bedrock llm error when evaluating rag qa validate_api_key by @arm-diaz in #350
• Added integration with Amazon API Gateway + bug fix in _answer_relevance.py by @brunopistone in #327
• feat: improve quality of answer correctness by @shahules786 in #339
• docs: add zeno visualization integration by @cabreraalex in #359
• fix(metrics): avoid getting none in context_recall by @yuukidach in #355
• fix: improved answer relevancy by @shahules786 in #346
• fix(types): fix pyright type issues with latest pyright version by @jjmachan in #366
• Retry for failed json loading by @shahules786 in #364
• fix: handle edge cases in prompt processing by @shahules786 in #374

New Contributors

• @kaiyikang made their first contribution in #328
• @ajinkyavbhandare made their first contribution in #329
• @brunopistone made their first contribution in #327
• @cabreraalex made their first contribution in #359
• @yuukidach made their first contribution in #355

Full Changelog: v0.0.21...v0.0.22