GML-2167 Release 2.0.1 - #53
Merged
Merged
Conversation
Add evaluator.py with Planned/Reactive/Classic agent support via --agent flag Add run_eval.sh, setup_graph.py, run_setup.sh, run_load.sh for eval workflow Add Apple_SEQ_10 dataset (questions, answers, 4 AAPL PDFs, exported graph) Fix agent_hallucination_check.py: langchain import updated to langchain_core Update docker-compose.yml: volume mounts for regression and test_questions Update requirements.txt: add deepeval==4.0.7, pin click>=8.0.0,<8.4.0
### **PR Type** Tests, Enhancement ___ ### **Description** - Add GraphRAG regression evaluation framework - Score hallucination confidence with reasons - Automate graph setup and evaluation runs - Mount regression datasets into container ___ ### Diagram Walkthrough ```mermaid flowchart LR dataset["Test dataset"] --> setup["Graph setup"] setup -- "creates and ingests" --> graph["GraphRAG graph"] graph -- "queries" --> evaluator["Regression evaluator"] evaluator -- "scores" --> metrics["Correctness and hallucination"] metrics -- "writes" --> results["CSV summary"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>agent_hallucination_check.py</strong><dd><code>Return hallucination confidence with reasoning</code> </dd></td> <td><a href="https://github.com/tigergraph/graphrag/pull/43/files#diff-1f325314ee4113efdf1a9f37c815c30b8b4edb9ae1a460782c58fe14ee61a31a">+94/-22</a> </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>5 files</summary><table> <tr> <td><strong>evaluator.py</strong><dd><code>Add GraphRAG regression evaluation CLI</code> </dd></td> <td><a href="https://github.com/tigergraph/graphrag/pull/43/files#diff-deb3e0fb06b21510a590a7d790001882764b9de49651ba313134f8d465b3c9cd">+748/-0</a> </td> </tr> <tr> <td><strong>setup_graph.py</strong><dd><code>Add regression graph setup automation</code> </dd></td> <td><a href="https://github.com/tigergraph/graphrag/pull/43/files#diff-070a368df3596599a0520014d3d8ca2263f674f693f3f6701f23bc1dfc7b644a">+349/-0</a> </td> </tr> <tr> <td><strong>run_eval.sh</strong><dd><code>Add containerized evaluation runner script</code> </dd></td> <td><a href="https://github.com/tigergraph/graphrag/pull/43/files#diff-090cf2badc203841428904cfb720fbff2a5a436152c08e068fad1c545260825b">+45/-0</a> </td> </tr> <tr> <td><strong>run_load_eval.sh</strong><dd><code>Add exported graph evaluation runner</code> </dd></td> <td><a href="https://github.com/tigergraph/graphrag/pull/43/files#diff-f5d39cff64af1c30c2acdd32ba2991ec718f7da3dba30aaad3f5b3de545341c3">+85/-0</a> </td> </tr> <tr> <td><strong>run_setup.sh</strong><dd><code>Add containerized graph setup runner</code> </dd></td> <td><a href="https://github.com/tigergraph/graphrag/pull/43/files#diff-1b86a187691c0ad76c412986f60ec1a059fd805808b3ec8a093d560580303ba2">+30/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Dependencies</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>requirements.txt</strong><dd><code>Bump pyTigerGraph minimum dependency version</code> </dd></td> <td><a href="https://github.com/tigergraph/graphrag/pull/43/files#diff-e66887509cf346e41e1a9ccb30ddab589260f362899946f6b26602b5e29c547c">+1/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>docker-compose.yml</strong><dd><code>Mount regression test directories into container</code> </dd></td> <td><a href="https://github.com/tigergraph/graphrag/pull/43/files#diff-e45e45baeda1c1e73482975a664062aa56f20c03dd9d64a827aba57775bed0d3">+2/-0</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___
…ogs (#50) Co-authored-by: Prins Kumar <prins.kumar@agivant.com>
… add progres… (#47) * fix: surface rebuild stage in UI during long ECC phases - add progress callbacks to embed, extract, summarize_communities; cache last ECC status to fix unknown time on timeout * fix: clear ECC status cache in finally and preserve cached status on timeout * feat: show document-level chunking progress during rebuild Expose progress_current/total/pct from ECC and render a bar in KGAdmin so the UI shows real progress while chunking documents, without wiping the bar when stream_docs finishes early. --------- Co-authored-by: Prins Kumar <prins.kumar@agivant.com>
- Classic chat reads the answer from providers that return content as blocks (Gemini 3+ and other reasoning models) instead of failing to parse it - Raise the LangChain dependency floors to the 1.x versions this release builds on Refs: GML-2165, GML-2137
- Restore the agent's hallucination check to its 2.0.0 form - Move the graded 0-1 confidence grader into the regression evaluation suite Refs: GML-2088
- Fail fast when the language model is unreachable during community summarization, so a knowledge-graph rebuild no longer hangs. - Leave a clear placeholder for community summaries that could not be generated so they can be regenerated once the model is reachable. - Show per-stage rebuild progress: a document-processing bar, the community-detection level, and a community-summarization bar. - End a rebuild with a warning when some community summaries could not be generated, instead of reporting a plain success. - Bump release to 2.0.1 and update README and CHANGELOG. Refs: GML-2163
- Replace the LICENSE file with the full AGPL-3.0 text. - Update every source-file license header to the AGPL-3.0 notice. - Document the license change in the README and CHANGELOG. Refs: GML-2166
- Replace the verbose GNU notice in source headers with a concise reference to the AGPL-3.0 license and the LICENSE file. - Note the relicense in the README release log and trim the 2.0.1 entry. Refs: GML-2166
- Remove the bind mounts of the regression code and test_questions from the default compose, so a normal deployment carries no test artifacts. - The regression run scripts now copy those files into the running container before a run and copy eval results back out. Refs: GML-2088
- Restore the AGPL warranty disclaimer and license-copy notice in source file headers, with the opening reworded to omit the "free software" phrasing. Refs: GML-2166
- Migrate the graphrag app from FastAPI on_event handlers to a lifespan handler (the ECC app already used lifespan). - Import the sunset langchain-experimental graph transformer lazily, so the warning only fires if that extractor is actually used. - Drop the deprecated useCert argument when cloning TigerGraph connections. Refs: GML-2137
- Mark the `graphrag` extractor mode as deprecated in the README; it relies on the sunset langchain-experimental package and is superseded by the default `llm` extractor. Code retained for now. Refs: GML-2137
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.