Skip to content

Feat/add knowleage base and agent#12

Draft
anuragk16 wants to merge 21 commits intoColoredCow:mainfrom
anuragk16:feat/add-knowleage-base-and-agent
Draft

Feat/add knowleage base and agent#12
anuragk16 wants to merge 21 commits intoColoredCow:mainfrom
anuragk16:feat/add-knowleage-base-and-agent

Conversation

@anuragk16
Copy link

@anuragk16 anuragk16 commented Mar 2, 2026

target issue : Knowledge Base for Agentic Systems #11

Testing in IAGES:

  • After adding the knowledge base and the agent i gave this prompt :
image
  • Then this :
image
  • So it created me these :
image
  • So for testing if the knowledge base is used by the agents or not, I give it a new requirement for the similar task. (
    02-architecture/adr-002-vcp-annual-revenue-lock.md | #597 — Revenue locked after provisional payment )

input

image

output

image image

Conclusion

  • It is using the knowledge base , also as the third primary source of truth (first is claude.md , and second is memory.md which is agent specific )

NOTE : Found some issue in the prompt and the business flow index file which is not creating a new tree sturcture instead it is extenting the indix file in the business flow for the all uses ( issue is file get to long in very short time and Ai will be reading the whole file which will make the token consuption more)

  • fixed this by adding a more generic prompt at the business flow index file.

Test 2

  • Without calling any specific agent , validating if the Claude is using the knowledge base or not for the information.
image

Conclusion : Yes , it is seeing the knowledge base as the first source then going to explore the codebase.

@anuragk16 anuragk16 marked this pull request as draft March 2, 2026 11:19
@anuragk16 anuragk16 marked this pull request as ready for review March 2, 2026 13:45
@anuragk16 anuragk16 marked this pull request as draft March 2, 2026 13:45
@anuragk16 anuragk16 marked this pull request as draft March 2, 2026 13:45
@anuragk16
Copy link
Author

anuragk16 commented Mar 2, 2026

Knowledge Base Adoption Testing Template

1) Project Info

  • Project Name: IAGES
  • Repository URL: https://github.com/coloredcow-admin/iages-backend
  • Branch / Commit: N/A (manual validation run)
  • Test Owner: @anuragk16
  • Test Date: 2026-03-03
  • Knowledge Base Mode: Using KB

2) Test Scope (Baseline Only)

  • Scope Type: Baseline
  • Goal: Confirm knowledge-base integration is safe to adopt and does not break normal agent workflows.
  • Out of Scope: Deep workflow-specific validations (to be refined later).

3) Baseline Gate Checklist (Mandatory)

Mark each item: PASS / FAIL / BLOCKED / NOT APPLICABLE

ID Check Status Evidence Notes
B1 Agent workflows run without hard failure PASS IAGES run notes Knowledge base + agent setup completed and produced outputs.
B2 User prompt instructions still take precedence over KB context PASS IAGES run notes Validation responses were KB-aware and aligned to prompt context.
B3 No unintended file changes outside expected scope BLOCKED IAGES run notes Not explicitly validated in this baseline run.
B4 If KB is missing/disabled, agents still run in fallback mode BLOCKED N/A Negative-path fallback test not executed yet.
B5 CI/check pipeline status is known and recorded PASS PR checks Status is visible/recorded; baseline interpretation pending full test matrix.

4) IAGES Testing Snapshot

  • Initial setup prompts were executed and KB artifacts were created.
  • Validation prompt used a similar requirement context:
    • 02-architecture/adr-002-vcp-annual-revenue-lock.md
    • #597 — Revenue locked after provisional payment
  • Observed outcome from the run:
    • Agent responses referenced and used KB context.
    • KB acted as an additional source of truth alongside CLAUDE.md and agent memory.

Evidence thread: Testing in IAGES

5) New Findings Update (Latest)

  • Found an issue in the prompt + business flow index behavior:
    • It was extending 01-business-flows/00-index.md instead of using tree-structured separate flow files.
    • This can make the index grow too fast and increase token usage because agents read larger files repeatedly.
  • Applied fix in IAGES test setup:
    • Updated with a more generic prompt direction in the business-flow index context to enforce tree structure behavior.

Test 2 (No specific agent invocation)

  • Test objective: Validate whether Claude uses the knowledge base even without explicitly calling a specific agent.
  • Result: PASS
  • Conclusion:
    • Claude checked KB first for domain/business-flow context, then explored codebase for implementation details.
    • This confirms KB is being treated as an early source of context in normal prompting.
  • Evidence: Screenshot shared in this PR thread (latest testing update).

6) Result Summary

  • Overall Result: GO WITH RISKS
  • Blocking Issues: B3 and B4 are still BLOCKED (not yet validated in this baseline run).
  • Known Risks Accepted for This Stage: Proceeding with baseline confidence only; deeper sync/fallback coverage pending.
  • Next Refinement Target (single next step): Run explicit B3 and B4 checks and then finalize GO/NO-GO.

7) Decision Rule (Simple)

Use this simple rule until process maturity increases:

  • GO only if all mandatory baseline checks (B1 to B5) are PASS or NOT APPLICABLE
  • NO-GO if any mandatory baseline check is FAIL
  • GO WITH RISKS if no FAIL, but one or more mandatory checks are BLOCKED

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant