Feat/confluence sharepoint okf sources - #108
Merged
Merged
Conversation
Add a single free-text --task flag that describes what to enrich and which
sources to read; a focused classifier resolves it into the agent's existing
typed source flags. The typed flags remain the canonical interface — this is
additive sugar that merges into them, and explicit flags always win.
tools/source_classifier.py (new):
* LLM classification of the task into typed source categories (Drive
folder/doc, local path, Confluence, SharePoint, GitHub repo, BigQuery
dataset/table) plus infra config (output_dir / location / entry_group)
and an inferred enrichment mode.
* Deterministic, model-free guardrail layer (unit-testable without Vertex):
G1 anti-hallucination (value must appear in the task), G2 URL/shape
cross-check that overrides the model, G3 shape validation, G4 confidence
floor, G5 dedup.
* Confluence/SharePoint URLs are deferred to the production partition_sources
parsers (page-id / space-key / site lifting); only a non-URL space KEY is
routed straight to --confluence_space.
agent_runner.py:
* --task / --task_dry_run flags; runs the classifier on the light model
(KC_LIGHT_MODEL or --model), echoes the resolved config + sources, then
auto-proceeds (or exits on --task_dry_run).
* Merge: explicit flags win, scalars keep the explicit value, lists union.
* --project now resolves flag -> GOOGLE_CLOUD_PROJECT -> ADC default (still
required overall); --model stays required. project/model are NOT parsed
from --task since they're needed to run the classifier itself.
* --output_dir defaults to a CWD subdir named after the dataset/entry group
when unset; mode is inferred from --task, falling back to the legacy
dataset=>table/else=>doc rule, and prompts interactively when ambiguous.
Backward compatible: when --task is absent the classifier is never invoked and
every source resolves to its explicit flag value exactly as before.
Add a short README section covering the --task flag: describe a run in plain English or keep using the classic flags (they mix freely; explicit flags win), and call out that --project and --model are still required even with --task since they're needed to run the classifier itself.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enrichment agent: natural-language --task input
Add a free-text --task flag: a lightweight classifier reads the description,
extracts the sources (Drive, local MD, Confluence, SharePoint, GitHub, BigQuery)
and mode, and resolves them into the existing typed flags. Deterministic
guardrails (anti-hallucination, URL cross-check, shape/confidence checks) filter
the parse; credentials are never read from the text.
--task and the classic flags mix freely — explicit flags always win, and omitting
--task leaves behavior unchanged. --project and --model are still required (needed
to run the classifier). Includes a README section documenting the new flow.