Add CommunityToolkit.DataIngestion package with LLM chunk processors

[luisquintanilla]

PR: Add CommunityToolkit.DataIngestion — LLM Chunk Enrichment Processors

Description

Adds a new CommunityToolkit.DataIngestion package providing 4 LLM-powered chunk processors for Microsoft.Extensions.DataIngestion pipelines. These processors run at ingestion time to enrich chunks with semantic metadata, dramatically improving downstream retrieval quality without adding latency to the query path.

What Each Processor Does

Processor	What It Does	Why It Helps
EntityExtractionProcessor	Identifies people, organizations, technologies, and versions in each chunk	Enables filtered vector search — combine semantic similarity with structured filters like "mentions Azure Functions"
TopicClassificationProcessor	Labels chunks with primary/secondary topics from your taxonomy	Enables topic-based routing — boost or restrict search to relevant domains
HypotheticalQueryProcessor	Generates questions each chunk answers, stores them as additional embeddings	Bridges the question↔answer gap — users ask questions but docs contain statements; this creates "question-space" vectors that match queries directly
TreeIndexProcessor	Creates document-level and corpus-level summary chunks	Enables multi-resolution retrieval (RAPTOR) — broad questions match summaries, specific questions match leaf chunks

Metadata Written

Processor	Keys	Example Value
EntityExtraction	entities_people, entities_organizations, entities_technologies, entities_versions	"Microsoft, .NET Foundation"
TopicClassification	topic_primary, topic_secondary	"security", "data,web"
HypotheticalQuery	chunk_type, parent_chunk_id, hypothetical_questions	"hypothetical_query", "12345"
TreeIndex	level, chunk_type, parent_id	1, "branch_summary", "root"

Also includes:

• IngestionPipelineBuilder<T> — fluent builder with convenience methods
• DataIngestionServiceCollectionExtensions — DI registration
• MetadataKeys — strongly-typed metadata key constants

Usage

// Register in DI
builder.Services.AddIngestionPipeline<string>()
    .UseEntityExtraction()
    .UseTopicClassification(o => o.Taxonomy = ["web", "data", "security"])
    .UseHypotheticalQueries(o => o.QuestionsPerChunk = 5)
    .UseTreeIndex();

// Build and run
var pipeline = pipelineBuilder.Build(serviceProvider, reader, chunker, writer);
await pipeline.RunAsync(documents);

Design Notes

Relationship to ProcessAsync in DataIngestion:

• This follows the same ProcessAsync pattern as Microsoft.Extensions.DataIngestion.IngestionPipeline.ProcessAsync
• The retrieval-side uses RetrieveAsync (distinct vocabulary for read vs. write paths)

FDG Improvements over original MEDIExtensions source:

• Metadata keys extracted to public const string fields in MetadataKeys class
• Optional ILogger<T> on all processors (was: silent exception swallowing)
• Internal JsonDefaults per package (clean boundary)

Dependencies

All stable, shipped packages:

• Microsoft.Extensions.DataIngestion
• Microsoft.Extensions.AI.Abstractions
• Microsoft.Extensions.DependencyInjection.Abstractions
• Microsoft.Extensions.Logging.Abstractions

Checklist

• Code compiles with 0 errors, 0 warnings
• XML documentation on all public APIs
• Follows .NET Framework Design Guidelines
• Unit tests (next PR or added before merge)
• README/docs update

Related

• Issue: [link to ingestion issue]
• Sibling PR: feat: Add Document Processing packages (MEDI) #3 (PdfPig document processing)
• Upstream: Microsoft.Extensions.DataIngestion (stable, no design review needed)