feat: v0.21.0 Connector Expansion + Multimodal + Observability

.claude/skills/feature/SKILL.md

@@ -92,9 +92,37 @@ See `/test` for detailed testing patterns. Key reminders:
 
 Create `examples/NN_feature_name.py` (use next number from Live Project State above).
 
-## 7. Write Documentation
+## 7. Write Documentation (MANDATORY, DO NOT SKIP)
 
-See `/docs` for detailed documentation patterns.
+Every feature MUST have documentation updated before the feature is considered complete.
+This is not optional. Failing to update docs is the same as shipping broken code.
+
+### 7a. Update existing module docs
+For EACH source file you modified or created, find the corresponding doc in `docs/modules/`
+and update it with the new feature. Add code examples, API signatures, and "Since: vX.Y.Z".
+
+### 7b. Update index docs
+- `docs/QUICKSTART.md` — add a "What's New" entry if user-facing
+- `docs/llms.txt` — update module descriptions, counts, and links
+- `docs/index.md` — update feature table and counts if applicable
+- `mkdocs.yml` — update nav labels (e.g., tool counts) if changed
+
+### 7c. Update landing page
+- `landing/index.html` — update stats bar counts (tests, examples, models, etc.)
+- Update comparison tables if the feature adds competitive advantages
+
+### 7d. Update notebook
+- `notebooks/getting_started.ipynb` — add a new step if the feature is user-facing
+
+### 7e. Verify docs build
+```bash
+cp CHANGELOG.md docs/CHANGELOG.md && mkdocs build
+```
+
+If you created new source modules, also update:
+- `docs/llms-full.txt` (regenerate with the build script)
+- `docs/ARCHITECTURE.md` if it changes the system architecture
+- `docs/MIGRATION.md` if it replaces or improves on a competitor feature
 
 ## 8. Run Full Test Suite
 

.github/workflows/docs.yml

@@ -41,6 +41,8 @@ jobs:
           cp landing/index.html site/index.html
           cp landing/eval-report-preview.html site/eval-report-preview.html 2>/dev/null || true
           cp landing/trace-preview.html site/trace-preview.html 2>/dev/null || true
+          mkdir -p site/simulations
+          cp landing/simulations/index.html site/simulations/index.html 2>/dev/null || true
 
       - name: Install selectools (for builder assembly)
         run: pip install -e .

CLAUDE.md

@@ -266,16 +266,21 @@ When implementing a new feature, ALWAYS complete ALL of these steps:
 - [ ] Run full suite: `pytest tests/ -x -q` — ALL must pass
 - [ ] Verify no tests were broken by the change
 
-### 4. Documentation Updates (ALL of these, every time)
+### 4. Documentation Updates (MANDATORY for every feature, DO NOT SKIP)
 
-- [ ] **Module doc**: Create or update `docs/modules/<MODULE>.md`
+Documentation is NOT optional. A feature without updated docs is an incomplete feature.
+
+- [ ] **Module docs**: For EACH source file modified, update the corresponding `docs/modules/*.md` with new sections, code examples, and "Since: vX.Y.Z"
 - [ ] **Architecture doc**: Update `docs/ARCHITECTURE.md` if it adds a new component
-- [ ] **Quickstart**: Update `docs/QUICKSTART.md` if it's user-facing
-- [ ] **docs/README.md**: Update the documentation index
-- [ ] **docs/index.md**: Update the landing page feature table and model counts
-- [ ] **Notebook**: Add section to `notebooks/getting_started.ipynb`
+- [ ] **Quickstart**: Update `docs/QUICKSTART.md` if it's user-facing (add "What's New" entry)
+- [ ] **docs/index.md**: Update the feature table and counts
+- [ ] **llms.txt**: Update module descriptions, counts, and links in `docs/llms.txt`
+- [ ] **Landing page**: Update `landing/index.html` stats bar (tests, examples, models, tools) and comparison tables if competitive
+- [ ] **mkdocs.yml**: Update nav labels (e.g., tool counts, store counts) if changed
+- [ ] **Notebook**: Add step to `notebooks/getting_started.ipynb`
 - [ ] **Example script**: Add `examples/NN_<feature>.py`
-- [ ] Verify all internal links work: `cp CHANGELOG.md docs/CHANGELOG.md && mkdocs build`
+- [ ] **Gallery**: Regenerate `landing/examples.json` if new examples added
+- [ ] **Verify build**: `cp CHANGELOG.md docs/CHANGELOG.md && mkdocs build`
 
 ### 5. Release Artifacts (for each release)
 

docs/QUICKSTART.md

@@ -545,6 +545,15 @@ print(f"Steps taken: {result.steps}")
 
 ---
 
+## What's New in v0.21.0
+
+- **3 new vector stores**: FAISS, Qdrant, pgvector -- see [RAG Pipeline](modules/RAG.md#faiss-v0210)
+- **4 new document loaders**: `from_csv`, `from_json`, `from_html`, `from_url` -- see [RAG Pipeline](modules/RAG.md#loading-from-csv-v0210)
+- **9 new toolbox tools**: code execution, web search, GitHub, database queries -- see [Toolbox](modules/TOOLBOX.md#code-tools-2--v0210)
+- **Azure OpenAI provider**: use Azure-hosted OpenAI models -- see [Providers](modules/PROVIDERS.md#azure-openai-provider-v0210)
+- **OTel + Langfuse observers**: ship traces to OpenTelemetry or Langfuse -- see [Providers](modules/PROVIDERS.md#observability-integrations-v0210)
+- **Multimodal messages**: `ContentPart`, `image_message()`, `text_content()` -- see [Streaming](modules/STREAMING.md#multimodal-messages-v0210)
+
 ## What's Next?
 
 You now know the core API. Here is where to go from here:
@@ -571,7 +580,12 @@ You now know the core API. Here is where to go from here:
 | Add audit logging | [Audit Guide](modules/AUDIT.md) |
 | Screen tool outputs for injection | [Security Guide](modules/SECURITY.md) |
 | Enable coherence checking | [Security Guide — Coherence](modules/SECURITY.md#coherence-checking) |
-| Use 24 pre-built tools | [Toolbox Guide](modules/TOOLBOX.md) |
+| Use 33 pre-built tools | [Toolbox Guide](modules/TOOLBOX.md) |
+| Use Azure OpenAI | [Providers Guide -- Azure](modules/PROVIDERS.md#azure-openai-provider-v0210) |
+| Send traces to OTel / Langfuse | [Providers Guide -- Observability](modules/PROVIDERS.md#observability-integrations-v0210) |
+| Load CSV, JSON, HTML, or URLs | [RAG Guide -- Loaders](modules/RAG.md#loading-from-csv-v0210) |
+| Use FAISS, Qdrant, or pgvector | [RAG Guide -- Vector Stores](modules/RAG.md#faiss-v0210) |
+| Send images to vision models | [Streaming Guide -- Multimodal](modules/STREAMING.md#multimodal-messages-v0210) |
 | Handle errors gracefully | [Exceptions Guide](modules/EXCEPTIONS.md) |
 | Look up model pricing at runtime | [Models Guide — Pricing API](modules/MODELS.md#programmatic-pricing-api) |
 | Use structured output helpers | [Agent Guide — Structured Helpers](modules/AGENT.md#standalone-helpers) |

docs/llms.txt

@@ -33,7 +33,7 @@ result = agent.run("Find our refund policy")
 
 - [Agent](https://johnnichev.github.io/selectools/modules/AGENT/): Agent class, AgentConfig, AgentResult — the central orchestrator
 - [Tools](https://johnnichev.github.io/selectools/modules/TOOLS/): @tool decorator, Tool class, ToolLoader, ToolRegistry
-- [Toolbox](https://johnnichev.github.io/selectools/modules/TOOLBOX/): 24 pre-built tools (file, web, data, datetime, text)
+- [Toolbox](https://johnnichev.github.io/selectools/modules/TOOLBOX/): 33 pre-built tools (file, web, data, datetime, text, code, search, github, db)
 - [Memory](https://johnnichev.github.io/selectools/modules/MEMORY/): ConversationMemory with sliding window and summarize-on-trim
 - [Streaming](https://johnnichev.github.io/selectools/modules/STREAMING/): Token-level astream() with tool call support
 
@@ -42,11 +42,11 @@ result = agent.run("Find our refund policy")
 - [RAG Pipeline](https://johnnichev.github.io/selectools/modules/RAG/): Document loading, chunking, retrieval
 - [Hybrid Search](https://johnnichev.github.io/selectools/modules/HYBRID_SEARCH/): BM25 + vector fusion with RRF
 - [Embeddings](https://johnnichev.github.io/selectools/modules/EMBEDDINGS/): OpenAI, Anthropic, Gemini, Cohere embedding providers
-- [Vector Stores](https://johnnichev.github.io/selectools/modules/VECTOR_STORES/): Memory, SQLite, Chroma, Pinecone backends
+- [Vector Stores](https://johnnichev.github.io/selectools/modules/VECTOR_STORES/): Memory, SQLite, Chroma, Pinecone, FAISS, Qdrant, pgvector backends
 
 ## Providers
 
-- [Providers](https://johnnichev.github.io/selectools/modules/PROVIDERS/): OpenAI, Anthropic, Gemini, Ollama + FallbackProvider
+- [Providers](https://johnnichev.github.io/selectools/modules/PROVIDERS/): OpenAI, Azure OpenAI, Anthropic, Gemini, Ollama + FallbackProvider
 - [Models](https://johnnichev.github.io/selectools/modules/MODELS/): 152 models with built-in pricing data
 
 ## Multi-Agent
@@ -76,6 +76,9 @@ result = agent.run("Find our refund policy")
 - [Reasoning Strategies](https://johnnichev.github.io/selectools/modules/REASONING_STRATEGIES/): ReAct, CoT, Plan-Then-Act
 - [Builder Docs](https://johnnichev.github.io/selectools/modules/builder/): Visual builder reference
 - [Templates](https://johnnichev.github.io/selectools/modules/TEMPLATES/): YAML agent configuration
+- [OTel Observer](https://johnnichev.github.io/selectools/modules/PROVIDERS/#observability-integrations-v0210): OpenTelemetry agent trace export
+- [Langfuse Observer](https://johnnichev.github.io/selectools/modules/PROVIDERS/#langfuseobserver): Langfuse agent trace export
+- [Multimodal Messages](https://johnnichev.github.io/selectools/modules/STREAMING/#multimodal-messages-v0210): ContentPart, image_message(), text_content()
 - [Stability Markers](https://johnnichev.github.io/selectools/modules/STABILITY/): @stable, @beta, @deprecated
 - [Changelog](https://johnnichev.github.io/selectools/CHANGELOG/): Release history
 - [Examples Gallery](https://johnnichev.github.io/selectools/examples/): 76 runnable scripts with categories

docs/modules/PROVIDERS.md

@@ -43,7 +43,7 @@ print(result.content)
 ---
 
 **Directory:** `src/selectools/providers/`
-**Files:** `base.py`, `openai_provider.py`, `anthropic_provider.py`, `gemini_provider.py`, `ollama_provider.py`, `fallback.py`
+**Files:** `base.py`, `openai_provider.py`, `azure_openai_provider.py`, `anthropic_provider.py`, `gemini_provider.py`, `ollama_provider.py`, `fallback.py`
 
 ## Table of Contents
 
@@ -164,7 +164,9 @@ class Provider(Protocol):
 All providers support namespace imports from the `selectools.providers` package:
 
 ```python
-from selectools.providers import OpenAIProvider, AnthropicProvider, GeminiProvider, OllamaProvider
+from selectools.providers import (
+    OpenAIProvider, AzureOpenAIProvider, AnthropicProvider, GeminiProvider, OllamaProvider
+)
 ```
 
 ### OpenAI Provider
@@ -258,6 +260,51 @@ provider = OllamaProvider(
 > response parsing, and streaming logic. Only pricing, error messages, and token
 > parameter naming differ between them.
 
+### Azure OpenAI Provider (v0.21.0)
+
+**Stability:** <span class="badge-beta">beta</span>
+
+```python
+from selectools.providers import AzureOpenAIProvider
+
+provider = AzureOpenAIProvider(
+    azure_endpoint="https://my-resource.openai.azure.com",
+    api_key="...",                    # Or set AZURE_OPENAI_API_KEY env var
+    azure_deployment="gpt-4o",        # Or set AZURE_OPENAI_DEPLOYMENT env var
+    api_version="2024-10-21",         # Azure API version (default)
+)
+
+# Features:
+# - Inherits all OpenAI capabilities (streaming, async, vision, tool calling)
+# - Azure Active Directory (AAD) token authentication
+# - Uses the openai SDK's built-in Azure support (no extra deps)
+```
+
+**API:** Azure OpenAI Service
+
+**Environment Variables:**
+
+| Variable | Description |
+|---|---|
+| `AZURE_OPENAI_ENDPOINT` | Azure resource endpoint URL |
+| `AZURE_OPENAI_API_KEY` | Azure API key (can omit if using AAD token) |
+| `AZURE_OPENAI_DEPLOYMENT` | Default deployment name |
+
+**AAD Token Authentication:**
+
+```python
+# Use Azure Active Directory instead of an API key
+provider = AzureOpenAIProvider(
+    azure_endpoint="https://my-resource.openai.azure.com",
+    azure_ad_token="eyJ...",          # AAD token
+    azure_deployment="gpt-4o",
+)
+```
+
+> **Implementation note**: `AzureOpenAIProvider` extends `OpenAIProvider`, overriding
+> only the client initialization to use `AzureOpenAI` / `AsyncAzureOpenAI` from the
+> OpenAI SDK. All complete/stream/acomplete/astream behaviour is inherited.
+
 ### Local Provider (Testing)
 
 ```python
@@ -833,6 +880,65 @@ The provider falls through to the next on:
 
 ---
 
+## Observability Integrations (v0.21.0)
+
+Two new observer implementations let you ship agent traces to external observability platforms.
+
+### OTelObserver
+
+**Stability:** <span class="badge-beta">beta</span>
+
+Sends agent traces to OpenTelemetry following the GenAI semantic conventions.
+
+```bash
+pip install opentelemetry-api
+```
+
+```python
+from selectools.observe.otel import OTelObserver
+
+agent = Agent(
+    tools=[...],
+    provider=provider,
+    config=AgentConfig(observers=[OTelObserver(tracer_name="my-app")]),
+)
+```
+
+Creates spans for `agent.run`, `gen_ai.chat` (LLM calls), and `tool.execute` (tool executions) with standard GenAI attributes like `gen_ai.usage.input_tokens` and `gen_ai.request.model`.
+
+### LangfuseObserver
+
+**Stability:** <span class="badge-beta">beta</span>
+
+Sends agent traces to Langfuse for observability, cost tracking, and debugging.
+
+```bash
+pip install langfuse
+```
+
+```python
+from selectools.observe.langfuse import LangfuseObserver
+
+observer = LangfuseObserver(
+    public_key="pk-...",     # Or set LANGFUSE_PUBLIC_KEY env var
+    secret_key="sk-...",     # Or set LANGFUSE_SECRET_KEY env var
+    host="https://...",      # Or set LANGFUSE_HOST env var (for self-hosted)
+)
+
+agent = Agent(
+    tools=[...],
+    provider=provider,
+    config=AgentConfig(observers=[observer]),
+)
+
+# On application shutdown
+observer.shutdown()
+```
+
+Both observers implement the standard `AgentObserver` protocol and can be composed with other observers (e.g. `LoggingObserver`, `AuditLogger`).
+
+---
+
 ## Further Reading
 
 - [Agent Module](AGENT.md) - How agents use providers