Skip to content

docs: add community development pointers and generalize local testing guide #150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
neuromechanist merged 1 commit into from
Feb 4, 2026
Merged

docs: add community development pointers and generalize local testing guide #150

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
272 changes: 48 additions & 224 deletions .context/local-testing-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,257 +1,81 @@
# Local Testing Guide for EEGLAB Assistant
# Local Testing Guide for Community Assistants

## Quick Test (Verify Configuration)
Quick reference for testing any community assistant locally. For the full guide, see https://docs.osc.earth/osa/registry/local-testing/

```bash
cd /Users/yahya/Documents/git/osa-phase1
## Quick Validation

# Run quick verification
uv run python test_eeglab_interactive.py
```bash
# Validate config loads
uv run pytest tests/test_core/ -k "community" -v

# Or programmatically
uv run python -c "
from pathlib import Path
from src.core.config.community import CommunityConfig
config = CommunityConfig.from_yaml(Path('src/assistants/COMMUNITY_ID/config.yaml'))
print(f'Loaded: {config.name} with {len(config.documentation)} docs')
"
```

## Full Backend Testing

### 1. Set Environment Variables
## Environment Variables

```bash
# Required: OpenRouter API key for LLM
export OPENROUTER_API_KEY="your-key-here"

# Optional: API keys for admin functions (sync)
# Optional: for sync operations
export API_KEYS="test-key-123"

# Optional: Specific EEGLAB key (if community has BYOK)
# export OPENROUTER_API_KEY_EEGLAB="eeglab-specific-key"
# Optional: community-specific key
# export OPENROUTER_API_KEY_COMMUNITY="key"
```

### 2. Start Backend Server
## Server Testing

```bash
cd /Users/yahya/Documents/git/osa-phase1

# Start development server
# Start dev server
uv run uvicorn src.api.main:app --reload --port 38528
```

Server will be available at: `http://localhost:38528`

### 3. Test Endpoints

#### A. List All Communities

```bash
# List communities (verify yours appears)
curl http://localhost:38528/communities | jq
```

**Expected response:**
```json
{
"communities": [
{
"id": "eeglab",
"name": "EEGLAB",
"description": "EEG signal processing and analysis toolbox",
"status": "available"
},
{
"id": "hed",
"name": "HED (Hierarchical Event Descriptors)",
...
}
]
}
```

#### B. Get EEGLAB Community Info

```bash
curl http://localhost:38528/communities/eeglab | jq
```

**Expected response:**
```json
{
"id": "eeglab",
"name": "EEGLAB",
"description": "EEG signal processing and analysis toolbox",
"status": "available",
"documentation_count": 26,
"github_repos": 6,
"has_sync_config": true
}
```

#### C. Ask a Question (Simple)

```bash
curl -X POST http://localhost:38528/eeglab/ask \
-H "Content-Type: application/json" \
-d '{
"question": "What is EEGLAB?",
"api_key": "your-openrouter-key"
}' | jq
```

**Expected response:**
```json
{
"answer": "EEGLAB is an interactive MATLAB toolbox...",
"sources": [
{
"title": "EEGLAB quickstart",
"url": "https://sccn.github.io/..."
}
]
}
```

#### D. Ask About ICA

```bash
curl -X POST http://localhost:38528/eeglab/ask \
# Ask a question
curl -X POST http://localhost:38528/COMMUNITY_ID/ask \
-H "Content-Type: application/json" \
-d '{
"question": "How do I run ICA in EEGLAB?",
"api_key": "your-openrouter-key"
}' | jq
-d '{"question": "What is this tool?", "api_key": "your-key"}' | jq
```

**Should mention:** ICA decomposition, ICLabel, artifact removal

#### E. Test Chat Endpoint
## CLI Testing (No Server Needed)

```bash
curl -X POST http://localhost:38528/eeglab/chat \
-H "Content-Type: application/json" \
-d '{
"messages": [
{"role": "user", "content": "What preprocessing steps should I do?"}
],
"api_key": "your-openrouter-key"
}' | jq
```

**Should mention:** Filtering, re-referencing, ICA, artifact removal
# Interactive chat
uv run osa chat --community COMMUNITY_ID --standalone

### 4. Test Documentation Retrieval

The assistant should automatically retrieve docs. Test by asking specific questions:

```bash
# Should trigger retrieve_eeglab_docs tool
curl -X POST http://localhost:38528/eeglab/ask \
-H "Content-Type: application/json" \
-d '{
"question": "How do I filter my EEG data in EEGLAB?",
"api_key": "your-openrouter-key",
"stream": false
}' | jq '.tool_calls'
# Single question
uv run osa ask --community COMMUNITY_ID "What is this tool?" --standalone
```

**Expected:** Should call `retrieve_eeglab_docs` with filter-related docs

### 5. Test via CLI (Easier!)
## Knowledge Sync

```bash
cd /Users/yahya/Documents/git/osa-phase1

# Set API key
export OPENROUTER_API_KEY="your-key-here"

# Start interactive chat
uv run osa chat --community eeglab --standalone

# Or ask single question
uv run osa ask --community eeglab "What is EEGLAB?" --standalone
uv run osa sync init --community COMMUNITY_ID
uv run osa sync github --community COMMUNITY_ID --full
uv run osa sync papers --community COMMUNITY_ID --citations
```

**CLI is easier for testing because:**
- Handles API key automatically
- Shows formatted output
- Interactive mode for multi-turn conversations
## Test Checklist

## Test Questions for EEGLAB

Good test questions to verify configuration:

1. **Basic Info:**
- "What is EEGLAB?"
- "What can EEGLAB do?"

2. **Preprocessing:**
- "What preprocessing steps should I do?"
- "How do I filter EEG data?"
- "How do I re-reference my data?"

3. **ICA:**
- "How do I run ICA in EEGLAB?"
- "What is ICLabel?"
- "How do I remove artifacts with ICA?"

4. **Plugins:**
- "What is clean_rawdata?"
- "How do I use ASR?"
- "What is the PREP pipeline?"

5. **Integration:**
- "How do I use EEGLAB with BIDS?"
- "Can I use EEGLAB with Python?"

6. **Knowledge Base (requires sync):**
- "What are the latest issues in the eeglab repo?"
- "Show me recent PRs in ICLabel"
- "Papers about EEGLAB ICA"
- [ ] Config validates without errors
- [ ] Community appears in `/communities`
- [ ] `/ask` endpoint returns relevant answers
- [ ] `/chat` endpoint works for multi-turn
- [ ] Preloaded docs are in context
- [ ] On-demand docs retrieved when relevant
- [ ] Documentation URLs in responses are valid
- [ ] CLI standalone mode works
- [ ] Knowledge sync completes (if configured)
- [ ] Assistant does not hallucinate PR/issue numbers

## Troubleshooting

### Server won't start

```bash
# Check if port is already in use
lsof -i :38528

# Use different port
uv run uvicorn src.api.main:app --reload --port 38529
```

### "Assistant not found" error

```bash
# Verify EEGLAB is registered
uv run python -c "from src.assistants import discover_assistants, registry; discover_assistants(); print('eeglab' in registry)"
```

### Documentation not retrieved

- Check that `retrieve_eeglab_docs` tool is available
- Check network access to sccn.github.io
- Check tool calls in response

### Knowledge base empty

- Knowledge base requires `API_KEYS` env var for sync
- Run sync locally:
```bash
export API_KEYS="test-key"
uv run osa sync init --community eeglab
uv run osa sync github --community eeglab --full
```

## Expected Behavior

**What works without knowledge sync:**
- ✓ Assistant creation
- ✓ System prompt
- ✓ Documentation retrieval (fetches from URLs)
- ✓ Answering questions about EEGLAB
- ✓ Providing guidance on workflows

**What needs knowledge sync:**
- ✗ Searching GitHub issues/PRs
- ✗ Listing recent activity
- ✗ Searching papers
- ✗ Citation counts

## Next: Epic Branch Workflow

See `epic-branch-workflow.md` for multi-phase development process.
- **Server won't start**: Check port with `lsof -i :38528`
- **Assistant not found**: Check discovery with `uv run python -c "from src.assistants import discover_assistants, registry; discover_assistants(); print([a.id for a in registry.list_available()])"`
- **Docs not retrieved**: Test source_url with `curl -I <url>`
- **Knowledge empty**: Run `uv run osa sync init --community COMMUNITY_ID` first
13 changes: 10 additions & 3 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,9 +146,16 @@ src/
- **.context/api_key_authorization_design.md** - API key auth and CORS
- **.context/security-architecture.md** - Security patterns

### Community Configuration
- **.context/yaml_registry.md** - YAML-based community config
- **.context/community_onboarding_review.md** - Onboarding guidelines
### Community Development (Adding/Modifying Communities)
- **Full docs site**: https://docs.osc.earth/osa/registry/ (canonical reference)
- [Adding a Community](https://docs.osc.earth/osa/registry/quick-start/) - Step-by-step guide
- [Local Testing](https://docs.osc.earth/osa/registry/local-testing/) - Testing a new community end-to-end
- [Schema Reference](https://docs.osc.earth/osa/registry/schema-reference/) - Full YAML config schema
- [Extensions](https://docs.osc.earth/osa/registry/extensions/) - Python plugins and MCP servers
- **.context/yaml_registry.md** - YAML-based community config (internal notes)
- **.context/community_onboarding_review.md** - Onboarding gap analysis
- **.context/local-testing-guide.md** - Quick local testing reference
- **Existing configs to reference**: `src/assistants/hed/config.yaml`, `src/assistants/eeglab/config.yaml`

### Tool System
- **.context/tool-system-guide.md** - How tools work and are registered
Expand Down