This directory contains testing and evaluation artifacts for the cultural heritage guide system.
Testing is organized by test type, with each type having a consistent structure:
testing/
├── diary-testing.md # Testing development diary
├── thesis-notes-testing.md # Case studies and methodologies
│
├── qualitative/ # Qualitative testing (quality assessment)
│ ├── scripts/ # Test scripts
│ ├── data/ # Raw test data (organized by date)
│ └── reports/ # Analysis reports (organized by date)
│
├── automated/ # Automated test suites
│ ├── scripts/ # pytest, integration tests
│ ├── data/ # Test fixtures
│ └── reports/ # Test results
│
├── performance/ # Performance testing
│ ├── scripts/ # Load tests, benchmarks
│ ├── data/ # Performance data
│ └── reports/ # Performance analysis
│
├── user-studies/ # User evaluation studies (Track 1)
│ ├── protocols/ # Study design, consent forms
│ ├── data/ # Participant data (anonymized)
│ └── reports/ # Analysis reports
│
└── evaluation/ # Evaluation metrics
├── scripts/ # Metric calculation scripts
├── data/ # Evaluation data
└── reports/ # Evaluation reports
- Type-based separation: Each test type has its own directory
- Consistent structure: scripts/, data/, reports/ in each type
- Date-based organization: data/ and reports/ use YYYY-MM-DD folders
- Scalability: Easy to add new test types as project grows
- diary-testing.md — Testing development diary
- thesis-notes-testing.md — Testing case studies and evaluation methodologies
Based on case studies from the project, the following testing philosophy has been established:
- Don't assume APIs behave as documented
- Test with real data, not just mocks
- Multiple test scenarios (easy, medium, hard, edge cases)
- Structured test suite with expected outcomes
- Document actual behavior vs expected behavior
- Reproducible test cases
- Mocks can encode same incorrect assumptions as production code
- Integration tests catch issues unit tests miss
- External API quirks only visible with real calls
For testing and evaluation work, use the tester agent:
- Configuration:
../.claude/agents/tester.md - Expertise: Integration testing, systematic evaluation, empirical validation
- Responsibilities: Testing all components, bug discovery, evaluation methodologies, research data collection
The tester agent:
- Writes to
diary-testing.md - Has read access to all directories
- Focuses on comprehensive testing and evaluation
- Reports bugs to other agents via user
The tester agent serves two purposes:
- Integration testing against real APIs
- Bug discovery and systematic debugging
- Performance benchmarking
- Security testing
- Accessibility testing (frontend)
- User study design and execution
- Quality metrics collection
- Evaluation methodologies for thesis
- Data collection for meta-research (Track 2)
| Area | Coverage |
|---|---|
| Tool execution | ✅ All 15 tools tested |
| Source types | ✅ Wikipedia, SNL, OGC, ArcGIS |
| Query types | ✅ Factual, spatial, text search |
| Error handling | ✅ Missing data, invalid input |
| Streaming | ✅ SSE events, token streaming |
| Frontend | ✅ UI rendering, interactions |
Five integration bugs discovered only in production deployment:
- Registry attribute error
- Missing get_tool() method
- Handler argument unpacking
- Pydantic JSON serialization
- API key trailing newline
Key insight: Each component worked in isolation; bugs existed only at integration boundaries.
Comprehensive 15-question test suite discovered:
- Duplicate source titles bug
- Markdown formatting inconsistencies
Key insight: Systematic testing discovered bugs that months of manual usage might not have surfaced.
cd qualitative/scripts
# Manual testing with full analysis
./manual_quality_test.sh
# Automated test suite (11 scenarios)
python test_chat_quality.pycd ../backend/MCP_server_for_apis
pytest # All tests
pytest --cov=src # With coverage
pytest -v -k "wikipedia" # Specific testscd ../frontend/MCP-kulturarv-interface
npm run build # Build verification
npm run test # Run tests (if configured)Integration tests should:
- Call real external APIs
- Test with multiple cities/queries/coordinates
- Verify results are actually from expected locations
- Document API behavior differences
See qualitative/scripts/ for examples of integration testing against real backend.
For user-centered evaluation (Track 1):
- Usability testing (task-based)
- Semi-structured interviews
- UX surveys
- Qualitative thematic analysis
For meta-research evaluation (Track 2):
- Development diary analysis
- Git history analysis
- Case study documentation
- Quantitative metrics (time, iterations, bugs)
For detailed case studies and testing patterns, see:
- thesis-notes-testing.md — Testing case studies
- ../CLAUDE.md — System architecture and known issues
This testing directory is maintained as a separate git repository for thesis citations and reproducibility.
Repository: https://github.com/andreasklae/thesis-tests
Purpose:
- Cite testing artifacts in thesis
- Provide empirical data for research
- Document testing methodology
- Preserve test results for analysis
What's included:
- ✅ Test scripts and methodologies
- ✅ Test data (JSON responses, logs)
- ✅ Analysis reports and findings
- ❌ API keys and credentials (.gitignore)
- ❌ Virtual environments (.gitignore)
Note: Tests require API keys and running backend infrastructure. See README-REPO.md for details on reproducing tests.