Version 1.4 in testing

[pendingintent]

I am currently testing the next release of the SoA Workbench - release 1.4. Listed below is a summary of changes included in the latest release.

What

This release expands the SoA Workbench from a visit/activity scheduling tool into a full USDM study definition platform. v1.4 ships an MCP server for agent-driven workflows, a complete Define-JSON generator, CRF specialization assignment, and twelve new study entity domains (organizations, roles, persons, estimands, indications, study interventions, titles, identifiers, amendments extensions, geographic scopes, BC categories, and SoA bundle import/export).

Why

The workbench previously covered the Schedule of Activities core (visits, activities, arms, epochs) but lacked the surrounding USDM study metadata needed to produce a submission-ready USDM package or a Define-XML/JSON artifact. v1.4 closes those gaps and adds an MCP server so AI agents can drive the workbench programmatically.

How

• MCP server (src/soa_builder/mcp/server.py): 11 tools covering SoA CRUD, visit/activity management, matrix retrieval, and USDM/Define-JSON export — enables Claude and other agents to interact with the workbench via the Model Context Protocol.
• Define-JSON generator (src/usdm/create_define_json.py, generate_define_json.py): Produces CDISC Define-JSON v2.1 from the workbench database, including concepts and conceptProperties; documented in help/DEFINE_JSON_GENERATOR_INTEGRATION.md.
• CRF specializations (routers/activities.py + templates/crf_*.html): Assigns CRF specializations to activity instances with extensionAttributes in USDM export.
• New entity domains — each with router, migration, USDM generator, templates, and full test coverage: Organizations, Persons, Roles, Estimands, Indications, Study Interventions, Study Titles, Study Identifiers, BC Categories.
• Amendments extended: Enrollments, geographic scopes (regions + country codes), and governance dates added as sub-entities.
• SoA bundle (routers/soa_bundle.py): Export/import of a complete SoA as a portable JSON bundle.
• HTML matrix export (templates/soa_matrix_export.html): Standalone HTML page of the SoA matrix for easy web deployment.
• Audit trail (audit.py): Centralised before/after audit logging extracted from inline router code.
• USDM systemVersion: Stamped from the current git branch name; release-v-* prefix normalized automatically.
• BCP response code cleanup (scripts/cleanup_bcp_response_codes.py): Fixes stale and mismatched BiomedicalConceptProperty response codes across all SoAs.
• config.env: Centralised environment variable management for deployment.

Changes

• MCP: New src/soa_builder/mcp/ package — server.py with 11 tools; pyproject.toml entry point added
• Define-JSON: src/usdm/create_define_json.py (4 501 lines), generate_define_json.py, UI at templates/define_json.html
• New routers: bc_categories, estimands, indications, organizations, persons, roles, soa_bundle, study_identifiers, study_interventions, study_titles
• CRF specializations: templates/crf_cell.html, crf_specialization_detail.html, crf_specializations.html
• Amendments: Extended templates for enrollments, geographic scopes, governance dates
• USDM generators: New generate_estimands, generate_indications, generate_organizations, generate_roles, generate_study_identifiers, generate_study_interventions, generate_study_titles, generate_bc_categories
• Audit: web/audit.py (319 lines) extracted from inline router code
• Migrations: migrate_database.py extended with all new entity tables
• Docs/help: Moved docs → help; added ALIGNMENT_EAGER_BCP_VS_DEFINE_JSON, BIOMEDICAL_CONCEPT_PROPERTY_EAGER_POPULATION, DEFINE_JSON_GENERATOR_INTEGRATION, DIFF_REPORT_ALL_USDM_ENTITIES, ORGANIZATIONS guides
• Output JSON: Updated USDM and Define-JSON snapshots for H2Q-MC-LZZT and NCT01797120
• CI: Removed release-* branch match from azure-deploy.yml (deploy only from master)

Testing

• Full test suite passes — 18 new test files added covering all new routers and generators
• MCP server tested end-to-end (tests/test_mcp_server.py, 295 lines)
• Define-JSON generation tested with concept/conceptProperty population (tests/test_define_json_concepts.py, test_define_json_generator.py)
• BCP response code cleanup script tested (tests/test_bcp_response_code_cleanup.py)
• CRF specialization CRUD and USDM export tested (tests/test_routers_activities_crf.py)
• SoA bundle export/import round-trip tested (tests/test_routers_soa_bundle.py)
• All new entity routers have dedicated test files

Notes for reviewers

• The src/usdm/create_define_json.py file is large (4 500+ lines) — it is a self-contained generator and can be skimmed structurally rather than line-by-line.
• migrate_database.py is the source of truth for all schema changes; review the new ALTER TABLE / CREATE TABLE blocks there.
• Subject data CSV files (files/subject_data/NCT01797120/) were removed — they are no longer needed in the repo.
• The large files/D1_Master Protocol…pdf (38 MB) was added to files/ as reference material; confirm this is intentional before merge if storage is a concern.