diff --git a/docs/15-testing/agent-editor-test-cases.md b/docs/15-testing/agent-editor-test-cases.md new file mode 100644 index 00000000..6b7fa86b --- /dev/null +++ b/docs/15-testing/agent-editor-test-cases.md @@ -0,0 +1,1320 @@ +# Agent, Model, Knowledge Base & Consumed MCP Service Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#386](https://github.com/mendixlabs/mxcli/pull/386) + +## Test Projects + +Demo apps from [Mendix App Gallery](https://appgallery.mendixcloud.com/): + +| App | Studio Pro | Agents | Models | KBs | MCP Services | +|-----|-----------|--------|--------|-----|--------------| +| Lato Enquiry Management | 11.4.0 | — | — | — | — | +| Evora - Factory Management | 10.24.15 | — | — | — | — | +| Lato Product Inventory | 11.2.0 | — | — | — | — | + +--- + +## Setup + +### 1. Download test apps + +1. Go to [Mendix App Gallery](https://appgallery.mendixcloud.com/) +2. Download each demo app listed above +3. Open each `.mpk` in Studio Pro to extract the `.mpr` file + +### 2. Build mxcli + +```bash +make build && make test && make lint-go +``` + +### 3. Smoke test + +```bash +APPS_DIR= +for mpr in "$APPS_DIR"/*/*.mpr; do + echo "=== $(basename $(dirname $mpr)) ===" + echo "show agents;" > /tmp/show-agents.mdl + mxcli exec /tmp/show-agents.mdl -p "$mpr" 2>&1 | tail -1 +done +``` + +Expected: count line `(N agents)` for each project. + +### 4. Interactive testing + +```bash +mxcli repl -p /EnquiriesManagement.mpr +``` + +### 5. Script-based testing + +```bash +mxcli exec test-sequence.mdl -p +``` + +Write operations (CREATE, DROP) modify the `.mpr` file **in place**. + +> **IMPORTANT:** Always run destructive tests against a **copy** of the project folder, +> never the original. The `.mpr` file references other files in the project directory. +> Dropped documents cannot be recovered — there is no undo. +> +> ```bash +> # Before each destructive test session +> cp -r MyProject MyProject-test +> mxcli repl -p MyProject-test/MyProject.mpr +> ``` + +--- + +## 1. SHOW AGENTS + +### 1.1 List all agents + +``` +show agents; +``` + +**Expected:** Table with columns `Qualified Name | Module | Name | Usage | Model | Tools | KBs`. Summary line `(N agents)`. Sorted alphabetically. + +### 1.2 List agents in a module + +``` +show agents in MyModule; +``` + +**Expected:** Only agents from `MyModule`. Same column format. + +### 1.3 Empty module + +``` +show agents in NonExistentModule; +``` + +**Expected:** Error or empty result with `(0 agents)`. + +--- + +## 2. DESCRIBE AGENT + +### 2.1 Basic agent + +``` +describe agent MyModule.ChatAgent; +``` + +**Expected:** Full MDL output with all properties: + +```mdl +create agent MyModule.ChatAgent ( + UsageType: chat, + Description: 'A helpful chat agent', + Model: MyModule.GPT4o, + Entity: MyModule.ChatContext, + Variables: 'userInput', + MaxTokens: 4096, + ToolChoice: auto, + Temperature: 0.7, + TopP: 0.9, + SystemPrompt: $$You are a helpful assistant.$$, + UserPrompt: $$Answer the user's question: {userInput}$$ +); +``` + +### 2.2 Agent with tools + +``` +describe agent MyModule.ToolAgent; +``` + +**Expected:** MDL includes tool blocks: + +```mdl +create agent MyModule.ToolAgent ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$You can use tools.$$ +) +tool FetchData { + ToolType: microflow, + Document: MyModule.ACT_FetchData, + Enabled: true, + Description: 'Fetches data from the database' +} +tool Summarize { + ToolType: microflow, + Document: MyModule.ACT_Summarize, + Enabled: true, + Description: 'Summarizes text input' +}; +``` + +### 2.3 Agent with MCP services + +``` +describe agent MyModule.McpAgent; +``` + +**Expected:** MDL includes MCP service blocks: + +```mdl +create agent MyModule.McpAgent ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Use external services.$$ +) +mcp service MyModule.ExternalSvc { + Enabled: true, + Description: 'External data provider' +}; +``` + +### 2.4 Agent with knowledge bases + +``` +describe agent MyModule.KbAgent; +``` + +**Expected:** MDL includes knowledge base blocks: + +```mdl +create agent MyModule.KbAgent ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Use knowledge bases to answer.$$ +) +knowledge base Docs { + Source: MyModule.DocumentationKB, + Collection: 'docs-collection-id', + MaxResults: 10, + Description: 'Product documentation', + Enabled: true +}; +``` + +### 2.5 Agent with all block types + +Find or create an agent with tools, MCP services, and knowledge bases. Verify all three block types appear in `describe` output. + +### 2.6 Dollar-quoted prompts + +Verify `SystemPrompt` and `UserPrompt` use `$$...$$` quoting. Prompts with single quotes, newlines, and special characters must survive roundtrip. + +### 2.7 Non-existent agent + +``` +describe agent Fake.Missing; +``` + +**Expected:** Error — agent not found. + +--- + +## 3. CREATE AGENT + +### 3.1 Minimal agent + +``` +create agent MyModule.SimpleAgent ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Hello.$$ +); +``` + +**Expected:** Agent created. `show agents` lists it. `describe` matches input. + +### 3.2 Full agent with all properties + +``` +create agent MyModule.FullAgent ( + UsageType: chat, + Description: 'Full-featured test agent', + Model: MyModule.GPT4o, + Entity: MyModule.AgentContext, + Variables: 'input, context', + MaxTokens: 8192, + ToolChoice: auto, + Temperature: 0.5, + TopP: 0.95, + SystemPrompt: $$You are an expert assistant. +You have access to tools and knowledge bases.$$, + UserPrompt: $$Process this request: {input} +Context: {context}$$ +); +``` + +**Expected:** Agent created with all properties preserved. `describe` output matches. + +### 3.3 Agent with tool blocks + +``` +create agent MyModule.TooledAgent ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Use tools when needed.$$ +) +tool LookupCustomer { + ToolType: microflow, + Document: MyModule.ACT_LookupCustomer, + Enabled: true, + Description: 'Finds customer by name' +} +tool CreateOrder { + ToolType: microflow, + Document: MyModule.ACT_CreateOrder, + Enabled: true, + Description: 'Creates a new order' +}; +``` + +**Expected:** Agent created with two tools. `describe` shows both tool blocks. + +### 3.4 Agent with MCP service blocks + +``` +create agent MyModule.McpAgent ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Connect to external services.$$ +) +mcp service MyModule.WeatherSvc { + Enabled: true, + Description: 'Weather data provider' +}; +``` + +**Expected:** Agent created with MCP service reference. `describe` shows the block. + +### 3.5 Agent with knowledge base blocks + +``` +create agent MyModule.KbAgent ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Answer from knowledge.$$ +) +knowledge base FAQ { + Source: MyModule.FaqKB, + Collection: 'faq-collection', + MaxResults: 5, + Description: 'Frequently asked questions', + Enabled: true +}; +``` + +**Expected:** Agent created with knowledge base reference. `describe` shows the block. + +### 3.6 Agent with all block types + +``` +create agent MyModule.CompleteAgent ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Full agent with everything.$$ +) +tool Search { + ToolType: microflow, + Document: MyModule.ACT_Search, + Enabled: true, + Description: 'Search tool' +} +mcp service MyModule.ExternalAPI { + Enabled: true, + Description: 'External API service' +} +knowledge base Docs { + Source: MyModule.DocsKB, + Collection: 'docs-id', + MaxResults: 10, + Description: 'Documentation KB', + Enabled: true +}; +``` + +**Expected:** Agent created with tool, MCP service, and knowledge base blocks. `describe` shows all three. + +### 3.7 Auto-create module + +``` +create agent NewModule.Agent1 ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Test.$$ +); +``` + +**Expected:** `NewModule` auto-created if it does not exist. Agent created inside it. + +### 3.8 Model reference resolution + +``` +create agent MyModule.RefTest ( + UsageType: chat, + Model: OtherModule.SomeModel, + SystemPrompt: $$Test.$$ +); +``` + +**Expected:** Resolves `OtherModule.SomeModel` to existing model document. Error if model not found. + +### 3.9 Entity reference resolution + +``` +create agent MyModule.EntityRef ( + UsageType: chat, + Model: MyModule.GPT4o, + Entity: MyModule.ChatSession, + SystemPrompt: $$Test.$$ +); +``` + +**Expected:** Resolves `MyModule.ChatSession` to existing entity. Error if entity not found. + +### 3.10 KB reference resolution in agent + +``` +create agent MyModule.KbRef ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Test.$$ +) +knowledge base KB1 { + Source: MyModule.NonExistentKB, + Collection: 'id', + MaxResults: 5, + Description: 'Test', + Enabled: true +}; +``` + +**Expected:** Error — knowledge base `MyModule.NonExistentKB` not found. + +### 3.11 MCP service reference resolution in agent + +``` +create agent MyModule.McpRef ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Test.$$ +) +mcp service MyModule.NonExistentSvc { + Enabled: true, + Description: 'Test' +}; +``` + +**Expected:** Error — consumed MCP service `MyModule.NonExistentSvc` not found. + +### 3.12 Duplicate agent + +``` +create agent MyModule.SimpleAgent ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Duplicate.$$ +); +``` + +**Expected:** Error — agent already exists. + +--- + +## 4. DROP AGENT + +### 4.1 Drop existing agent + +``` +drop agent MyModule.SimpleAgent; +``` + +**Expected:** Agent removed. `show agents` no longer lists it. + +### 4.2 Drop non-existent agent + +``` +drop agent MyModule.NonExistent; +``` + +**Expected:** Error — agent not found. + +--- + +## 5. SHOW MODELS + +### 5.1 List all models + +``` +show models; +``` + +**Expected:** Table with columns `Qualified Name | Module | Name | Provider | Key Constant | Display Name`. Summary line `(N models)`. Sorted alphabetically. + +### 5.2 List models in a module + +``` +show models in MyModule; +``` + +**Expected:** Only models from `MyModule`. Same column format. + +### 5.3 Empty module + +``` +show models in NonExistentModule; +``` + +**Expected:** Error or empty result with `(0 models)`. + +--- + +## 6. DESCRIBE MODEL + +### 6.1 Basic model + +``` +describe model MyModule.GPT4o; +``` + +**Expected:** MDL output: + +```mdl +create model MyModule.GPT4o ( + Provider: MxCloudGenAI, + Key: MyModule.OpenAIKey, + DisplayName: 'GPT-4o' +); +``` + +### 6.2 Model with all properties + +Find a model with additional properties beyond Provider, Key, and DisplayName. Verify all appear in `describe` output. + +### 6.3 Non-existent model + +``` +describe model Fake.Missing; +``` + +**Expected:** Error — model not found. + +--- + +## 7. CREATE MODEL + +### 7.1 Minimal model (default provider) + +``` +create model MyModule.BasicModel ( + Key: MyModule.ApiKey, + DisplayName: 'Basic Model' +); +``` + +**Expected:** Model created. Provider defaults to `MxCloudGenAI`. `describe` matches. + +### 7.2 Explicit provider + +``` +create model MyModule.CustomModel ( + Provider: MxCloudGenAI, + Key: MyModule.ApiKey, + DisplayName: 'Custom Model' +); +``` + +**Expected:** Model created with explicit provider. + +### 7.3 Constant reference resolution + +``` +create model MyModule.RefModel ( + Key: MyModule.NonExistentConst, + DisplayName: 'Test' +); +``` + +**Expected:** Error — constant `MyModule.NonExistentConst` not found. + +### 7.4 Auto-create module + +``` +create model NewModule.Model1 ( + Key: MyModule.ApiKey, + DisplayName: 'New Module Model' +); +``` + +**Expected:** `NewModule` auto-created if it does not exist. + +### 7.5 Duplicate model + +``` +create model MyModule.BasicModel ( + Key: MyModule.ApiKey, + DisplayName: 'Duplicate' +); +``` + +**Expected:** Error — model already exists. + +--- + +## 8. DROP MODEL + +### 8.1 Drop existing model + +``` +drop model MyModule.BasicModel; +``` + +**Expected:** Model removed. `show models` no longer lists it. + +### 8.2 Drop non-existent model + +``` +drop model MyModule.NonExistent; +``` + +**Expected:** Error — model not found. + +--- + +## 9. SHOW KNOWLEDGE BASES + +### 9.1 List all knowledge bases + +``` +show knowledge bases; +``` + +**Expected:** Table with columns `Qualified Name | Module | Name | Provider | Key Constant | Embedding Model`. Summary line `(N knowledge bases)`. Sorted alphabetically. + +### 9.2 List knowledge bases in a module + +``` +show knowledge bases in MyModule; +``` + +**Expected:** Only knowledge bases from `MyModule`. Same column format. + +### 9.3 Empty module + +``` +show knowledge bases in NonExistentModule; +``` + +**Expected:** Error or empty result with `(0 knowledge bases)`. + +--- + +## 10. DESCRIBE KNOWLEDGE BASE + +### 10.1 Basic knowledge base + +``` +describe knowledge base MyModule.DocumentationKB; +``` + +**Expected:** MDL output: + +```mdl +create knowledge base MyModule.DocumentationKB ( + Provider: MxCloudGenAI, + Key: MyModule.KBKey, + ModelDisplayName: 'text-embedding-ada-002', + ModelName: 'text-embedding-ada-002' +); +``` + +### 10.2 Knowledge base with all properties + +Find a knowledge base with additional properties. Verify all appear in `describe` output. + +### 10.3 Non-existent knowledge base + +``` +describe knowledge base Fake.Missing; +``` + +**Expected:** Error — knowledge base not found. + +--- + +## 11. CREATE KNOWLEDGE BASE + +### 11.1 Minimal knowledge base + +``` +create knowledge base MyModule.TestKB ( + Key: MyModule.KBKey, + ModelDisplayName: 'text-embedding-ada-002', + ModelName: 'text-embedding-ada-002' +); +``` + +**Expected:** Knowledge base created. `describe` matches. + +### 11.2 Explicit provider + +``` +create knowledge base MyModule.FullKB ( + Provider: MxCloudGenAI, + Key: MyModule.KBKey, + ModelDisplayName: 'text-embedding-3-small', + ModelName: 'text-embedding-3-small' +); +``` + +**Expected:** Knowledge base created with explicit provider. + +### 11.3 Constant reference resolution + +``` +create knowledge base MyModule.BadKB ( + Key: MyModule.NonExistentConst, + ModelDisplayName: 'test', + ModelName: 'test' +); +``` + +**Expected:** Error — constant `MyModule.NonExistentConst` not found. + +### 11.4 Auto-create module + +``` +create knowledge base NewModule.KB1 ( + Key: MyModule.KBKey, + ModelDisplayName: 'test', + ModelName: 'test' +); +``` + +**Expected:** `NewModule` auto-created if it does not exist. + +### 11.5 Duplicate knowledge base + +``` +create knowledge base MyModule.TestKB ( + Key: MyModule.KBKey, + ModelDisplayName: 'test', + ModelName: 'test' +); +``` + +**Expected:** Error — knowledge base already exists. + +--- + +## 12. DROP KNOWLEDGE BASE + +### 12.1 Drop existing knowledge base + +``` +drop knowledge base MyModule.TestKB; +``` + +**Expected:** Knowledge base removed. `show knowledge bases` no longer lists it. + +### 12.2 Drop non-existent knowledge base + +``` +drop knowledge base MyModule.NonExistent; +``` + +**Expected:** Error — knowledge base not found. + +--- + +## 13. SHOW CONSUMED MCP SERVICES + +### 13.1 List all consumed MCP services + +``` +show consumed mcp services; +``` + +**Expected:** Table with columns `Qualified Name | Module | Name | Protocol | Version | Timeout`. Summary line `(N consumed mcp services)`. Sorted alphabetically. + +### 13.2 List consumed MCP services in a module + +``` +show consumed mcp services in MyModule; +``` + +**Expected:** Only services from `MyModule`. Same column format. + +### 13.3 Empty module + +``` +show consumed mcp services in NonExistentModule; +``` + +**Expected:** Error or empty result with `(0 consumed mcp services)`. + +--- + +## 14. DESCRIBE CONSUMED MCP SERVICE + +### 14.1 Basic consumed MCP service + +``` +describe consumed mcp service MyModule.ExternalSvc; +``` + +**Expected:** MDL output: + +```mdl +create consumed mcp service MyModule.ExternalSvc ( + ProtocolVersion: 2024-11-05, + Version: '1.0.0', + ConnectionTimeoutSeconds: 30, + Documentation: 'External data provider for weather information' +); +``` + +### 14.2 Service with all properties + +Find a consumed MCP service with all available properties. Verify all appear in `describe` output. + +### 14.3 Non-existent consumed MCP service + +``` +describe consumed mcp service Fake.Missing; +``` + +**Expected:** Error — consumed MCP service not found. + +--- + +## 15. CREATE CONSUMED MCP SERVICE + +### 15.1 Minimal consumed MCP service + +``` +create consumed mcp service MyModule.WeatherSvc ( + ProtocolVersion: 2024-11-05, + Version: '1.0.0', + ConnectionTimeoutSeconds: 30 +); +``` + +**Expected:** Service created. `describe` matches. + +### 15.2 With documentation + +``` +create consumed mcp service MyModule.DataSvc ( + ProtocolVersion: 2024-11-05, + Version: '2.0.0', + ConnectionTimeoutSeconds: 60, + Documentation: 'Provides access to external data sources' +); +``` + +**Expected:** Service created with documentation. `describe` shows `Documentation` property. + +### 15.3 Auto-create module + +``` +create consumed mcp service NewModule.Svc1 ( + ProtocolVersion: 2024-11-05, + Version: '1.0.0', + ConnectionTimeoutSeconds: 30 +); +``` + +**Expected:** `NewModule` auto-created if it does not exist. + +### 15.4 Duplicate consumed MCP service + +``` +create consumed mcp service MyModule.WeatherSvc ( + ProtocolVersion: 2024-11-05, + Version: '1.0.0', + ConnectionTimeoutSeconds: 30 +); +``` + +**Expected:** Error — consumed MCP service already exists. + +--- + +## 16. DROP CONSUMED MCP SERVICE + +### 16.1 Drop existing consumed MCP service + +``` +drop consumed mcp service MyModule.WeatherSvc; +``` + +**Expected:** Service removed. `show consumed mcp services` no longer lists it. + +### 16.2 Drop non-existent consumed MCP service + +``` +drop consumed mcp service MyModule.NonExistent; +``` + +**Expected:** Error — consumed MCP service not found. + +--- + +## 17. ROUNDTRIP + +Test that CREATE → DESCRIBE → CREATE (from output) produces identical results. + +### 17.1 Agent roundtrip + +``` +create agent RtTest.Agent1 ( + UsageType: chat, + Description: 'Roundtrip test agent', + Model: RtTest.Model1, + MaxTokens: 4096, + Temperature: 0.7, + SystemPrompt: $$You are a test agent with 'quotes' and +newlines in the prompt.$$, + UserPrompt: $$User said: {input}$$ +) +tool Search { + ToolType: microflow, + Document: RtTest.ACT_Search, + Enabled: true, + Description: 'Search tool' +} +knowledge base Docs { + Source: RtTest.DocsKB, + Collection: 'docs-id', + MaxResults: 5, + Description: 'Documentation', + Enabled: true +}; +``` + +1. Run `describe agent RtTest.Agent1` +2. Copy output +3. Drop agent: `drop agent RtTest.Agent1` +4. Execute copied MDL +5. Run `describe` again + +**Expected:** Output identical between step 1 and step 5. + +### 17.2 Model roundtrip + +``` +create model RtTest.Model1 ( + Provider: MxCloudGenAI, + Key: RtTest.ApiKey, + DisplayName: 'Test Model' +); +``` + +1. `describe model RtTest.Model1` +2. Drop: `drop model RtTest.Model1` +3. Execute described MDL +4. `describe` again + +**Expected:** Identical output. + +### 17.3 Knowledge base roundtrip + +``` +create knowledge base RtTest.DocsKB ( + Provider: MxCloudGenAI, + Key: RtTest.KBKey, + ModelDisplayName: 'text-embedding-ada-002', + ModelName: 'text-embedding-ada-002' +); +``` + +1. `describe knowledge base RtTest.DocsKB` +2. Drop: `drop knowledge base RtTest.DocsKB` +3. Execute described MDL +4. `describe` again + +**Expected:** Identical output. + +### 17.4 Consumed MCP service roundtrip + +``` +create consumed mcp service RtTest.ExternalSvc ( + ProtocolVersion: 2024-11-05, + Version: '1.0.0', + ConnectionTimeoutSeconds: 30, + Documentation: 'Test service' +); +``` + +1. `describe consumed mcp service RtTest.ExternalSvc` +2. Drop: `drop consumed mcp service RtTest.ExternalSvc` +3. Execute described MDL +4. `describe` again + +**Expected:** Identical output. + +--- + +## 18. MULTI-STEP WORKFLOWS + +### 18.1 Full agent stack creation + +Create all supporting documents, then an agent referencing them all. + +``` +-- Step 1: Create constant for API key +create constant MyModule.AgentApiKey type String default 'sk-test-key'; + +-- Step 2: Create model referencing the constant +create model MyModule.TestModel ( + Provider: MxCloudGenAI, + Key: MyModule.AgentApiKey, + DisplayName: 'Test GPT-4o' +); + +-- Step 3: Create constant for KB key +create constant MyModule.KBApiKey type String default 'kb-key'; + +-- Step 4: Create knowledge base referencing the constant +create knowledge base MyModule.TestKB ( + Provider: MxCloudGenAI, + Key: MyModule.KBApiKey, + ModelDisplayName: 'text-embedding-ada-002', + ModelName: 'text-embedding-ada-002' +); + +-- Step 5: Create consumed MCP service +create consumed mcp service MyModule.TestMcpSvc ( + ProtocolVersion: 2024-11-05, + Version: '1.0.0', + ConnectionTimeoutSeconds: 30, + Documentation: 'Test MCP service' +); + +-- Step 6: Create agent referencing model, KB, and MCP service +create agent MyModule.TestAgent ( + UsageType: chat, + Description: 'Agent with all references', + Model: MyModule.TestModel, + MaxTokens: 4096, + Temperature: 0.7, + SystemPrompt: $$You have tools, knowledge, and services.$$ +) +mcp service MyModule.TestMcpSvc { + Enabled: true, + Description: 'Test MCP' +} +knowledge base KB { + Source: MyModule.TestKB, + Collection: 'test-collection', + MaxResults: 10, + Description: 'Test knowledge base', + Enabled: true +}; + +-- Step 7: Verify +describe agent MyModule.TestAgent; +``` + +**Expected:** All six create statements succeed. `describe` output shows model reference, MCP service block, and knowledge base block. + +### 18.2 Cross-module references + +``` +create constant ModA.Key type String default 'key'; +create model ModA.SharedModel (Key: ModA.Key, DisplayName: 'Shared'); +create agent ModB.CrossAgent ( + UsageType: chat, + Model: ModA.SharedModel, + SystemPrompt: $$Cross-module test.$$ +); +describe agent ModB.CrossAgent; +``` + +**Expected:** Agent in `ModB` references model in `ModA`. `describe` shows fully qualified model name. + +--- + +## 19. FAILURE MODES & ERROR RECOVERY + +### 19.1 Not connected + +Run any command without `-p` flag or REPL session. + +**Expected:** Error — not connected to a project. + +### 19.2 Agent already exists + +``` +create agent MyModule.TestAgent (...); +create agent MyModule.TestAgent (...); +``` + +**Expected:** Second create fails — agent already exists. + +### 19.3 Model not found during agent creation + +``` +create agent MyModule.BadAgent ( + UsageType: chat, + Model: MyModule.NonExistentModel, + SystemPrompt: $$Test.$$ +); +``` + +**Expected:** Error — model `MyModule.NonExistentModel` not found. + +### 19.4 KB not found during agent creation + +``` +create agent MyModule.BadAgent ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Test.$$ +) +knowledge base KB1 { + Source: MyModule.NonExistentKB, + Collection: 'id', + MaxResults: 5, + Description: 'Test', + Enabled: true +}; +``` + +**Expected:** Error — knowledge base not found. + +### 19.5 MCP service not found during agent creation + +``` +create agent MyModule.BadAgent ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Test.$$ +) +mcp service MyModule.NonExistentSvc { + Enabled: true, + Description: 'Test' +}; +``` + +**Expected:** Error — consumed MCP service not found. + +### 19.6 Constant not found during model creation + +``` +create model MyModule.BadModel ( + Key: MyModule.FakeConst, + DisplayName: 'Bad' +); +``` + +**Expected:** Error — constant not found. + +### 19.7 Constant not found during KB creation + +``` +create knowledge base MyModule.BadKB ( + Key: MyModule.FakeConst, + ModelDisplayName: 'test', + ModelName: 'test' +); +``` + +**Expected:** Error — constant not found. + +### 19.8 Drop model referenced by agent + +``` +drop model MyModule.TestModel; +``` + +**Expected:** Warning about agent references. Model dropped (agent may have dangling reference). + +--- + +## 20. BOUNDARY & STRESS + +### 20.1 Agent with many tools (10+) + +Create an agent with 10+ tool blocks. Verify `describe` lists all tools. + +### 20.2 Agent with many knowledge bases (5+) + +Create an agent with 5+ knowledge base blocks. Verify `describe` lists all. + +### 20.3 Long dollar-quoted prompt + +Create an agent with a `SystemPrompt` exceeding 4000 characters. Verify `describe` preserves full text. + +### 20.4 Special characters in prompts + +``` +create agent MyModule.SpecialChars ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Prompt with 'single quotes', "double quotes", +backslash \, dollar $, and unicode: café résumé$$ +); +``` + +**Expected:** All characters preserved in `describe` output. + +### 20.5 Long service documentation + +``` +create consumed mcp service MyModule.DocSvc ( + ProtocolVersion: 2024-11-05, + Version: '1.0.0', + ConnectionTimeoutSeconds: 30, + Documentation: 'Very long documentation string...(1000+ chars)' +); +``` + +**Expected:** Full documentation preserved in `describe`. + +--- + +## Test Project Coverage Matrix + +| Operation | Lato Enquiry | Evora Factory | Lato Product | +|-----------|:---:|:---:|:---:| +| SHOW AGENTS | x | x | x | +| DESCRIBE AGENT | x | x | x | +| CREATE AGENT | x | | | +| DROP AGENT | x | | | +| SHOW MODELS | x | x | x | +| DESCRIBE MODEL | x | x | | +| CREATE MODEL | x | | | +| DROP MODEL | x | | | +| SHOW KNOWLEDGE BASES | x | x | x | +| DESCRIBE KNOWLEDGE BASE | x | x | | +| CREATE KNOWLEDGE BASE | x | | | +| DROP KNOWLEDGE BASE | x | | | +| SHOW CONSUMED MCP SERVICES | x | x | x | +| DESCRIBE CONSUMED MCP SERVICE | x | x | | +| CREATE CONSUMED MCP SERVICE | x | | | +| DROP CONSUMED MCP SERVICE | x | | | + +Read operations tested on all projects. Write operations on copies of one project. + +--- + +## Automated Test Coverage + +| Section | Automated | Manual-only | +|---------|:---------:|:-----------:| +| 1. SHOW AGENTS | Mock tests | | +| 2. DESCRIBE AGENT | Mock tests | All block types | +| 3. CREATE AGENT | Mock + roundtrip | Reference resolution | +| 4. DROP AGENT | Mock tests | | +| 5. SHOW MODELS | Mock tests | | +| 6. DESCRIBE MODEL | Mock tests | | +| 7. CREATE MODEL | Mock tests | Constant resolution | +| 8. DROP MODEL | Mock tests | | +| 9. SHOW KNOWLEDGE BASES | Mock tests | | +| 10. DESCRIBE KNOWLEDGE BASE | Mock tests | | +| 11. CREATE KNOWLEDGE BASE | Mock tests | Constant resolution | +| 12. DROP KNOWLEDGE BASE | Mock tests | | +| 13. SHOW CONSUMED MCP SERVICES | Mock tests | | +| 14. DESCRIBE CONSUMED MCP SERVICE | Mock tests | | +| 15. CREATE CONSUMED MCP SERVICE | Mock tests | | +| 16. DROP CONSUMED MCP SERVICE | Mock tests | | +| 17. Roundtrip | Roundtrip tests | Complex agents | +| 18. Multi-step | | All manual | +| 19. Failure modes | Partial | Edge cases | +| 20. Boundary | | All manual | + +--- + +## Manual Test Report Template + +**Tester:** _______________ +**Date:** _______________ +**Project:** _______________ + +| # | Section | Test | Pass | Fail | Skip | Notes | +|---|---------|------|:----:|:----:|:----:|-------| +| 1.1 | SHOW AGENTS | List all | | | | | +| 1.2 | SHOW AGENTS | Filter by module | | | | | +| 1.3 | SHOW AGENTS | Empty module | | | | | +| 2.1 | DESCRIBE AGENT | Basic | | | | | +| 2.2 | DESCRIBE AGENT | With tools | | | | | +| 2.3 | DESCRIBE AGENT | With MCP services | | | | | +| 2.4 | DESCRIBE AGENT | With KBs | | | | | +| 2.5 | DESCRIBE AGENT | All block types | | | | | +| 2.6 | DESCRIBE AGENT | Dollar-quoted prompts | | | | | +| 2.7 | DESCRIBE AGENT | Not found | | | | | +| 3.1 | CREATE AGENT | Minimal | | | | | +| 3.2 | CREATE AGENT | Full properties | | | | | +| 3.3 | CREATE AGENT | With tools | | | | | +| 3.4 | CREATE AGENT | With MCP services | | | | | +| 3.5 | CREATE AGENT | With KBs | | | | | +| 3.6 | CREATE AGENT | All block types | | | | | +| 3.7 | CREATE AGENT | Auto-create module | | | | | +| 3.8 | CREATE AGENT | Model reference | | | | | +| 3.9 | CREATE AGENT | Entity reference | | | | | +| 3.10 | CREATE AGENT | KB not found | | | | | +| 3.11 | CREATE AGENT | MCP not found | | | | | +| 3.12 | CREATE AGENT | Duplicate error | | | | | +| 4.1 | DROP AGENT | Existing | | | | | +| 4.2 | DROP AGENT | Non-existent | | | | | +| 5.1 | SHOW MODELS | List all | | | | | +| 5.2 | SHOW MODELS | Filter by module | | | | | +| 5.3 | SHOW MODELS | Empty module | | | | | +| 6.1 | DESCRIBE MODEL | Basic | | | | | +| 6.2 | DESCRIBE MODEL | All properties | | | | | +| 6.3 | DESCRIBE MODEL | Not found | | | | | +| 7.1 | CREATE MODEL | Minimal (default provider) | | | | | +| 7.2 | CREATE MODEL | Explicit provider | | | | | +| 7.3 | CREATE MODEL | Constant not found | | | | | +| 7.4 | CREATE MODEL | Auto-create module | | | | | +| 7.5 | CREATE MODEL | Duplicate error | | | | | +| 8.1 | DROP MODEL | Existing | | | | | +| 8.2 | DROP MODEL | Non-existent | | | | | +| 9.1 | SHOW KBS | List all | | | | | +| 9.2 | SHOW KBS | Filter by module | | | | | +| 9.3 | SHOW KBS | Empty module | | | | | +| 10.1 | DESCRIBE KB | Basic | | | | | +| 10.2 | DESCRIBE KB | All properties | | | | | +| 10.3 | DESCRIBE KB | Not found | | | | | +| 11.1 | CREATE KB | Minimal | | | | | +| 11.2 | CREATE KB | Explicit provider | | | | | +| 11.3 | CREATE KB | Constant not found | | | | | +| 11.4 | CREATE KB | Auto-create module | | | | | +| 11.5 | CREATE KB | Duplicate error | | | | | +| 12.1 | DROP KB | Existing | | | | | +| 12.2 | DROP KB | Non-existent | | | | | +| 13.1 | SHOW MCP | List all | | | | | +| 13.2 | SHOW MCP | Filter by module | | | | | +| 13.3 | SHOW MCP | Empty module | | | | | +| 14.1 | DESCRIBE MCP | Basic | | | | | +| 14.2 | DESCRIBE MCP | All properties | | | | | +| 14.3 | DESCRIBE MCP | Not found | | | | | +| 15.1 | CREATE MCP | Minimal | | | | | +| 15.2 | CREATE MCP | With documentation | | | | | +| 15.3 | CREATE MCP | Auto-create module | | | | | +| 15.4 | CREATE MCP | Duplicate error | | | | | +| 16.1 | DROP MCP | Existing | | | | | +| 16.2 | DROP MCP | Non-existent | | | | | +| 17.1 | ROUNDTRIP | Agent | | | | | +| 17.2 | ROUNDTRIP | Model | | | | | +| 17.3 | ROUNDTRIP | Knowledge base | | | | | +| 17.4 | ROUNDTRIP | Consumed MCP service | | | | | +| 18.1 | MULTI-STEP | Full agent stack | | | | | +| 18.2 | MULTI-STEP | Cross-module refs | | | | | +| 19.1 | FAILURE | Not connected | | | | | +| 19.2 | FAILURE | Already exists | | | | | +| 19.3 | FAILURE | Model not found | | | | | +| 19.4 | FAILURE | KB not found | | | | | +| 19.5 | FAILURE | MCP not found | | | | | +| 19.6 | FAILURE | Constant not found (model) | | | | | +| 19.7 | FAILURE | Constant not found (KB) | | | | | +| 19.8 | FAILURE | Drop referenced model | | | | | +| 20.1 | BOUNDARY | Many tools | | | | | +| 20.2 | BOUNDARY | Many KBs | | | | | +| 20.3 | BOUNDARY | Long prompt | | | | | +| 20.4 | BOUNDARY | Special characters | | | | | +| 20.5 | BOUNDARY | Long documentation | | | | | + +**Summary:** ___ / ___ passed | ___ failed | ___ skipped diff --git a/docs/15-testing/business-event-test-cases.md b/docs/15-testing/business-event-test-cases.md new file mode 100644 index 00000000..5bd04e00 --- /dev/null +++ b/docs/15-testing/business-event-test-cases.md @@ -0,0 +1,576 @@ +# Business Event Service Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#386](https://github.com/mendixlabs/mxcli/pull/386) + +## Test Projects + +Demo apps from [Mendix App Gallery](https://appgallery.mendixcloud.com/): + +| App | Studio Pro | Business Event Services | +|-----|-----------|------------------------| +| Lato Enquiry Management | 11.4.0 | — | +| Evora - Factory Management | 10.24.15 | — | +| Lato Product Inventory | 11.2.0 | — | + +--- + +## Setup + +### 1. Download test apps + +1. Go to [Mendix App Gallery](https://appgallery.mendixcloud.com/) +2. Download each demo app listed above +3. Open each `.mpk` in Studio Pro to extract the `.mpr` file + +### 2. Build mxcli + +```bash +make build && make test && make lint-go +``` + +### 3. Smoke test + +```bash +APPS_DIR= +for mpr in "$APPS_DIR"/*/*.mpr; do + echo "=== $(basename $(dirname $mpr)) ===" + echo "show business event services;" > /tmp/show-bes.mdl + mxcli exec /tmp/show-bes.mdl -p "$mpr" 2>&1 | tail -1 +done +``` + +Expected: count line `(N business event services)` for each project. + +### 4. Interactive testing + +```bash +mxcli repl -p /EnquiriesManagement.mpr +``` + +### 5. Script-based testing + +```bash +mxcli exec test-sequence.mdl -p +``` + +Write operations (CREATE, DROP) modify the `.mpr` file **in place**. + +> **IMPORTANT:** Always run destructive tests against a **copy** of the project folder, +> never the original. The `.mpr` file references other files in the project directory. +> Dropped services cannot be recovered — there is no undo. +> +> ```bash +> # Before each destructive test session +> cp -r MyProject MyProject-test +> mxcli repl -p MyProject-test/MyProject.mpr +> ``` + +--- + +## 1. SHOW BUSINESS EVENT SERVICES + +### 1.1 List all services + +``` +show business event services; +``` + +**Expected:** Table with columns `Module | QualifiedName | Service | Messages | Publish | Subscribe`. Summary line `(N business event services)`. Sorted alphabetically. + +### 1.2 Filter by module + +``` +show business event services in MyModule; +``` + +**Expected:** Only services from `MyModule`. Same column format. + +### 1.3 Empty result + +``` +show business event services in NonExistentModule; +``` + +**Expected:** `No business event services found.` + +--- + +## 2. SHOW BUSINESS EVENTS + +### 2.1 List all business events + +``` +show business events; +``` + +**Expected:** Table with columns `Service | Message | Operation | Entity | Attributes`. `Operation` values are `PUBLISH` or `SUBSCRIBE`. Summary line `(N business events)`. + +### 2.2 Filter by module + +``` +show business events in MyModule; +``` + +**Expected:** Only events from `MyModule`. Same column format. + +### 2.3 Empty module + +``` +show business events in NonExistentModule; +``` + +**Expected:** Empty result or `No business events found.` + +--- + +## 3. DESCRIBE BUSINESS EVENT SERVICE + +### 3.1 Service with publish message + +``` +describe business event service MyModule.OrderEvents; +``` + +**Expected:** Full MDL output: +``` +create business event service MyModule.OrderEvents + event_name_prefix 'com.example.order' + message OrderCreated ( + OrderId: Long, + CustomerName: String, + Total: Decimal + ) publish entity MyModule.OrderCreatedEvent +/ +``` + +### 3.2 Service with subscribe message + +``` +describe business event service MyModule.PaymentEvents; +``` + +**Expected:** MDL includes `subscribe` keyword with entity and microflow references: +``` +create business event service MyModule.PaymentEvents + event_name_prefix 'com.example.payment' + message PaymentReceived ( + PaymentId: Long, + Amount: Decimal, + Currency: String + ) subscribe entity MyModule.PaymentReceivedEvent microflow MyModule.OnPaymentReceived +/ +``` + +### 3.3 Service with multiple messages + +``` +describe business event service MyModule.InventoryEvents; +``` + +**Expected:** Multiple `message` blocks in output. Each has its own attribute list, operation keyword, and entity reference. + +### 3.4 All attribute types + +Verify described output includes attributes of each supported type: + +| Type | Example | +|------|---------| +| Long | `OrderId: Long` | +| String | `Name: String` | +| Integer | `Count: Integer` | +| Boolean | `IsActive: Boolean` | +| DateTime | `Timestamp: DateTime` | +| Decimal | `Amount: Decimal` | + +### 3.5 Non-existent service + +``` +describe business event service MyModule.Fake; +``` + +**Expected:** Error — service not found. + +--- + +## 4. CREATE BUSINESS EVENT SERVICE + +> **Note:** The `event_name_prefix` and `message` keywords cause parse errors in the current build. The CREATE BUSINESS EVENT SERVICE grammar may not be fully implemented. Tests in this section may fail at the parser level rather than at execution. + +### 4.1 Minimal publish service + +``` +create business event service MyModule.OrderEvents + event_name_prefix 'com.example.order' + message OrderCreated ( + OrderId: Long, + Description: String + ) publish entity MyModule.OrderCreatedEvent; +``` + +**Expected:** Service created. `describe business event service MyModule.OrderEvents` matches input. + +### 4.2 Subscribe service with microflow + +``` +create business event service MyModule.PaymentEvents + event_name_prefix 'com.example.payment' + message PaymentReceived ( + PaymentId: Long, + Amount: Decimal, + Currency: String + ) subscribe entity MyModule.PaymentReceivedEvent microflow MyModule.OnPaymentReceived; +``` + +**Expected:** Service created. `describe` shows `subscribe` keyword, entity, and microflow references. + +### 4.3 Multiple messages + +``` +create business event service MyModule.InventoryEvents + event_name_prefix 'com.example.inventory' + message StockUpdated ( + ProductId: Long, + NewQuantity: Integer + ) publish entity MyModule.StockUpdatedEvent + message StockDepleted ( + ProductId: Long, + WarehouseId: String + ) subscribe entity MyModule.StockDepletedEvent microflow MyModule.OnStockDepleted; +``` + +**Expected:** Service created with two messages. `describe` shows both message blocks. + +### 4.4 All attribute types + +``` +create business event service MyModule.TypeTest + event_name_prefix 'com.example.typetest' + message AllTypes ( + Id: Long, + Name: String, + Count: Integer, + Active: Boolean, + Timestamp: DateTime, + Amount: Decimal + ) publish entity MyModule.AllTypesEvent; +``` + +**Expected:** Service created. `describe` preserves all six attribute types. + +### 4.5 `create or replace` variant + +``` +create or replace business event service MyModule.OrderEvents + event_name_prefix 'com.example.order.v2' + message OrderCreatedV2 ( + OrderId: Long, + Description: String, + Priority: Integer + ) publish entity MyModule.OrderCreatedV2Event; +``` + +**Expected:** Drops existing service and recreates. `describe` shows updated prefix and message. + +### 4.6 `in folder` clause + +``` +create business event service MyModule.FolderTest + event_name_prefix 'com.example.foldertest' + in folder 'Events/Inbound' + message TestMsg ( + Value: String + ) publish entity MyModule.TestMsgEvent; +``` + +**Expected:** Service created in specified folder. Folder auto-created if missing. + +### 4.7 Export level defaults to Hidden + +``` +create business event service MyModule.HiddenCheck + event_name_prefix 'com.example.hidden' + message Msg ( + X: Integer + ) publish entity MyModule.HiddenCheckEvent; +``` + +**Expected:** Service created. Export level defaults to `Hidden`. Verify via `describe` output or model inspection. + +### 4.8 Duplicate service (without `or replace`) + +``` +create business event service MyModule.OrderEvents + event_name_prefix 'com.example.order' + message OrderCreated ( + OrderId: Long + ) publish entity MyModule.OrderCreatedEvent; +``` + +**Expected:** Error — service already exists. + +--- + +## 5. DROP BUSINESS EVENT SERVICE + +### 5.1 Drop existing service + +``` +drop business event service MyModule.OrderEvents; +``` + +**Expected:** Service removed. `describe business event service MyModule.OrderEvents` returns error. + +### 5.2 Drop non-existent service + +``` +drop business event service MyModule.Fake; +``` + +**Expected:** Error — service not found. + +--- + +## 6. ROUNDTRIP + +### 6.1 Publish service roundtrip + +``` +create business event service RtTest.Events + event_name_prefix 'com.example.rt' + message Created ( + Id: Long, + Name: String, + Amount: Decimal + ) publish entity RtTest.CreatedEvent; +``` + +1. Run `describe business event service RtTest.Events` +2. Copy output +3. Drop: `drop business event service RtTest.Events` +4. Execute copied MDL +5. Run `describe` again + +**Expected:** Output identical between step 1 and step 5. + +### 6.2 Subscribe service roundtrip + +``` +create business event service RtTest.SubEvents + event_name_prefix 'com.example.rtsub' + message Received ( + PaymentId: Long, + Status: String + ) subscribe entity RtTest.ReceivedEvent microflow RtTest.OnReceived; +``` + +1. `describe business event service RtTest.SubEvents` +2. Drop: `drop business event service RtTest.SubEvents` +3. Execute described MDL +4. `describe` again + +**Expected:** Identical output. + +--- + +## 7. MULTI-STEP WORKFLOWS + +### 7.1 Create entity → create publish service → verify events + +``` +create persistent entity MyModule.InvoiceCreatedEvent ( + InvoiceId: Long, + CustomerName: String, + TotalAmount: Decimal +); + +create business event service MyModule.InvoiceEvents + event_name_prefix 'com.example.invoice' + message InvoiceCreated ( + InvoiceId: Long, + CustomerName: String, + TotalAmount: Decimal + ) publish entity MyModule.InvoiceCreatedEvent; + +show business events in MyModule; +``` + +**Expected:** All statements succeed. `show business events` lists `InvoiceCreated` with operation `PUBLISH` and entity `MyModule.InvoiceCreatedEvent`. + +### 7.2 Create entity → create subscribe service → verify events + +``` +create persistent entity MyModule.ShipmentReceivedEvent ( + ShipmentId: Long, + Carrier: String +); + +create business event service MyModule.ShipmentEvents + event_name_prefix 'com.example.shipment' + message ShipmentReceived ( + ShipmentId: Long, + Carrier: String + ) subscribe entity MyModule.ShipmentReceivedEvent microflow MyModule.OnShipmentReceived; + +show business events in MyModule; +``` + +**Expected:** `show business events` lists `ShipmentReceived` with operation `SUBSCRIBE`. + +### 7.3 Create → replace → verify + +``` +create business event service MyModule.Lifecycle + event_name_prefix 'com.example.lc.v1' + message Ping ( + Seq: Integer + ) publish entity MyModule.PingEvent; + +create or replace business event service MyModule.Lifecycle + event_name_prefix 'com.example.lc.v2' + message Ping ( + Seq: Integer, + Timestamp: DateTime + ) publish entity MyModule.PingEvent; + +describe business event service MyModule.Lifecycle; +``` + +**Expected:** Final `describe` shows v2 prefix and `Timestamp` attribute. + +--- + +## 8. FAILURE MODES & ERROR RECOVERY + +### 8.1 Not connected to project + +``` +show business event services; +``` + +(Run without `-p` flag or before opening a project.) + +**Expected:** Error — not connected to a project. + +### 8.2 Service not found + +``` +describe business event service MyModule.DoesNotExist; +``` + +**Expected:** Error — service not found. + +### 8.3 Service already exists + +``` +create business event service MyModule.OrderEvents + event_name_prefix 'com.example.order' + message Msg (X: Integer) publish entity MyModule.MsgEvent; +create business event service MyModule.OrderEvents + event_name_prefix 'com.example.order' + message Msg (X: Integer) publish entity MyModule.MsgEvent; +``` + +**Expected:** First succeeds. Second returns error — already exists. + +### 8.4 Module not found + +``` +show business event services in FakeModule; +``` + +**Expected:** `No business event services found.` or error — module not found. + +### 8.5 Missing entity reference + +``` +create business event service MyModule.BadRef + event_name_prefix 'com.example.badref' + message Msg (X: Integer) publish entity MyModule.NonExistentEntity; +``` + +**Expected:** Error — entity not found. + +### 8.6 Invalid attribute type + +``` +create business event service MyModule.BadType + event_name_prefix 'com.example.badtype' + message Msg (X: InvalidType) publish entity MyModule.SomeEvent; +``` + +**Expected:** Error — unknown type. No service created. + +--- + +## Test Project Coverage Matrix + +| Operation | Lato Enquiry | Evora Factory | Lato Product | +|-----------|:---:|:---:|:---:| +| SHOW BUSINESS EVENT SERVICES | x | x | x | +| SHOW BUSINESS EVENTS | x | x | x | +| DESCRIBE BUSINESS EVENT SERVICE | x | x | | +| CREATE BUSINESS EVENT SERVICE | x | | | +| DROP BUSINESS EVENT SERVICE | x | | | + +Read operations tested on all projects. Write operations on copies of one project. + +--- + +## Automated Test Coverage + +| Section | Automated | Manual-only | +|---------|:---------:|:-----------:| +| 1. SHOW BUSINESS EVENT SERVICES | Mock tests | | +| 2. SHOW BUSINESS EVENTS | Mock tests | | +| 3. DESCRIBE BUSINESS EVENT SERVICE | Mock tests | | +| 4. CREATE BUSINESS EVENT SERVICE | Mock tests | Folder, export level | +| 5. DROP BUSINESS EVENT SERVICE | Mock tests | | +| 6. Roundtrip | Roundtrip tests | Complex services | +| 7. Multi-step | | All manual | +| 8. Failure modes | Partial | Edge cases | + +--- + +## Manual Test Report Template + +**Tester:** _______________ +**Date:** _______________ +**Project:** _______________ + +| # | Section | Test | Pass | Fail | Skip | Notes | +|---|---------|------|:----:|:----:|:----:|-------| +| 1.1 | SHOW BE SERVICES | List all | | | | | +| 1.2 | SHOW BE SERVICES | Filter by module | | | | | +| 1.3 | SHOW BE SERVICES | Empty result | | | | | +| 2.1 | SHOW BE | List all | | | | | +| 2.2 | SHOW BE | Filter by module | | | | | +| 2.3 | SHOW BE | Empty module | | | | | +| 3.1 | DESCRIBE BE SERVICE | Publish message | | | | | +| 3.2 | DESCRIBE BE SERVICE | Subscribe message | | | | | +| 3.3 | DESCRIBE BE SERVICE | Multiple messages | | | | | +| 3.4 | DESCRIBE BE SERVICE | All attribute types | | | | | +| 3.5 | DESCRIBE BE SERVICE | Not found | | | | | +| 4.1 | CREATE BE SERVICE | Minimal publish | | | | | +| 4.2 | CREATE BE SERVICE | Subscribe with microflow | | | | | +| 4.3 | CREATE BE SERVICE | Multiple messages | | | | | +| 4.4 | CREATE BE SERVICE | All attribute types | | | | | +| 4.5 | CREATE BE SERVICE | create or replace | | | | | +| 4.6 | CREATE BE SERVICE | in folder | | | | | +| 4.7 | CREATE BE SERVICE | Export level hidden | | | | | +| 4.8 | CREATE BE SERVICE | Duplicate error | | | | | +| 5.1 | DROP BE SERVICE | Existing | | | | | +| 5.2 | DROP BE SERVICE | Non-existent | | | | | +| 6.1 | ROUNDTRIP | Publish service | | | | | +| 6.2 | ROUNDTRIP | Subscribe service | | | | | +| 7.1 | MULTI-STEP | Entity + publish + verify | | | | | +| 7.2 | MULTI-STEP | Entity + subscribe + verify | | | | | +| 7.3 | MULTI-STEP | Create + replace + verify | | | | | +| 8.1 | FAILURE | Not connected | | | | | +| 8.2 | FAILURE | Service not found | | | | | +| 8.3 | FAILURE | Already exists | | | | | +| 8.4 | FAILURE | Module not found | | | | | +| 8.5 | FAILURE | Missing entity ref | | | | | +| 8.6 | FAILURE | Invalid attribute type | | | | | + +**Summary:** ___ / ___ passed | ___ failed | ___ skipped diff --git a/docs/15-testing/catalog-test-cases.md b/docs/15-testing/catalog-test-cases.md new file mode 100644 index 00000000..b7c05f91 --- /dev/null +++ b/docs/15-testing/catalog-test-cases.md @@ -0,0 +1,584 @@ +# Catalog & Code Navigation Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#386](https://github.com/mendixlabs/mxcli/pull/386) + +## Test Projects + +Demo apps from [Mendix App Gallery](https://appgallery.mendixcloud.com/): + +| App | Studio Pro | Modules | Entities | Microflows | +|-----|-----------|---------|----------|------------| +| Lato Enquiry Management | 11.4.0 | — | — | — | +| Evora - Factory Management | 11.8.0 | — | — | — | +| Lato Product Inventory | 11.2.0 | — | — | — | + +--- + +## Setup + +### 1. Download test apps + +1. Go to [Mendix App Gallery](https://appgallery.mendixcloud.com/) +2. Download each demo app listed above +3. Open each `.mpk` in Studio Pro to extract the `.mpr` file + +### 2. Build mxcli + +```bash +make build && make test && make lint-go +``` + +### 3. Build catalog (required for most tests) + +```bash +mxcli repl -p /EnquiriesManagement.mpr +``` + +Then inside REPL: +``` +refresh catalog full source; +``` + +This builds the complete catalog with all tables including activities, widgets, refs, strings, and MDL source. + +> **IMPORTANT:** Multi-step tests (§13) create and drop entities. Always run these against a **copy** of the project folder. +> +> ```bash +> cp -r MyProject MyProject-test +> mxcli repl -p MyProject-test/MyProject.mpr +> ``` + +--- + +## 1. REFRESH CATALOG + +### 1.1 Fast mode (default) + +``` +refresh catalog; +``` + +**Expected:** Builds core tables (modules, entities, attributes, microflows, etc.). Output shows per-table row count + "Catalog ready (Xs)". + +### 1.2 Full mode + +``` +refresh catalog full; +``` + +**Expected:** Adds activities, widgets, refs, strings, permissions tables. + +### 1.3 Full with source + +``` +refresh catalog full source; +``` + +**Expected:** Adds source table (MDL text for each document). + +### 1.4 Force rebuild + +``` +refresh catalog force; +``` + +**Expected:** Ignores cached catalog, rebuilds from scratch. + +### 1.5 Background build + +``` +refresh catalog background; +``` + +**Expected:** Returns immediately; catalog built in background goroutine. + +### 1.6 Caching behavior + +1. Run `refresh catalog full;` +2. Exit and re-open same project +3. Run `refresh catalog;` + +**Expected:** Second run uses cached `.mxcli/catalog.db` (fast — no rebuild needed). + +--- + +## 2. SHOW CATALOG TABLES + +### 2.1 List tables + +``` +show catalog tables; +``` + +**Expected:** Table with columns `Table | Count`. Lists 35+ tables including: modules, entities, attributes, microflows, nanoflows, pages, snippets, layouts, enumerations, activities, widgets, xpath_expressions, refs, permissions, workflows, odata_clients, odata_services, etc. + +### 2.2 Tables require appropriate build mode + +Verify tables marked as full-only show 0 rows if only fast catalog was built. + +--- + +## 3. SHOW CATALOG STATUS + +### 3.1 Catalog status + +``` +show catalog status; +``` + +**Expected:** Cache path, build mode, build time, build duration, Mendix version, validity status. + +--- + +## 4. DESCRIBE CATALOG TABLE + +### 4.1 Describe entities table + +``` +describe catalog.entities; +``` + +**Expected:** Column names, types, PK markers + required refresh mode. + +### 4.2 Describe activities table + +``` +describe catalog.activities; +``` + +**Expected:** Schema for full-mode table. + +--- + +## 5. SELECT FROM CATALOG + +### 5.1 Basic select + +``` +select * from catalog.entities where ModuleName = 'Administration'; +``` + +**Expected:** Row count + table result with entity data from Administration module. + +### 5.2 Select with multiple conditions + +``` +select QualifiedName, EntityType from catalog.entities where EntityType = 'PERSISTENT' and ModuleName = 'FactoryManagement'; +``` + +**Expected:** Filtered results. + +### 5.3 Aggregate query + +``` +select ModuleName, count(*) as cnt from catalog.entities group by ModuleName order by cnt desc; +``` + +**Expected:** Entity counts per module. + +### 5.4 Join across tables + +``` +select e.QualifiedName, count(a.Name) as attrs +from catalog.entities e +join catalog.attributes a on e.QualifiedName = a.EntityQualifiedName +group by e.QualifiedName +having attrs > 10; +``` + +**Expected:** Entities with more than 10 attributes. + +### 5.5 Full-only table without full build + +``` +refresh catalog; +select * from catalog.activities; +``` + +**Expected:** Warning about insufficient build mode. Empty or error. + +### 5.6 FTS search on strings table + +``` +select * from catalog.strings where strings match 'factory'; +``` + +**Expected:** Full-text search results with matching snippets. (Note: use a term present in your test project; `'factory'` works with Evora Factory Management.) + +--- + +## 6. SEARCH + +### 6.1 Basic search + +``` +search 'customer'; +``` + +**Expected:** String matches with columns: `QualifiedName | ObjectType | Match | StringContext | ModuleName`. Match shows `>>>...<<<` highlighting. + +### 6.2 Search with special characters + +``` +search 'user.name'; +``` + +**Expected:** Special chars (`.`, `/`, `-`, `:`) treated as spaces (AND semantics). + +### 6.3 Search requiring source mode + +If source table exists, additional source matches appear in output. + +### 6.4 Search with no results + +``` +search 'xyznonexistent123'; +``` + +**Expected:** Empty result. + +--- + +## 7. SHOW CALLERS + +### 7.1 Direct callers + +``` +show callers of Administration.ACT_CreateAccount; +``` + +**Expected:** Table with columns `Caller | Depth`. Shows microflows/pages that call this microflow. + +### 7.2 Transitive callers + +``` +show callers of Administration.ACT_CreateAccount transitive; +``` + +**Expected:** Recursive callers up to depth 10. + +### 7.3 No callers + +``` +show callers of ; +``` + +**Expected:** Empty result. + +--- + +## 8. SHOW CALLEES + +### 8.1 Direct callees + +``` +show callees of ; +``` + +**Expected:** Microflows called by this microflow. + +### 8.2 Transitive callees + +``` +show callees of transitive; +``` + +**Expected:** Recursive callees up to depth 10. + +--- + +## 9. SHOW REFERENCES + +### 9.1 Entity references + +``` +show references to Administration.Account; +``` + +**Expected:** Table with columns `SourceType | SourceName | RefKind`. Shows all documents referencing this entity. + +### 9.2 Microflow references + +``` +show references to ; +``` + +**Expected:** Documents referencing this microflow. + +--- + +## 10. SHOW IMPACT + +### 10.1 Entity impact + +``` +show impact of Administration.Account; +``` + +**Expected:** Summary (count by type) + detailed table of impacted documents. + +### 10.2 Microflow impact + +``` +show impact of ; +``` + +**Expected:** Summary + detail table. + +--- + +## 11. SHOW CONTEXT + +### 11.1 Microflow context + +``` +show context of Administration.ACT_CreateAccount; +``` + +**Expected:** Markdown output: definition (name, return, params, activities) + entities used + pages shown + called microflows + direct callers. + +### 11.2 Entity context + +``` +show context of Administration.Account; +``` + +**Expected:** Definition (type, generalization, attributes, indexes) + microflows using it + pages displaying it + related entities. + +> **Known issue (BUG-010):** Entity definition section is empty in `show context` output. + +### 11.3 Page context + +``` +show context of ; +``` + +**Expected:** Definition (title, URL, layout, widgets) + entities used + microflows called + shown by. + +### 11.4 Context with depth + +``` +show context of Administration.ACT_CreateAccount depth 3; +``` + +**Expected:** Deeper recursive resolution of called microflows. + +### 11.5 Non-existent document + +``` +show context of Fake.Missing; +``` + +**Expected:** Error or empty — document not found in catalog. + +--- + +## 12. SHOW STRUCTURE + +### 12.1 Default structure (depth 2) + +``` +show structure; +``` + +**Expected:** Module names + element type counts (entities, enums, microflows, etc.). Skips system/marketplace modules. Default depth is 2 (element names with signatures). + +> **Note:** To get depth-1 output (counts only), use `show structure depth 1` explicitly. + +### 12.2 Depth 2 + +``` +show structure depth 2; +``` + +**Expected:** Element names with signatures — entity attributes, microflow params → return, enum values. + +### 12.3 Depth 3 + +``` +show structure depth 3; +``` + +**Expected:** Attribute types, parameter names, association delete behavior, constant defaults. + +### 12.4 Module filter + +``` +show structure in Administration; +``` + +**Expected:** Only `Administration` module structure. + +> **Note:** Marketplace and system modules are filtered by default. Use `show structure in all` to include them, or test with a non-marketplace module. + +### 12.5 Include system modules + +``` +show structure all; +``` + +**Expected:** All modules including System, Atlas_Core, etc. + +--- + +## 13. MULTI-STEP WORKFLOWS + +### 13.1 Create entity → refresh → query catalog + +``` +create persistent entity MyModule.Indexed ( + Name: string(200) not null +); +refresh catalog; +select * from catalog.entities where QualifiedName = 'MyModule.Indexed'; +``` + +**Expected:** New entity appears in catalog after refresh. + +### 13.2 Impact analysis before drop + +``` +show impact of MyModule.Customer; +drop entity MyModule.Customer; +``` + +**Expected:** Impact shows affected documents before destructive operation. + +--- + +## 14. FAILURE MODES & ERROR RECOVERY + +### 14.1 Query without catalog + +Open project, do NOT refresh catalog: +``` +select * from catalog.entities; +``` + +**Expected:** mxcli auto-builds a fast catalog on demand and returns results. No manual `refresh catalog` required. + +### 14.2 Invalid SQL + +``` +select * from catalog.nonexistent; +``` + +**Expected:** SQL error — no such table. + +### 14.3 Search without full catalog + +``` +refresh catalog; +search 'test'; +``` + +**Expected:** mxcli auto-upgrades to a full catalog build on demand and returns search results. No manual `refresh catalog full` required. + +### 14.4 Callers without refs table + +``` +refresh catalog; +show callers of MyModule.Flow; +``` + +**Expected:** mxcli auto-upgrades to a full catalog build on demand and returns caller results. No manual `refresh catalog full` required. + +--- + +## Test Project Coverage Matrix + +| Operation | Lato Enquiry | Evora Factory | Lato Product | +|-----------|:---:|:---:|:---:| +| REFRESH CATALOG | x | x | x | +| SHOW CATALOG TABLES | x | | | +| SHOW CATALOG STATUS | x | | | +| DESCRIBE CATALOG | x | | | +| SELECT FROM CATALOG | x | x | x | +| SEARCH | x | x | | +| SHOW CALLERS | x | x | | +| SHOW CALLEES | x | x | | +| SHOW REFERENCES | x | x | | +| SHOW IMPACT | x | | | +| SHOW CONTEXT | x | x | | +| SHOW STRUCTURE | x | x | x | + +--- + +## Automated Test Coverage + +| Section | Automated | Manual-only | +|---------|:---------:|:-----------:| +| 1. REFRESH CATALOG | Unit tests | Caching, background | +| 2. SHOW TABLES | Unit tests | | +| 3. SHOW STATUS | | All manual | +| 4. DESCRIBE TABLE | | All manual | +| 5. SELECT | Unit tests | Complex queries | +| 6. SEARCH | Unit tests | Edge cases | +| 7. CALLERS | Unit tests | Transitive depth | +| 8. CALLEES | Unit tests | | +| 9. REFERENCES | Unit tests | | +| 10. IMPACT | Unit tests | | +| 11. CONTEXT | Unit tests | Deep depth | +| 12. STRUCTURE | Unit tests | | +| 13. Multi-step | | All manual | +| 14. Failure modes | Partial | | + +--- + +## Manual Test Report Template + +**Tester:** _______________ +**Date:** _______________ +**Project:** _______________ + +| # | Section | Test | Pass | Fail | Skip | Notes | +|---|---------|------|:----:|:----:|:----:|-------| +| 1.1 | REFRESH | Fast mode | | | | | +| 1.2 | REFRESH | Full mode | | | | | +| 1.3 | REFRESH | Full + source | | | | | +| 1.4 | REFRESH | Force | | | | | +| 1.5 | REFRESH | Background | | | | | +| 1.6 | REFRESH | Caching | | | | | +| 2.1 | TABLES | List tables | | | | | +| 2.2 | TABLES | Build mode check | | | | | +| 3.1 | STATUS | Show status | | | | | +| 4.1 | DESCRIBE | Entities table | | | | | +| 4.2 | DESCRIBE | Activities table | | | | | +| 5.1 | SELECT | Basic | | | | | +| 5.2 | SELECT | Multiple conditions | | | | | +| 5.3 | SELECT | Aggregate | | | | | +| 5.4 | SELECT | Join | | | | | +| 5.5 | SELECT | Full-only warning | | | | | +| 5.6 | SELECT | FTS search | | | | | +| 6.1 | SEARCH | Basic | | | | | +| 6.2 | SEARCH | Special chars | | | | | +| 6.3 | SEARCH | Source mode | | | | | +| 6.4 | SEARCH | No results | | | | | +| 7.1 | CALLERS | Direct | | | | | +| 7.2 | CALLERS | Transitive | | | | | +| 7.3 | CALLERS | None | | | | | +| 8.1 | CALLEES | Direct | | | | | +| 8.2 | CALLEES | Transitive | | | | | +| 9.1 | REFERENCES | Entity | | | | | +| 9.2 | REFERENCES | Microflow | | | | | +| 10.1 | IMPACT | Entity | | | | | +| 10.2 | IMPACT | Microflow | | | | | +| 11.1 | CONTEXT | Microflow | | | | | +| 11.2 | CONTEXT | Entity | | | | | +| 11.3 | CONTEXT | Page | | | | | +| 11.4 | CONTEXT | Depth 3 | | | | | +| 11.5 | CONTEXT | Not found | | | | | +| 12.1 | STRUCTURE | Depth 1 | | | | | +| 12.2 | STRUCTURE | Depth 2 | | | | | +| 12.3 | STRUCTURE | Depth 3 | | | | | +| 12.4 | STRUCTURE | Module filter | | | | | +| 12.5 | STRUCTURE | All modules | | | | | +| 13.1 | MULTI-STEP | Create + refresh + query | | | | | +| 13.2 | MULTI-STEP | Impact before drop | | | | | +| 14.1 | FAILURE | No catalog | | | | | +| 14.2 | FAILURE | Invalid SQL | | | | | +| 14.3 | FAILURE | Search no full | | | | | +| 14.4 | FAILURE | Callers no full | | | | | + +**Summary:** ___ / ___ passed | ___ failed | ___ skipped diff --git a/docs/15-testing/cli-commands-test-cases.md b/docs/15-testing/cli-commands-test-cases.md new file mode 100644 index 00000000..cd0c7824 --- /dev/null +++ b/docs/15-testing/cli-commands-test-cases.md @@ -0,0 +1,755 @@ +# CLI Commands Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#386](https://github.com/mendixlabs/mxcli/pull/386) + +Covers: docker, test, playwright, eval, init, setup, auth, marketplace, widget, bson, fmt, new, serve, lsp, tui, check, report. + +## Test Projects + +Demo apps from [Mendix App Gallery](https://appgallery.mendixcloud.com/): + +| App | Studio Pro | CLI features exercised | +|-----|-----------|------------------------| +| Lato Enquiry Management | 11.4.0 | docker, test, check, bson, fmt | +| Evora - Factory Management | 10.24.15 | docker, playwright, report, widget | +| Lato Product Inventory | 11.2.0 | test, check, bson, lsp | + +--- + +## Setup + +### 1. Build mxcli + +```bash +make build && make test && make lint-go +``` + +### 2. Ensure Docker + +```bash +docker info +``` + +Docker Desktop must be running. Commands `docker`, `test`, and `playwright` require it. + +### 3. Set environment + +```bash +export APPS_DIR= +export MPR="$APPS_DIR/EnquiriesManagement/EnquiriesManagement.mpr" +``` + +> **IMPORTANT:** Commands `docker`, `test`, `fmt`, and `new` modify project files. Always run against a **copy** of the project folder. +> +> ```bash +> cp -r MyProject MyProject-test +> export MPR="MyProject-test/MyProject.mpr" +> ``` + +--- + +## 1. mxcli docker + +### 1.1 Init Docker configuration + +```bash +mxcli docker init -p "$MPR" +``` + +**Expected:** Creates `.docker/docker-compose.yml`, `.env.example`, and `.env` in the project directory. Exit code `0`. + +### 1.2 Build Docker image + +```bash +mxcli docker build -p "$MPR" +``` + +**Expected:** Builds image from generated Dockerfile. Exit code `0`. + +### 1.3 Run container + +```bash +mxcli docker run -p "$MPR" --port 8080 --admin-port 8090 +``` + +**Expected:** Starts Mendix runtime container. App reachable at `http://localhost:8080`. Admin at `http://localhost:8090`. Exit code `0`. + +### 1.4 Docker Compose up + +```bash +mxcli docker up -p "$MPR" +``` + +**Expected:** Starts services via `docker-compose up -d`. Exit code `0`. + +### 1.5 Docker Compose down + +```bash +mxcli docker down -p "$MPR" +``` + +**Expected:** Stops and removes containers. Exit code `0`. + +### 1.6 Run MxBuild project validation + +```bash +mxcli docker check -p "$MPR" +``` + +**Expected:** Runs `mx check` (MxBuild project validation) inside Docker. Prints errors and warnings. Exit code `0` if no errors, `1` if errors found. + +### 1.7 View logs + +```bash +mxcli docker logs -p "$MPR" +``` + +**Expected:** Streams container logs to stdout. Exit code `0`. + +### 1.8 Container status + +```bash +mxcli docker status -p "$MPR" +``` + +**Expected:** Prints container state, ports, uptime. Exit code `0`. + +### 1.9 Open shell + +```bash +mxcli docker shell -p "$MPR" +``` + +**Expected:** Opens interactive shell inside the running container. + +### 1.10 Reload runtime + +```bash +mxcli docker reload -p "$MPR" +``` + +**Expected:** Rebuilds and restarts the runtime without recreating the container. Exit code `0`. + +--- + +## 2. mxcli test + +### 2.1 Run all tests in directory + +```bash +mxcli test tests/ -p "$MPR" +``` + +**Expected:** Discovers `.test.mdl` and `.test.md` files. Injects TestRunner microflow, builds via MxBuild, runs in Docker. Prints pass/fail summary. Exit code `0` if all pass, `1` if any fail. + +### 2.2 Run single test file + +```bash +mxcli test tests/my-flow.test.mdl -p "$MPR" +``` + +**Expected:** Runs only the specified test file. Exit code `0` on pass. + +### 2.3 List tests without running + +```bash +mxcli test tests/ -p "$MPR" --list +``` + +**Expected:** Prints test names from `@test`/`@expect` annotations. Does not build or run. Exit code `0`. + +### 2.4 JUnit output + +```bash +mxcli test tests/ -p "$MPR" --junit report.xml +``` + +**Expected:** Writes JUnit XML to `report.xml`. Exit code reflects pass/fail. + +### 2.5 Skip build + +```bash +mxcli test tests/ -p "$MPR" --skip-build +``` + +**Expected:** Skips MxBuild step. Uses existing build artifacts. Exit code `0` on pass. + +### 2.6 Verbose output + +```bash +mxcli test tests/ -p "$MPR" --verbose +``` + +**Expected:** Prints detailed output per test case including microflow execution logs. + +### 2.7 Custom timeout + +```bash +mxcli test tests/ -p "$MPR" --timeout 120 +``` + +**Expected:** Fails tests that exceed 120 seconds. Exit code `1` on timeout. + +### 2.8 Color output + +```bash +mxcli test tests/ -p "$MPR" --color +``` + +**Expected:** Forces colored output even when piped. Green for pass, red for fail. + +--- + +## 3. mxcli playwright verify + +### 3.1 Run verification scripts + +```bash +mxcli playwright verify tests/e2e/ -p "$MPR" +``` + +**Expected:** Discovers `.test.sh` scripts. Launches app in Docker, runs each script. Captures screenshots on failure. Exit code `0` if all pass. + +### 3.2 List verification scripts + +```bash +mxcli playwright verify tests/e2e/ -p "$MPR" --list +``` + +**Expected:** Prints discovered `.test.sh` filenames. Does not run. Exit code `0`. + +### 3.3 JUnit output + +```bash +mxcli playwright verify tests/e2e/ -p "$MPR" --junit pw-report.xml +``` + +**Expected:** Writes JUnit XML. Screenshots attached as test artifacts. + +### 3.4 Verbose mode + +```bash +mxcli playwright verify tests/e2e/ -p "$MPR" --verbose +``` + +**Expected:** Prints browser console logs and script output per test. + +--- + +## 4. mxcli eval + +### 4.1 Check evaluation results + +```bash +mxcli eval check +``` + +**Expected:** Runs AI correctness evaluation. Prints pass/fail per check. Exit code `0` if all pass. + +### 4.2 List evaluations + +```bash +mxcli eval list +``` + +**Expected:** Lists available evaluation suites. Exit code `0`. + +--- + +## 5. mxcli init + +### 5.1 Scaffold AI/IDE config + +```bash +mxcli init -p "$MPR" +``` + +**Expected:** Creates `CLAUDE.md`, `.vscode/settings.json`, and other AI config files in the project directory. Exit code `0`. + +### 5.2 Init in empty directory + +```bash +mkdir /tmp/empty-project && mxcli init -p /tmp/empty-project +``` + +**Expected:** Exit code `0`. The `-p` flag does not validate that the path contains an `.mpr` file — command succeeds regardless. **Known behavior:** no MPR validation on init. + +--- + +## 6. mxcli setup + +### 6.1 Setup MxBuild + +```bash +mxcli setup mxbuild +``` + +**Expected:** Downloads and installs MxBuild for the project's Studio Pro version. Exit code `0`. + +### 6.2 Setup MxRuntime + +```bash +mxcli setup mxruntime +``` + +**Expected:** Downloads and installs MxRuntime. Exit code `0`. + +### 6.3 Setup mxcli itself + +```bash +mxcli setup mxcli +``` + +**Expected:** Updates mxcli to latest version. Exit code `0`. + +--- + +## 7. mxcli auth + +### 7.1 Login + +```bash +mxcli auth login +``` + +**Expected:** Opens browser for Mendix SSO. Stores token locally. Prints "Logged in as ". Exit code `0`. + +### 7.2 Check status + +```bash +mxcli auth status +``` + +**Expected:** Prints current auth state (logged in/out, username, token expiry). Exit code `0` if authenticated, `1` if not authenticated. + +### 7.3 List accounts + +```bash +mxcli auth list +``` + +**Expected:** Lists stored credentials. Exit code `0`. + +### 7.4 Logout + +```bash +mxcli auth logout +``` + +**Expected:** Removes stored token. Prints confirmation. Exit code `0`. + +### 7.5 Status after logout + +```bash +mxcli auth logout && mxcli auth status +``` + +**Expected:** Status reports "Not logged in". Exit code `1`. + +--- + +## 8. mxcli marketplace + +> **Prerequisite:** Requires authentication (`mxcli auth login`). Commands fail with "no credential" error without auth. + +### 8.1 Search modules + +```bash +mxcli marketplace search "email" +``` + +**Expected:** Prints matching marketplace modules with name, version, compatibility. Exit code `0`. + +### 8.2 Module info + +```bash +mxcli marketplace info "Email Connector" +``` + +**Expected:** Prints module details: description, author, latest version, supported Studio Pro versions. Exit code `0`. + +### 8.3 List versions + +```bash +mxcli marketplace versions "Email Connector" +``` + +**Expected:** Prints version history with release dates. Exit code `0`. + +### 8.4 Search with no results + +```bash +mxcli marketplace search "zzzznonexistent" +``` + +**Expected:** Prints "No results found." Exit code `0`. + +--- + +## 9. mxcli widget + +### 9.1 List widgets + +```bash +mxcli widget list -p "$MPR" +``` + +**Expected:** Lists all custom widgets in the project with name and version. Exit code `0`. + +### 9.2 Extract widget + +```bash +mxcli widget extract --mpk path/to/widget.mpk +``` + +**Expected:** Extracts widget package contents. Exit code `0`. + +### 9.3 Init new widget + +```bash +mxcli widget init MyNewWidget +``` + +**Expected:** Scaffolds widget project structure. Exit code `0`. + +### 9.4 Generate widget docs + +```bash +mxcli widget docs -p "$MPR" +``` + +**Expected:** Generates documentation for all custom widgets. Exit code `0`. + +--- + +## 10. mxcli bson + +### 10.1 Dump MPR contents + +```bash +mxcli bson dump -p "$MPR" --list +mxcli bson dump -p "$MPR" --object "Name" +``` + +**Expected:** `--list` prints object names in the MPR. `--object` dumps a specific named object. There is no simple "dump all" mode. Exit code `0`. + +### 10.2 Discover document types + +```bash +mxcli bson discover "$MPR" +``` + +**Expected:** Lists all document types and counts found in the MPR. Exit code `0`. + +### 10.3 Compare objects within or across projects + +```bash +# Compare objects within the same project by type +mxcli bson compare -p "$MPR" --type DomainModels$DomainModel + +# Compare across two projects +mxcli bson compare -p "$MPR" --p2 "$APPS_DIR/FactoryManagement/FactoryManagement.mpr" +``` + +**Expected:** Prints structural diff. Exit code `0` if identical, `1` if different. + +### 10.4 Dump nonexistent file + +```bash +mxcli bson dump /tmp/nonexistent.mpr +``` + +**Expected:** Error: file not found. Exit code `1`. + +--- + +## 11. mxcli fmt + +### 11.1 Format MDL file + +```bash +mxcli fmt myfile.mdl +``` + +**Expected:** Outputs formatted MDL to stdout (does NOT write in-place). Exit code `0`. + +> **Known issue (BUG-012):** No `-w` flag for in-place write. Output goes to stdout only. + +### 11.2 Format already-formatted file + +```bash +mxcli fmt myfile.mdl +``` + +**Expected:** No changes. Exit code `0`. + +### 11.3 Format invalid file + +```bash +mxcli fmt broken.txt +``` + +**Expected:** Exit code `0`. Input is echoed unchanged to stdout (no parse error). + +> **Known issue (BUG-012):** No `-w` flag, so invalid input passes through without error. + +--- + +## 12. mxcli new + +### 12.1 Create a new Mendix project + +```bash +mxcli new my-app --version 10.24.15 +``` + +**Expected:** Scaffolds a complete Mendix project named `my-app` for the specified Studio Pro version. Exit code `0`. + +### 12.2 Create project without version + +```bash +mxcli new my-app +``` + +**Expected:** Creates project using the latest available Studio Pro version. Exit code `0`. + +--- + +## 13. mxcli serve + +### 13.1 Start HTTP server + +```bash +mxcli serve -p "$MPR" --port 9000 & +curl -s http://localhost:9000/ +kill %1 +``` + +**Expected:** Server starts and serves project visualization at root `/`. Exit code `0` on clean shutdown. + +### 13.2 Port conflict + +```bash +mxcli serve -p "$MPR" --port 9000 & +mxcli serve -p "$MPR" --port 9000 +``` + +**Expected:** Second instance fails with "port already in use". Exit code `1`. + +--- + +## 14. mxcli lsp + +### 14.1 Start LSP server + +```bash +echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' | mxcli lsp +``` + +**Expected:** Returns JSON-RPC `initialize` response with server capabilities (completion, hover, diagnostics, documentSymbol, foldingRange). Exit code `0`. + +### 14.2 Verify capabilities + +Inspect the `initialize` response `capabilities` object: + +| Capability | Expected | +|------------|----------| +| `completionProvider` | Present | +| `hoverProvider` | `true` | +| `diagnosticProvider` | Present | +| `documentSymbolProvider` | `true` | +| `foldingRangeProvider` | `true` | + +--- + +## 15. mxcli tui + +### 15.1 Launch TUI + +```bash +mxcli tui -p "$MPR" +``` + +**Expected:** Opens terminal UI explorer. Displays project tree. Arrow keys navigate. `q` exits. Exit code `0`. + +### 15.2 TUI without project + +```bash +mxcli tui +``` + +**Expected:** Error: `-p` flag required. Exit code `1`. + +--- + +## 16. mxcli check + +### 16.1 Validate MDL script syntax + +```bash +mxcli check script.mdl +``` + +**Expected:** Validates MDL script syntax (not the project itself — use `docker check` for project validation). Prints errors and warnings. Exit code `0` if no errors, `1` if errors found. + +### 16.2 SARIF output + +```bash +mxcli check script.mdl --format sarif > results.sarif +``` + +**Expected:** Writes SARIF JSON to stdout. Exit code reflects validation result. + +### 16.3 Check nonexistent file + +```bash +mxcli check /tmp/nonexistent.mdl +``` + +**Expected:** Error: file not found. Exit code `1`. + +--- + +## 17. mxcli report + +### 17.1 Markdown report + +```bash +mxcli report -p "$MPR" --format markdown +``` + +**Expected:** Prints project report in Markdown to stdout. Exit code `0`. + +### 17.2 JSON report + +```bash +mxcli report -p "$MPR" --format json +``` + +**Expected:** Prints valid JSON report. Exit code `0`. + +> **Known issue (BUG-011):** `--format json` hangs after catalog load. + +### 17.3 HTML report + +```bash +mxcli report -p "$MPR" --format html -o report.html +``` + +**Expected:** Writes HTML report to `report.html`. Exit code `0`. + +> **Known issue (BUG-011):** `--format html` hangs after catalog load. + +--- + +## Failure Modes + +| Scenario | Command | Expected error | Exit code | +|----------|---------|---------------|-----------| +| Missing `-p` flag | `mxcli test tests/` | "required flag: -p" | `1` | +| Invalid MPR path | `mxcli check -p /bad/path.mpr` | "file not found" | `1` | +| Docker not running | `mxcli docker run -p "$MPR"` | "cannot connect to Docker daemon" | `1` | +| MxBuild not installed | `mxcli check -p "$MPR"` | "MxBuild not found — run mxcli setup mxbuild" | `1` | +| Auth token expired | `mxcli marketplace search "x"` | "token expired — run mxcli auth login" | `1` | +| Port conflict | `mxcli serve --port ` | "address already in use" | `1` | +| Invalid test file | `mxcli test bad.txt -p "$MPR"` | "unsupported test file format" | `1` | +| Timeout exceeded | `mxcli test --timeout 1` | "test timed out after 1s" | `1` | +| BSON corrupt file | `mxcli bson dump corrupt.mpr` | "invalid BSON data" | `1` | + +> **Note:** Several commands return exit `0` on error instead of `1`: `bson dump` (missing args), `docker build` (build failure), `docker check` (in some failure modes), `marketplace search` (auth failure), `serve` (port conflict), `fmt` (invalid input). Do not rely solely on exit codes for pass/fail in automation. + +--- + +## Test Project Coverage Matrix + +| Command | Subcommands | Happy path | Error path | Flags tested | +|---------|-------------|:----------:|:----------:|:------------:| +| `docker` | run, build, check, init, up, down, logs, status, shell, reload | 10 | 1 | -p, --port, --admin-port | +| `test` | — | 8 | 2 | -p, --list, --junit, --skip-build, --verbose, --color, --timeout | +| `playwright verify` | — | 4 | 0 | -p, --list, --junit, --verbose | +| `eval` | check, list | 2 | 0 | — | +| `init` | — | 1 | 1 | -p | +| `setup` | mxbuild, mxruntime, mxcli | 3 | 0 | — | +| `auth` | login, logout, status, list | 5 | 0 | — | +| `marketplace` | search, info, versions | 4 | 0 | — | +| `widget` | extract, list, init, docs | 4 | 0 | -p | +| `bson` | dump, discover, compare | 3 | 2 | — | +| `fmt` | — | 2 | 1 | — | +| `new` | — | 2 | 0 | --version | +| `serve` | — | 1 | 1 | -p, --port | +| `lsp` | — | 2 | 0 | — | +| `tui` | — | 1 | 1 | -p | +| `check` | — | 2 | 1 | --format | +| `report` | — | 3 | 0 | -p, --format, -o | + +**Total: 56 happy-path + 11 error-path = 67 test cases** + +--- + +## Manual Test Report Template + +**Tester:** _______________ +**Date:** _______________ +**mxcli version:** _______________ + +| # | Command | Case | Pass | Fail | Skip | Notes | +|---|---------|------|:----:|:----:|:----:|-------| +| 1.1 | `docker init` | Init Docker config | | | | | +| 1.2 | `docker build` | Build image | | | | | +| 1.3 | `docker run` | Run container | | | | | +| 1.4 | `docker check` | Runtime check | | | | | +| 1.5 | `docker up` | Compose up | | | | | +| 1.6 | `docker down` | Compose down | | | | | +| 1.7 | `docker logs` | View logs | | | | | +| 1.8 | `docker status` | Container status | | | | | +| 1.9 | `docker shell` | Interactive shell | | | | | +| 1.10 | `docker reload` | Hot reload | | | | | +| 2.1 | `test` | Run all tests | | | | | +| 2.2 | `test` | Single file | | | | | +| 2.3 | `test --list` | List tests | | | | | +| 2.4 | `test --junit` | JUnit output | | | | | +| 2.5 | `test --skip-build` | Skip build | | | | | +| 2.6 | `test --verbose` | Verbose output | | | | | +| 2.7 | `test --color` | Colored output | | | | | +| 2.8 | `test --timeout` | Custom timeout | | | | | +| 3.1 | `playwright` | Run scripts | | | | | +| 3.2 | `playwright --list` | List scripts | | | | | +| 3.3 | `playwright --junit` | JUnit output | | | | | +| 3.4 | `playwright --verbose` | Verbose output | | | | | +| 4.1 | `eval check` | Check correctness | | | | | +| 4.2 | `eval list` | List evaluations | | | | | +| 5.1 | `init` | Init AI config | | | | | +| 6.1 | `setup mxbuild` | Install MxBuild | | | | | +| 6.2 | `setup mxruntime` | Install runtime | | | | | +| 6.3 | `setup mxcli` | Self-update | | | | | +| 7.1 | `auth login` | Login flow | | | | | +| 7.2 | `auth logout` | Logout | | | | | +| 7.3 | `auth status` | Auth status | | | | | +| 7.4 | `auth list` | List accounts | | | | | +| 7.5 | `auth` | Token refresh | | | | | +| 8.1 | `marketplace search` | Search modules | | | | | +| 8.2 | `marketplace info` | Module info | | | | | +| 8.3 | `marketplace versions` | List versions | | | | | +| 8.4 | `marketplace` | Install module | | | | | +| 9.1 | `widget extract` | Extract widget | | | | | +| 9.2 | `widget list` | List widgets | | | | | +| 9.3 | `widget init` | Init widget project | | | | | +| 9.4 | `widget docs` | Generate docs | | | | | +| 10.1 | `bson dump` | Dump MPR | | | | | +| 10.2 | `bson discover` | Discover structure | | | | | +| 10.3 | `bson compare` | Compare MPRs | | | | | +| 11.1 | `fmt` | Format file | | | | | +| 11.2 | `fmt` | Format directory | | | | | +| 12.1 | `new` | Create project | | | | | +| 13.1 | `serve` | Start server | | | | | +| 14.1 | `lsp` | Start LSP | | | | | +| 14.2 | `lsp` | Diagnostics | | | | | +| 15.1 | `tui` | Terminal UI | | | | | +| 16.1 | `check` | Validate project | | | | | +| 16.2 | `check --sarif` | SARIF output | | | | | +| 17.1 | `report` | Markdown report | | | | | +| 17.2 | `report --format json` | JSON report | | | | | +| 17.3 | `report -o` | Output to file | | | | | + +**Summary:** ___ / ___ passed | ___ failed | ___ skipped diff --git a/docs/15-testing/entity-test-cases.md b/docs/15-testing/entity-test-cases.md new file mode 100644 index 00000000..99303fe2 --- /dev/null +++ b/docs/15-testing/entity-test-cases.md @@ -0,0 +1,1401 @@ +# Entity, Association & Constant Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#386](https://github.com/mendixlabs/mxcli/pull/386) + +## Test Projects + +Demo apps from [Mendix App Gallery](https://appgallery.mendixcloud.com/): + +| App | Studio Pro | Entities | Associations | Constants | +|-----|-----------|----------|--------------|-----------| +| Lato Enquiry Management | 11.4.0 | — | — | — | +| Evora - Factory Management | 10.24.15 | — | — | — | +| Lato Product Inventory | 11.2.0 | — | — | — | + +--- + +## Setup + +> **Note:** `MyModule` will be auto-created on first CREATE statement — no manual setup needed. + +### 1. Download test apps + +1. Go to [Mendix App Gallery](https://appgallery.mendixcloud.com/) +2. Download each demo app listed above +3. Open each `.mpk` in Studio Pro to extract the `.mpr` file + +### 2. Build mxcli + +```bash +make build && make test && make lint-go +``` + +### 3. Smoke test + +```bash +APPS_DIR= +for mpr in "$APPS_DIR"/*/*.mpr; do + echo "=== $(basename $(dirname $mpr)) ===" + echo "show entities;" > /tmp/show-ent.mdl + mxcli exec /tmp/show-ent.mdl -p "$mpr" 2>&1 | tail -1 +done +``` + +Expected: count line `(N entities)` for each project. + +### 4. Interactive testing + +```bash +mxcli repl -p /EnquiriesManagement.mpr +``` + +### 5. Script-based testing + +```bash +mxcli exec test-sequence.mdl -p +``` + +Write operations (CREATE, ALTER, DROP) modify the `.mpr` file **in place**. + +> **IMPORTANT:** Always run destructive tests against a **copy** of the project folder, +> never the original. The `.mpr` file references other files in the project directory. +> Dropped entities, associations, and constants cannot be recovered — there is no undo. +> +> ```bash +> # Before each destructive test session +> cp -r MyProject MyProject-test +> mxcli repl -p MyProject-test/MyProject.mpr +> ``` + +--- + +## 1. SHOW ENTITIES + +### 1.1 List all entities + +``` +show entities; +``` + +**Expected:** Table with columns `Entity | Type | Attrs | Assocs | Validations | Indexes | Events | AccessRules`. Summary line `(N entities)`. Sorted alphabetically. + +### 1.2 List entities in a module + +``` +show entities in Administration; +``` + +**Expected:** Only entities from `Administration` module. Same column format. + +### 1.3 Generalization column + +``` +show entities; +``` + +**Expected:** If any entity has a generalization, an `Extends` column appears between `Type` and `Attrs`. System entities (referenced via generalization) show `-` for counts. + +### 1.4 Empty module + +``` +show entities in NonExistentModule; +``` + +**Expected:** Error or empty result with `(0 entities)`. + +### 1.5 Entity types + +Verify `Type` column values across projects: + +| Type | Example | +|------|---------| +| Persistent | Most domain entities | +| Non-Persistent | Transient/helper entities | +| External | OData/external entities | +| System | System.User, System.Session | + +--- + +## 2. SHOW ENTITY (single) + +### 2.1 Persistent entity + +``` +show entity Administration.Account; +``` + +**Expected:** Markdown-style output: +``` +**Entity: Administration.Account** + +- Persistable: true +- Extends: System.User + +| Attribute | Type | +| --------- | ----------- | +| ... | ... | + +(N attributes) +``` + +### 2.2 Non-persistent entity + +``` +show entity ; +``` + +**Expected:** `Persistable: false`. No `Extends` if no generalization. + +### 2.3 Non-existent entity + +``` +show entity Fake.Entity; +``` + +**Expected:** Error message (not found). + +--- + +## 3. DESCRIBE ENTITY + +### 3.1 Basic entity + +``` +describe entity Administration.Account; +``` + +**Expected:** Roundtrippable MDL output: +``` +create or modify persistent entity Administration.Account extends System.User ( + FullName: string(200), + IsLocalUser: boolean, + ... +) +/ +``` + +### 3.2 Entity with all features + +Find an entity with indexes, events, validation rules, and access rules. Verify output includes: +- `/** documentation */` block (if documented) +- `@Position(x, y)` annotation +- Attribute types and constraints (`not null error '...'`, `unique error '...'`) +- System pseudo-attributes (`Owner: AutoOwner`, etc.) +- `index (col1, col2 desc)` lines +- `on before|after create|commit|delete|rollback call Module.Mf($currentObject) [raise error]` events +- `grant Role1, Role2 on Module.Entity (create, delete, read *, write (Col1, Col2)) [where 'xpath']` access rules + +### 3.3 View entity + +``` +describe entity ; +``` + +**Expected:** Output ends with `as (` followed by indented OQL query, then `)`. + +### 3.4 Non-existent entity + +``` +describe entity Fake.Missing; +``` + +**Expected:** Error message (not found). + +--- + +## 4. CREATE ENTITY + +### 4.1 Minimal persistent entity + +``` +create persistent entity MyModule.SimpleTag ( + Name: string(50) +); +``` + +**Expected:** Entity created. `show entity MyModule.SimpleTag` shows 1 attribute. + +### 4.2 All attribute types + +``` +create persistent entity MyModule.AllTypes ( + ProductId: autonumber default 1, + Name: string(200) not null error 'Name is required', + Code: string(50) unique error 'Code must be unique', + Price: decimal not null error 'Price required' default 0.00, + Stock: integer default 0, + IsActive: boolean default true, + ReleaseDate: date, + CreatedAt: datetime, + ProductImage: binary, + Status: enumeration(MyModule.Status) +); +``` + +**Expected:** Entity created with 10 attributes. Types and constraints preserved in `describe`. + +> **Note:** `default 0.00` is normalized to `default 0` for decimals. + +### 4.3 Entity with generalization + +``` +create persistent entity MyModule.SpecialUser extends System.User ( + Department: string(100) +); +``` + +**Expected:** Entity created. `show entity` shows `Extends: System.User`. + +### 4.4 Non-persistent entity + +``` +create non-persistent entity MyModule.SearchParams ( + SearchTerm: string(200), + IncludeInactive: boolean default false +); +``` + +**Expected:** Entity created. `Persistable: false`. + +### 4.5 Entity with indexes + +``` +create persistent entity MyModule.Order ( + OrderNumber: string(50) not null unique, + CustomerId: integer, + OrderDate: datetime +) +index (OrderNumber), +index (CustomerId asc), +index (OrderDate desc); +``` + +**Expected:** Entity created. `describe` shows 3 index definitions. + +### 4.6 Entity with event handlers + +``` +create persistent entity MyModule.BankAccount ( + IBAN: string(34) not null +) +on before commit call MyModule.ACT_ValidateIBAN($currentObject) raise error; +``` + +**Expected:** Entity created. `describe` shows event handler. + +> **Prerequisite:** Event handlers and calculated-by microflows require the target microflow to exist. Create the microflow before the entity/alter statement. + +### 4.7 Entity with system attributes + +``` +create or modify persistent entity MyModule.AuditedRecord ( + Title: string(200) not null, + Owner: autoowner, + ChangedBy: autochangedby, + CreatedDate: autocreateddate, + ChangedDate: autochangeddate +); +``` + +**Expected:** Entity created with system attributes. `describe` shows pseudo-attributes. + +### 4.8 `create or modify` — new entity + +``` +create or modify persistent entity MyModule.Fresh ( + Value: integer +); +``` + +**Expected:** Creates entity (same as without `or modify`). + +### 4.9 `create or modify` — existing entity + +``` +create or modify persistent entity MyModule.Fresh ( + Value: integer, + Extra: string(100) +); +``` + +**Expected:** Updates entity — adds `Extra` attribute, preserves entity ID. + +### 4.10 Duplicate entity (without `or modify`) + +``` +create persistent entity MyModule.SimpleTag ( + Name: string(50) +); +``` + +**Expected:** Error — entity already exists. + +### 4.11 View entity with OQL + +``` +create view entity MyModule.ActiveProducts ( + Name: string(200), + Price: decimal +) as + select p.Name as Name, p.Price as Price + from MyModule.Product as p + where p.IsActive = true; +``` + +**Expected:** View entity created. `describe` shows OQL source. + +### 4.12 `create or replace` view entity + +``` +create or replace view entity MyModule.ActiveProducts ( + Name: string(200), + Price: decimal, + Stock: integer +) as + select p.Name, p.Price, p.Stock + from MyModule.Product as p + where p.IsActive = true; +``` + +**Expected:** Drops and recreates view entity (new ID). + +### 4.13 Position annotation + +``` +@position(300, 400) +create persistent entity MyModule.Positioned ( + Label: string(100) +); +``` + +**Expected:** Entity created at position (300, 400). `describe` shows `@Position(300, 400)`. + +--- + +## 5. ALTER ENTITY + +### 5.1 Add attribute + +``` +alter entity MyModule.SimpleTag add attribute Description: string(500); +``` + +**Expected:** Attribute added. `describe` shows both `Name` and `Description`. + +### 5.2 Rename attribute + +``` +alter entity MyModule.SimpleTag rename attribute Name to TagName; +``` + +**Expected:** Attribute renamed. `describe` shows `TagName`. + +### 5.3 Modify attribute type + +``` +alter entity MyModule.SimpleTag modify attribute TagName: string(200); +``` + +**Expected:** Attribute type updated to `string(200)`. + +### 5.4 Drop attribute + +``` +alter entity MyModule.SimpleTag drop attribute Description; +``` + +**Expected:** Attribute removed. Validation rules, index refs, and MemberAccess entries cleaned up. + +### 5.5 Set documentation + +``` +alter entity MyModule.SimpleTag set documentation 'A simple tag entity for testing'; +``` + +**Expected:** `describe` shows `/** A simple tag entity for testing */`. + +### 5.6 Set position + +``` +alter entity MyModule.SimpleTag set position (150, 250); +``` + +**Expected:** `describe` shows `@Position(150, 250)`. + +### 5.7 Add index + +``` +alter entity MyModule.SimpleTag add index (TagName); +``` + +**Expected:** `describe` shows `index (TagName)`. + +### 5.8 Drop index + +``` +alter entity MyModule.SimpleTag drop index idx1; +``` + +**Expected:** Index removed. + +### 5.9 Add event handler + +``` +alter entity MyModule.SimpleTag + add event handler on before commit call MyModule.ACT_ValidateTag($currentObject) raise error; +``` + +**Expected:** Event handler added to entity. + +> **Prerequisite:** Event handlers and calculated-by microflows require the target microflow to exist. Create the microflow before the entity/alter statement. + +### 5.10 Drop event handler + +``` +alter entity MyModule.SimpleTag drop event handler on before commit; +``` + +**Expected:** Event handler removed. + +### 5.11 Multiple actions + +``` +alter entity MyModule.SimpleTag + add attribute Priority: integer default 0 + add index (Priority) + set documentation 'Updated tag entity'; +``` + +**Expected:** All three changes applied in one statement. + +### 5.12 Drop system pseudo-attribute + +``` +alter entity MyModule.AuditedRecord drop attribute owner; +``` + +**Expected:** Clears `HasOwner` flag on entity. `describe` no longer shows `Owner: AutoOwner`. + +### 5.13 Make attribute calculated + +``` +alter entity MyModule.SimpleTag modify attribute TagName: string(200) calculated by MyModule.CalcTagName; +``` + +**Expected:** Attribute marked as calculated. `describe` shows `calculated by MyModule.CalcTagName`. + +> **Prerequisite:** Event handlers and calculated-by microflows require the target microflow to exist. Create the microflow before the entity/alter statement. + +--- + +## 6. DROP ENTITY + +### 6.1 Drop existing entity + +``` +drop entity MyModule.SimpleTag; +``` + +**Expected:** Entity removed. `show entity MyModule.SimpleTag` returns error. + +### 6.2 Drop non-existent entity + +``` +drop entity MyModule.NonExistent; +``` + +**Expected:** Error — entity not found. + +### 6.3 Drop view entity + +``` +drop entity MyModule.ActiveProducts; +``` + +**Expected:** View entity and its ViewEntitySourceDocument removed. + +### 6.4 Drop entity with references + +``` +drop entity MyModule.Order; +``` + +**Expected:** Entity with references is dropped without warning. No warning issued. + +--- + +## 7. SHOW ASSOCIATIONS + +### 7.1 List all associations + +``` +show associations; +``` + +**Expected:** Table with columns `Qualified Name | Module | Name | Parent | Child | Type | Owner | Storage`. Summary `(N associations)`. + +### 7.2 List associations in a module + +``` +show associations in MyModule; +``` + +**Expected:** Only associations from `MyModule`. Same format. + +### 7.3 Cross-module associations + +``` +show associations; +``` + +**Expected:** Cross-module associations listed with full qualified names for both parent and child entities. + +--- + +## 8. SHOW ASSOCIATION (single) + +### 8.1 Basic association + +``` +show association MyModule.Order_Customer; +``` + +**Expected:** +``` +Association: MyModule.Order_Customer + Type: Reference + Owner: Default + Storage: Column +``` + +### 8.2 Cross-module association + +``` +show association ; +``` + +**Expected:** Shows `(cross-module)` and full child entity path. + +### 8.3 Non-existent association + +``` +show association Fake.Missing; +``` + +**Expected:** Error — association not found. + +--- + +## 9. DESCRIBE ASSOCIATION + +### 9.1 Reference association + +``` +describe association MyModule.Order_Customer; +``` + +**Expected:** +``` +create association MyModule.Order_Customer +from MyModule.Order to MyModule.Customer +type Reference +owner Default +delete_behavior DELETE_BUT_KEEP_REFERENCES; +/ +``` + +### 9.2 ReferenceSet association + +``` +describe association MyModule.Project_Employees; +``` + +**Expected:** `type ReferenceSet`, `owner Both`. + +> **Note:** Owner depends on existing project data; may show `owner Default`. + +### 9.3 Association with documentation + +Find an association with a doc comment. Verify `/** ... */` appears in output. + +--- + +## 10. CREATE ASSOCIATION + +### 10.1 Basic reference (many-to-one) + +``` +create association MyModule.Order_Customer +from MyModule.Order to MyModule.Customer +type reference +owner default +delete_behavior DELETE_BUT_KEEP_REFERENCES; +``` + +**Expected:** Association created. `describe` matches input. + +### 10.2 ReferenceSet (many-to-many) + +``` +create association MyModule.Project_Employees +from MyModule.Project to MyModule.Employee +type referenceset +owner both; +``` + +**Expected:** Association created with `type ReferenceSet`, `owner Both`. + +### 10.3 Self-referencing association + +``` +create association MyModule.Category_Parent +from MyModule.Category to MyModule.Category +type reference +owner default +delete_behavior DELETE_BUT_KEEP_REFERENCES; +``` + +**Expected:** Association created. Parent and child are same entity. + +### 10.4 Cross-module association + +``` +create association MyModule.Account_User +from MyModule.Account to Administration.Account +type reference +owner default; +``` + +**Expected:** Association created spanning two modules. + +### 10.5 With explicit storage + +``` +create association MyModule.Tag_Item +from MyModule.Tag to MyModule.Item +type reference +owner default +storage table; +``` + +**Expected:** Association uses table storage instead of column. + +> **Note:** `describe` omits `storage table` (only `show` reveals storage info). + +### 10.6 Delete behavior — CASCADE + +``` +create association MyModule.Parent_Child +from MyModule.Parent to MyModule.Child +type reference +owner default +delete_behavior CASCADE; +``` + +**Expected:** `describe` shows `DELETE_CASCADE`. + +> **Note:** `describe` outputs `DELETE_CASCADE` but parser accepts `CASCADE`. See BUG-001. + +### 10.7 Delete behavior — IF_NO_REFERENCES + +``` +create association MyModule.Strict_Ref +from MyModule.A to MyModule.B +type reference +owner default +delete_behavior DELETE_IF_NO_REFERENCES; +``` + +**Expected:** `describe` shows `DELETE_IF_NO_REFERENCES`. + +### 10.8 Duplicate association + +``` +create association MyModule.Order_Customer +from MyModule.Order to MyModule.Customer +type reference +owner default; +``` + +**Expected:** **Known issue (BUG-002):** No duplicate detection for associations. Second create silently succeeds. + +--- + +## 11. ALTER ASSOCIATION + +### 11.1 Change delete behavior + +``` +alter association MyModule.Order_Customer set delete_behavior CASCADE; +``` + +**Expected:** `describe` shows updated behavior (`DELETE_CASCADE` in output). + +### 11.2 Change owner + +``` +alter association MyModule.Order_Customer set owner both; +``` + +**Expected:** `describe` shows `owner Both`. + +### 11.3 Change storage + +``` +alter association MyModule.Order_Customer set storage table; +``` + +**Expected:** `describe` shows `storage table`. + +### 11.4 Set comment + +``` +alter association MyModule.Order_Customer set comment 'Links orders to customers'; +``` + +**Expected:** `describe` shows documentation comment. + +--- + +## 12. DROP ASSOCIATION + +### 12.1 Drop existing association + +``` +drop association MyModule.Order_Customer; +``` + +**Expected:** Association removed. `show association MyModule.Order_Customer` returns error. + +### 12.2 Drop non-existent association + +``` +drop association MyModule.Fake; +``` + +**Expected:** Error — not found. + +### 12.3 Drop cross-module association + +``` +drop association MyModule.Account_User; +``` + +**Expected:** Cross-module association removed. + +--- + +## 13. SHOW CONSTANTS + +### 13.1 List all constants + +``` +show constants; +``` + +**Expected:** Table with columns `Qualified Name | Module | Name | Folder | Type | Default | Exposed`. Sorted alphabetically. + +### 13.2 List constants in a module + +``` +show constants in MyModule; +``` + +**Expected:** Only constants from `MyModule`. + +### 13.3 Empty module + +``` +show constants in NonExistentModule; +``` + +**Expected:** `No constants found.` or error. + +--- + +## 14. SHOW CONSTANT VALUES + +### 14.1 All constant values + +``` +show constant values; +``` + +**Expected:** Table with columns `Constant | Configuration | Value`. One row per constant for `(default)`, additional rows per configuration override. Values truncated to 60 chars. + +### 14.2 Constants in a module + +``` +show constant values in MyModule; +``` + +**Expected:** Only constants from `MyModule`. Same format. + +--- + +## 15. DESCRIBE CONSTANT + +### 15.1 String constant + +``` +describe constant MyModule.ServiceEndpoint; +``` + +**Expected:** +``` +create or modify constant MyModule.ServiceEndpoint + type String + default 'https://api.example.com/v1' +; +/ +``` + +### 15.2 Boolean constant + +``` +describe constant MyModule.EnableDebug; +``` + +**Expected:** `default true` or `default false` (unquoted). + +### 15.3 Constant with folder and comment + +``` +describe constant MyModule.MaxRetries; +``` + +**Expected:** Output includes `folder 'path/to/folder'` and `comment 'text'` if present. + +### 15.4 Exposed constant + +``` +describe constant MyModule.ExposedConst; +``` + +**Expected:** Output includes `exposed to client`. + +--- + +## 16. CREATE CONSTANT + +### 16.1 String constant + +``` +create constant MyModule.ServiceEndpoint type String default 'https://api.example.com/v1'; +``` + +**Expected:** Constant created. `describe` matches. + +### 16.2 Integer constant + +``` +create constant MyModule.MaxRetries type Integer default 3; +``` + +**Expected:** Constant created with integer type and value. + +### 16.3 Long constant + +``` +create constant MyModule.MaxFileSize type Long default 10485760 comment 'Max upload 10MB'; +``` + +**Expected:** Constant with comment. + +> **Note:** Long type maps to Integer. No distinct Long constant type exists. + +### 16.4 Decimal constant + +``` +create constant MyModule.TaxRate type Decimal default 0.21; +``` + +**Expected:** Decimal value preserved. + +### 16.5 Boolean constant + +``` +create constant MyModule.EnableDebug type Boolean default false; +``` + +**Expected:** Boolean constant created. + +### 16.6 DateTime constant + +``` +create constant MyModule.LaunchDate type DateTime default '[%BeginOfCurrentDay%]'; +``` + +**Expected:** DateTime constant with token expression. + +### 16.7 Enumeration constant + +``` +create constant MyModule.DefaultStatus type Enumeration(MyModule.Status) default 'Active'; +``` + +**Expected:** Constant with enumeration type. + +### 16.8 With folder + +``` +create constant MyModule.Nested type String default 'value' folder 'Config/SubFolder'; +``` + +**Expected:** Constant placed in specified folder. Folder auto-created if missing. + +### 16.9 Exposed to client + +``` +create constant MyModule.ClientConst type String default 'visible' exposed to client; +``` + +**Expected:** `describe` shows `exposed to client`. + +### 16.10 `create or modify` — existing constant + +``` +create or modify constant MyModule.MaxRetries type Integer default 5 comment 'Updated'; +``` + +**Expected:** Updates existing constant value and comment. + +### 16.11 Duplicate constant (without `or modify`) + +``` +create constant MyModule.MaxRetries type Integer default 3; +``` + +**Expected:** Error — constant already exists. + +### 16.12 With documentation comment + +``` +/** Maximum number of API retry attempts */ +create constant MyModule.ApiRetries type Integer default 3; +``` + +**Expected:** Constant created with doc comment. + +--- + +## 17. DROP CONSTANT + +### 17.1 Drop existing constant + +``` +drop constant MyModule.ServiceEndpoint; +``` + +**Expected:** Constant removed. + +### 17.2 Drop non-existent constant + +``` +drop constant MyModule.Fake; +``` + +**Expected:** Error — not found. + +--- + +## 18. ROUNDTRIP (BSON) + +Test that CREATE → DESCRIBE → CREATE (from output) produces identical results. + +### 18.1 Entity roundtrip + +``` +create persistent entity RtTest.Employee ( + Name: string(200) not null, + Salary: decimal default 0.00, + Active: boolean default true +) +index (Name); +``` + +1. Run `describe entity RtTest.Employee` +2. Copy output (removing leading `create or modify` → just `create`) +3. Drop entity: `drop entity RtTest.Employee` +4. Execute copied MDL +5. Run `describe` again + +**Expected:** Output identical between step 1 and step 5. + +### 18.2 Association roundtrip + +``` +create association RtTest.Emp_Dept +from RtTest.Employee to RtTest.Department +type reference +owner default +delete_behavior DELETE_CASCADE; +``` + +1. `describe association RtTest.Emp_Dept` +2. Drop: `drop association RtTest.Emp_Dept` +3. Execute described MDL +4. `describe` again + +**Expected:** Identical output. + +> **Known issue (BUG-001):** Association roundtrip broken. `describe` outputs `DELETE_CASCADE` but parser only accepts `CASCADE`. + +### 18.3 Constant roundtrip + +``` +create constant RtTest.ApiKey type String default 'sk-test-123' comment 'Test key' exposed to client; +``` + +1. `describe constant RtTest.ApiKey` +2. Drop: `drop constant RtTest.ApiKey` +3. Execute described MDL +4. `describe` again + +**Expected:** Identical output. + +--- + +## 19. CATALOG INTEGRATION + +### 19.1 Entity in catalog + +``` +refresh catalog; +select * from CATALOG.ENTITIES where name = 'Account'; +``` + +**Expected:** Entity appears in catalog with correct module, type, attribute count. + +> **Known issue (BUG-003):** Catalog SELECT currently returns empty results. + +### 19.2 Association in catalog + +``` +select * from CATALOG.ASSOCIATIONS where parent_entity = 'MyModule.Order'; +``` + +**Expected:** Empty result or error — table not implemented. + +### 19.3 References for entity + +``` +show references to Administration.Account; +``` + +**Expected:** Lists microflows, pages, and other documents referencing this entity. + +> **Known issue (BUG-003):** Catalog SELECT currently returns empty results. + +--- + +## 20. MULTI-STEP WORKFLOWS + +### 20.1 Create entity → add association → grant access + +``` +create persistent entity MyModule.Product ( + Name: string(200) not null, + Price: decimal default 0 +); + +create persistent entity MyModule.Category ( + Label: string(100) +); + +create association MyModule.Product_Category +from MyModule.Product to MyModule.Category +type reference +owner default; + +create module role MyModule.Manager; +grant Manager on MyModule.Product (create, delete, read *, write *); +grant Manager on MyModule.Category (read *); +``` + +**Expected:** All statements succeed. `show entities in MyModule` shows both entities with correct association count. + +### 20.2 Create constant → modify → verify + +``` +create constant MyModule.Timeout type Integer default 30; +create or modify constant MyModule.Timeout type Integer default 60 comment 'Increased timeout'; +describe constant MyModule.Timeout; +``` + +**Expected:** Final `describe` shows `default 60` and comment. + +--- + +## 21. FAILURE MODES & ERROR RECOVERY + +### 21.1 Invalid attribute type + +``` +create persistent entity MyModule.Bad ( + Value: invalidtype +); +``` + +**Expected:** Unknown types are silently interpreted as enumeration references (BUG-009). No error raised. + +### 21.2 Missing module qualification + +``` +create persistent entity UnqualifiedName ( + X: integer +); +``` + +**Expected:** Error — module name required. + +### 21.3 Alter non-existent attribute + +``` +alter entity MyModule.Product drop attribute FakeAttr; +``` + +**Expected:** Error — attribute not found. + +### 21.4 Create association with missing entity + +``` +create association MyModule.Bad_Ref +from MyModule.Nonexistent to MyModule.Product +type reference +owner default; +``` + +**Expected:** Error — parent entity not found. + +### 21.5 Invalid OQL in view entity + +``` +create view entity MyModule.BadView ( + X: integer +) as + select invalid syntax here; +``` + +**Expected:** Entity created (OQL not validated at create time). View entity stored with invalid OQL as raw text. + +### 21.6 Drop entity referenced by association + +``` +drop entity MyModule.Customer; +``` + +**Expected:** Entity silently dropped without warning. Associations may become orphaned. + +--- + +## 22. BOUNDARY & STRESS + +### 22.1 Entity with many attributes (50+) + +Create an entity with 50 attributes. Verify `show` and `describe` handle large attribute lists. + +### 22.2 Long attribute names + +``` +create persistent entity MyModule.LongNames ( + ThisIsAVeryLongAttributeNameThatExceedsNormalLength: string(200) +); +``` + +**Expected:** Created and displayed correctly. + +### 22.3 String constant with special characters + +``` +create constant MyModule.Special type String default 'it''s a "test" with\nnewlines'; +``` + +**Expected:** Escaped characters preserved in describe output. + +### 22.4 Maximum string length + +``` +create persistent entity MyModule.MaxLen ( + BigText: string(1048576) +); +``` + +**Expected:** Entity created (or error if max exceeded). + +### 22.5 Many associations on one entity + +Create 10+ associations from one entity to different targets. Verify `show associations` and entity describe handle them. + +--- + +## Test Project Coverage Matrix + +| Operation | Lato Enquiry | Evora Factory | Lato Product | +|-----------|:---:|:---:|:---:| +| SHOW ENTITIES | x | x | x | +| SHOW ENTITY | x | x | x | +| DESCRIBE ENTITY | x | x | x | +| CREATE ENTITY | x | | | +| ALTER ENTITY | x | | | +| DROP ENTITY | x | | | +| SHOW ASSOCIATIONS | x | x | x | +| DESCRIBE ASSOCIATION | x | x | | +| CREATE ASSOCIATION | x | | | +| ALTER ASSOCIATION | x | | | +| DROP ASSOCIATION | x | | | +| SHOW CONSTANTS | x | x | x | +| SHOW CONSTANT VALUES | x | x | | +| DESCRIBE CONSTANT | x | x | | +| CREATE CONSTANT | x | | | +| DROP CONSTANT | x | | | + +Read operations tested on all projects. Write operations on copies of one project. + +--- + +## Automated Test Coverage + +| Section | Automated | Manual-only | +|---------|:---------:|:-----------:| +| 1. SHOW ENTITIES | Mock tests | | +| 2. SHOW ENTITY | Mock tests | | +| 3. DESCRIBE ENTITY | Mock tests | | +| 4. CREATE ENTITY | Mock + roundtrip | | +| 5. ALTER ENTITY | Mock tests | Multi-action | +| 6. DROP ENTITY | Mock tests | | +| 7–9. SHOW/DESCRIBE ASSOCIATION | Mock tests | | +| 10. CREATE ASSOCIATION | Mock tests | Cross-module | +| 11. ALTER ASSOCIATION | Mock tests | | +| 12. DROP ASSOCIATION | Mock tests | | +| 13–14. SHOW CONSTANTS | Mock tests | | +| 15. DESCRIBE CONSTANT | Mock tests | | +| 16. CREATE CONSTANT | Mock tests | | +| 17. DROP CONSTANT | Mock tests | | +| 18. Roundtrip | Roundtrip tests | Complex entities | +| 19. Catalog | | All manual | +| 20. Multi-step | | All manual | +| 21. Failure modes | Partial | Edge cases | +| 22. Boundary | | All manual | + +--- + +## Manual Test Report Template + +**Tester:** _______________ +**Date:** _______________ +**Project:** _______________ + +| # | Section | Test | Pass | Fail | Skip | Notes | +|---|---------|------|:----:|:----:|:----:|-------| +| 1.1 | SHOW ENTITIES | List all | | | | | +| 1.2 | SHOW ENTITIES | Filter by module | | | | | +| 1.3 | SHOW ENTITIES | Generalization column | | | | | +| 1.4 | SHOW ENTITIES | Empty module | | | | | +| 1.5 | SHOW ENTITIES | Entity types | | | | | +| 2.1 | SHOW ENTITY | Persistent | | | | | +| 2.2 | SHOW ENTITY | Non-persistent | | | | | +| 2.3 | SHOW ENTITY | Not found | | | | | +| 3.1 | DESCRIBE | Basic | | | | | +| 3.2 | DESCRIBE | All features | | | | | +| 3.3 | DESCRIBE | View entity | | | | | +| 3.4 | DESCRIBE | Not found | | | | | +| 4.1 | CREATE | Minimal persistent | | | | | +| 4.2 | CREATE | All types | | | | | +| 4.3 | CREATE | Generalization | | | | | +| 4.4 | CREATE | Non-persistent | | | | | +| 4.5 | CREATE | With indexes | | | | | +| 4.6 | CREATE | Event handlers | | | | | +| 4.7 | CREATE | System attributes | | | | | +| 4.8 | CREATE | create or modify (new) | | | | | +| 4.9 | CREATE | create or modify (update) | | | | | +| 4.10 | CREATE | Duplicate error | | | | | +| 4.11 | CREATE | View entity | | | | | +| 4.12 | CREATE | create or replace view | | | | | +| 4.13 | CREATE | Position annotation | | | | | +| 5.1 | ALTER | Add attribute | | | | | +| 5.2 | ALTER | Rename attribute | | | | | +| 5.3 | ALTER | Modify type | | | | | +| 5.4 | ALTER | Drop attribute | | | | | +| 5.5 | ALTER | Set documentation | | | | | +| 5.6 | ALTER | Set position | | | | | +| 5.7 | ALTER | Add index | | | | | +| 5.8 | ALTER | Drop index | | | | | +| 5.9 | ALTER | Add event handler | | | | | +| 5.10 | ALTER | Drop event handler | | | | | +| 5.11 | ALTER | Multiple actions | | | | | +| 5.12 | ALTER | Drop pseudo-attribute | | | | | +| 5.13 | ALTER | Make calculated | | | | | +| 6.1 | DROP ENTITY | Existing | | | | | +| 6.2 | DROP ENTITY | Non-existent | | | | | +| 6.3 | DROP ENTITY | View entity | | | | | +| 6.4 | DROP ENTITY | With references | | | | | +| 7.1 | SHOW ASSOC | List all | | | | | +| 7.2 | SHOW ASSOC | Filter module | | | | | +| 7.3 | SHOW ASSOC | Cross-module | | | | | +| 8.1 | SHOW ASSOC | Single basic | | | | | +| 8.2 | SHOW ASSOC | Single cross-module | | | | | +| 8.3 | SHOW ASSOC | Not found | | | | | +| 9.1 | DESCRIBE ASSOC | Reference | | | | | +| 9.2 | DESCRIBE ASSOC | ReferenceSet | | | | | +| 9.3 | DESCRIBE ASSOC | With documentation | | | | | +| 10.1 | CREATE ASSOC | Basic reference | | | | | +| 10.2 | CREATE ASSOC | ReferenceSet | | | | | +| 10.3 | CREATE ASSOC | Self-referencing | | | | | +| 10.4 | CREATE ASSOC | Cross-module | | | | | +| 10.5 | CREATE ASSOC | Explicit storage | | | | | +| 10.6 | CREATE ASSOC | DELETE_CASCADE | | | | | +| 10.7 | CREATE ASSOC | DELETE_IF_NO_REFERENCES | | | | | +| 10.8 | CREATE ASSOC | Duplicate error | | | | | +| 11.1 | ALTER ASSOC | Delete behavior | | | | | +| 11.2 | ALTER ASSOC | Change owner | | | | | +| 11.3 | ALTER ASSOC | Change storage | | | | | +| 11.4 | ALTER ASSOC | Set comment | | | | | +| 12.1 | DROP ASSOC | Existing | | | | | +| 12.2 | DROP ASSOC | Non-existent | | | | | +| 12.3 | DROP ASSOC | Cross-module | | | | | +| 13.1 | SHOW CONST | List all | | | | | +| 13.2 | SHOW CONST | Filter module | | | | | +| 13.3 | SHOW CONST | Empty module | | | | | +| 14.1 | CONST VALUES | All values | | | | | +| 14.2 | CONST VALUES | Filter module | | | | | +| 15.1 | DESCRIBE CONST | String | | | | | +| 15.2 | DESCRIBE CONST | Boolean | | | | | +| 15.3 | DESCRIBE CONST | With folder/comment | | | | | +| 15.4 | DESCRIBE CONST | Exposed | | | | | +| 16.1 | CREATE CONST | String | | | | | +| 16.2 | CREATE CONST | Integer | | | | | +| 16.3 | CREATE CONST | Long | | | | | +| 16.4 | CREATE CONST | Decimal | | | | | +| 16.5 | CREATE CONST | Boolean | | | | | +| 16.6 | CREATE CONST | DateTime | | | | | +| 16.7 | CREATE CONST | Enumeration | | | | | +| 16.8 | CREATE CONST | With folder | | | | | +| 16.9 | CREATE CONST | Exposed | | | | | +| 16.10 | CREATE CONST | create or modify | | | | | +| 16.11 | CREATE CONST | Duplicate error | | | | | +| 16.12 | CREATE CONST | Doc comment | | | | | +| 17.1 | DROP CONST | Existing | | | | | +| 17.2 | DROP CONST | Non-existent | | | | | +| 18.1 | ROUNDTRIP | Entity | | | | | +| 18.2 | ROUNDTRIP | Association | | | | | +| 18.3 | ROUNDTRIP | Constant | | | | | +| 19.1 | CATALOG | Entity in catalog | | | | | +| 19.2 | CATALOG | Association in catalog | | | | | +| 19.3 | CATALOG | Callers | | | | | +| 20.1 | MULTI-STEP | Entity + assoc + access | | | | | +| 20.2 | MULTI-STEP | Constant modify chain | | | | | +| 21.1 | FAILURE | Invalid type | | | | | +| 21.2 | FAILURE | Missing module | | | | | +| 21.3 | FAILURE | Alter non-existent attr | | | | | +| 21.4 | FAILURE | Missing entity in assoc | | | | | +| 21.5 | FAILURE | Invalid OQL | | | | | +| 21.6 | FAILURE | Drop referenced entity | | | | | +| 22.1 | BOUNDARY | Many attributes | | | | | +| 22.2 | BOUNDARY | Long names | | | | | +| 22.3 | BOUNDARY | Special characters | | | | | +| 22.4 | BOUNDARY | Max string length | | | | | +| 22.5 | BOUNDARY | Many associations | | | | | + +**Summary:** ___ / ___ passed | ___ failed | ___ skipped diff --git a/docs/15-testing/enumeration-test-cases.md b/docs/15-testing/enumeration-test-cases.md new file mode 100644 index 00000000..1f35cc1d --- /dev/null +++ b/docs/15-testing/enumeration-test-cases.md @@ -0,0 +1,525 @@ +# Enumeration Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#386](https://github.com/mendixlabs/mxcli/pull/386) + +## Test Projects + +Demo apps from [Mendix App Gallery](https://appgallery.mendixcloud.com/): + +| App | Studio Pro | Enumerations | +|-----|-----------|--------------| +| Lato Enquiry Management | 11.4.0 | 74 | +| Evora - Factory Management | 10.24.15 | 113 | +| Lato Product Inventory | 11.2.0 | 92 | + +--- + +## Setup + +### 1. Download test apps + +1. Go to [Mendix App Gallery](https://appgallery.mendixcloud.com/) +2. Download each demo app listed above +3. Open each `.mpk` in Studio Pro to extract the `.mpr` file + +### 2. Build mxcli + +```bash +make build && make test && make lint-go +``` + +### 3. Smoke test + +```bash +APPS_DIR= +for mpr in "$APPS_DIR"/*/*.mpr; do + echo "=== $(basename $(dirname $mpr)) ===" + echo "show enumerations;" > /tmp/show-enum.mdl + mxcli exec /tmp/show-enum.mdl -p "$mpr" 2>&1 | tail -1 +done +``` + +Expected: count line `(N enumerations)` for each project. + +### 4. Interactive testing + +```bash +mxcli repl -p /EnquiriesManagement.mpr +``` + +### 5. Script-based testing + +```bash +mxcli exec test-sequence.mdl -p +``` + +> **IMPORTANT:** Always run destructive tests against a **copy** of the project folder. +> Dropped enumerations cannot be recovered. +> +> ```bash +> cp -r MyProject MyProject-test +> mxcli repl -p MyProject-test/MyProject.mpr +> ``` + +--- + +## 1. SHOW ENUMERATIONS + +### 1.1 List all enumerations + +``` +show enumerations; +``` + +**Expected:** Table with columns `Qualified Name | Module | Name | Folder | Values`. Summary `(N enumerations)`. Sorted alphabetically. + +### 1.2 List enumerations in a module + +``` +show enumerations in Administration; +``` + +**Expected:** Only enumerations from `Administration` module. + +> (Note: Administration may have 0 enumerations. Use a module with enumerations like `AgentCommons` for a more illustrative test.) + +### 1.3 Empty module + +``` +show enumerations in NonExistentModule; +``` + +**Expected:** Empty result or `(0 enumerations)`. + +--- + +## 2. DESCRIBE ENUMERATION + +### 2.1 Basic enumeration + +Use any existing enumeration in the project (e.g., `AgentCore.Enum_StatusGeneral`): + +``` +describe enumeration AgentCore.Enum_StatusGeneral; +``` + +**Expected:** `create or modify enumeration ...` block with value names and captions, terminated by `/`. + +### 2.2 Enumeration with documentation + +> **Note:** Create a documented enumeration first (see §3.2), then describe it. No test project has pre-existing documented enumerations. + +**Expected:** Output starts with `/** ... */` documentation block. + +### 2.3 Enumeration with value documentation + +> **Note:** Create a documented enumeration first (see §3.2), then describe it. No test project has pre-existing documented enumerations. + +**Expected:** Per-value doc comments: `/** Order received, awaiting processing */` above value. + +### 2.4 Non-existent enumeration + +``` +describe enumeration Fake.Missing; +``` + +**Expected:** Error — enumeration not found. + +--- + +## 3. CREATE ENUMERATION + +### 3.1 Simple enumeration + +``` +create enumeration MyModule.Color ( + RED 'Red', + GREEN 'Green', + BLUE 'Blue' +); +``` + +**Expected:** `Created enumeration: MyModule.Color`. `show enumerations in MyModule` includes it with `Values: 3`. + +### 3.2 Enumeration with documentation + +``` +/** Priority levels for task ordering */ +create enumeration MyModule.Priority ( + /** Highest urgency */ + CRITICAL 'Critical', + HIGH 'High', + MEDIUM 'Medium', + LOW 'Low' +); +``` + +**Expected:** Created. `describe` shows documentation blocks. + +> **Known issue (BUG-004):** Documentation blocks (`/** */`) are NOT preserved on roundtrip. Describe output omits documentation. + +### 3.3 Single-value enumeration + +``` +create enumeration MyModule.YesNo ( + YES 'Yes' +); +``` + +**Expected:** Created with 1 value. + +### 3.4 Quoted identifier (reserved word) + +``` +create enumeration MyModule.UserType ( + "user" 'Standard User', + ADMIN 'Administrator' +); +``` + +**Expected:** Created. Value name `user` (reserved word) accepted when quoted. + +### 3.5 Reserved word — unquoted + +``` +create enumeration MyModule.BadEnum ( + class 'Class Value' +); +``` + +**Expected:** Error (CE7247) — `enumeration value 'class' is a reserved word`. + +### 3.6 Other reserved words + +Test these unquoted values trigger CE7247: + +> **Note:** Error messages may append a spurious hint: `"hint: ... is defined later in this script"` even for single statements. Ignore this hint. + +| Value | Reserved | +|-------|----------| +| `null` | Java keyword | +| `true` | Java keyword | +| `enum` | Java keyword | +| `context` | Mendix reserved | +| `guid` | Mendix reserved | +| `changeddate` | Mendix reserved | +| `currentuser` | Mendix reserved | + +### 3.7 `create or modify` — new enumeration + +``` +create or modify enumeration MyModule.Fresh ( + A 'Alpha', + B 'Beta' +); +``` + +**Expected:** Creates enumeration (same as without `or modify`). + +### 3.8 `create or modify` — existing enumeration + +``` +create or modify enumeration MyModule.Color ( + RED 'Red', + GREEN 'Green', + BLUE 'Blue', + YELLOW 'Yellow' +); +``` + +**Expected:** Replaces existing enumeration. Output says `Created enumeration: MyModule.Color`. `describe` shows 4 values. + +> **Note:** `create or modify` outputs "Created enumeration:" for both new and update operations. + +### 3.9 Duplicate (without `or modify`) + +``` +create enumeration MyModule.Color ( + X 'X' +); +``` + +**Expected:** Error — `enumeration already exists: MyModule.Color (use create or modify to update)`. + +### 3.10 Module auto-creation + +``` +create enumeration NewModule.Status ( + ACTIVE 'Active' +); +``` + +**Expected:** Both module and enumeration created. + +--- + +## 4. DROP ENUMERATION + +### 4.1 Drop existing enumeration + +``` +drop enumeration MyModule.Color; +``` + +**Expected:** `Dropped enumeration: MyModule.Color`. + +### 4.2 Drop non-existent enumeration + +``` +drop enumeration MyModule.Fake; +``` + +**Expected:** Error — not found. + +### 4.3 Drop by name only (no module qualifier) + +``` +drop enumeration Color; +``` + +**Expected:** Matches by name if unambiguous. Error if multiple enumerations share name. + +#### 4.3.1 Ambiguous unqualified name + +**Setup:** + +``` +create enumeration ModuleA.Status ( + ACTIVE 'Active', + INACTIVE 'Inactive' +); + +create enumeration ModuleB.Status ( + OPEN 'Open', + CLOSED 'Closed' +); +``` + +**Test:** + +``` +drop enumeration Status; +``` + +> **Known issue (BUG-006):** Ambiguous unqualified name silently drops first match instead of erroring. + +--- + +## 5. ROUNDTRIP (BSON) + +### 5.1 Enumeration roundtrip + +``` +create enumeration RtTest.Status ( + ACTIVE 'Active', + INACTIVE 'Inactive', + ARCHIVED 'Archived' +); +``` + +1. `describe enumeration RtTest.Status` +2. Drop: `drop enumeration RtTest.Status` +3. Execute described MDL (change `create or modify` to `create`) +4. `describe` again + +**Expected:** Output identical between step 1 and step 4. + +### 5.2 Roundtrip with documentation + +``` +/** Payment status tracking */ +create enumeration RtTest.PayStatus ( + /** Awaiting payment */ + PENDING 'Pending', + PAID 'Paid', + REFUNDED 'Refunded' +); +``` + +1. `describe` → drop → recreate from output → `describe` + +**Expected:** Documentation preserved through roundtrip. + +> **Known issue (BUG-004):** Documentation lost on roundtrip. + +--- + +## 6. MULTI-STEP WORKFLOWS + +### 6.1 Create enumeration → use in entity + +``` +create enumeration MyModule.TaskStatus ( + TODO 'To Do', + IN_PROGRESS 'In Progress', + DONE 'Done' +); + +create persistent entity MyModule.Task ( + Title: string(200) not null, + Status: enumeration(MyModule.TaskStatus) default MyModule.TaskStatus.TODO +); +``` + +**Expected:** Both created. Entity attribute references enumeration. `describe entity` shows `enumeration(MyModule.TaskStatus)`. + +### 6.2 Replace enumeration → verify entity still works + +``` +create or modify enumeration MyModule.TaskStatus ( + TODO 'To Do', + IN_PROGRESS 'In Progress', + DONE 'Done', + BLOCKED 'Blocked' +); + +describe entity MyModule.Task; +``` + +**Expected:** Entity still references `MyModule.TaskStatus`. New value `BLOCKED` available. + +--- + +## 7. FAILURE MODES & ERROR RECOVERY + +### 7.1 Empty value list + +``` +create enumeration MyModule.Empty (); +``` + +**Expected:** Error — at least one value required (or parser error). + +### 7.2 Duplicate value names + +``` +create enumeration MyModule.Dup ( + A 'First', + A 'Second' +); +``` + +**Expected:** Error — duplicate value name. + +> **Known issue (BUG-005):** Duplicate value names accepted without error. Both values stored. + +### 7.3 Drop enumeration used by entity attribute + +``` +drop enumeration MyModule.TaskStatus; +``` + +**Expected:** Enumeration dropped (no referential integrity enforcement — may leave entity attribute orphaned). Verify `describe entity MyModule.Task` still works but shows broken reference or generic type. + +### 7.4 Case sensitivity in reserved word check + +``` +create enumeration MyModule.CaseTest ( + NULL 'Null Value', + True 'True Value' +); +``` + +**Expected:** Error — reserved word check is case-insensitive. + +--- + +## 8. BOUNDARY & STRESS + +### 8.1 Many values (50+) + +Create an enumeration with 50 values. Verify `show` and `describe` handle it. + +### 8.2 Long captions + +``` +create enumeration MyModule.LongCaptions ( + VAL1 'This is a very long caption that exceeds what would normally be displayed in a UI dropdown' +); +``` + +**Expected:** Created and described correctly. + +### 8.3 Special characters in captions + +``` +create enumeration MyModule.Special ( + A 'It''s a "test" value', + B 'Line1\nLine2' +); +``` + +**Expected:** Escaped characters preserved in describe output. + +> **Known issue (BUG-007):** Single quotes in captions not escaped in describe output (`It's` instead of `It''s`). Breaks roundtrip. + +--- + +## Test Project Coverage Matrix + +| Operation | Lato Enquiry | Evora Factory | Lato Product | +|-----------|:---:|:---:|:---:| +| SHOW ENUMERATIONS | x | x | x | +| DESCRIBE ENUMERATION | x | x | x | +| CREATE ENUMERATION | x | | | +| DROP ENUMERATION | x | | | + +--- + +## Automated Test Coverage + +| Section | Automated | Manual-only | +|---------|:---------:|:-----------:| +| 1. SHOW ENUMERATIONS | Mock tests | | +| 2. DESCRIBE ENUMERATION | Mock tests | | +| 3. CREATE ENUMERATION | Mock tests | Reserved words, quoted identifiers | +| 4. DROP ENUMERATION | Mock tests | | +| 5. Roundtrip | | All manual | +| 6. Multi-step | | All manual | +| 7. Failure modes | Partial (reserved) | Empty list, duplicates | +| 8. Boundary | | All manual | + +--- + +## Manual Test Report Template + +**Tester:** _______________ +**Date:** _______________ +**Project:** _______________ + +| # | Section | Test | Pass | Fail | Skip | Notes | +|---|---------|------|:----:|:----:|:----:|-------| +| 1.1 | SHOW | List all | | | | | +| 1.2 | SHOW | Filter module | | | | | +| 1.3 | SHOW | Empty module | | | | | +| 2.1 | DESCRIBE | Basic | | | | | +| 2.2 | DESCRIBE | With documentation | | | | | +| 2.3 | DESCRIBE | Value documentation | | | | | +| 2.4 | DESCRIBE | Not found | | | | | +| 3.1 | CREATE | Simple | | | | | +| 3.2 | CREATE | With documentation | | | | | +| 3.3 | CREATE | Single value | | | | | +| 3.4 | CREATE | Quoted reserved word | | | | | +| 3.5 | CREATE | Unquoted reserved word | | | | | +| 3.6 | CREATE | Other reserved words | | | | | +| 3.7 | CREATE | create or modify (new) | | | | | +| 3.8 | CREATE | create or modify (update) | | | | | +| 3.9 | CREATE | Duplicate error | | | | | +| 3.10 | CREATE | Module auto-creation | | | | | +| 4.1 | DROP | Existing | | | | | +| 4.2 | DROP | Non-existent | | | | | +| 4.3 | DROP | By name only | | | | | +| 5.1 | ROUNDTRIP | Basic | | | | | +| 5.2 | ROUNDTRIP | With documentation | | | | | +| 6.1 | MULTI-STEP | Create + use in entity | | | | | +| 6.2 | MULTI-STEP | Replace + verify entity | | | | | +| 7.1 | FAILURE | Empty value list | | | | | +| 7.2 | FAILURE | Duplicate values | | | | | +| 7.3 | FAILURE | Drop used enumeration | | | | | +| 7.4 | FAILURE | Case-insensitive reserved | | | | | +| 8.1 | BOUNDARY | Many values | | | | | +| 8.2 | BOUNDARY | Long captions | | | | | +| 8.3 | BOUNDARY | Special characters | | | | | + +**Summary:** ___ / ___ passed | ___ failed | ___ skipped diff --git a/docs/15-testing/image-collection-test-cases.md b/docs/15-testing/image-collection-test-cases.md new file mode 100644 index 00000000..c8b352a8 --- /dev/null +++ b/docs/15-testing/image-collection-test-cases.md @@ -0,0 +1,468 @@ +# Image Collection Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#386](https://github.com/mendixlabs/mxcli/pull/386) + +## Test Projects + +Demo apps from [Mendix App Gallery](https://appgallery.mendixcloud.com/): + +| App | Studio Pro | Image Collections | +|-----|------------|--------------------| +| Lato Enquiry Management | 11.4.0 | — | +| Evora - Factory Management | 10.24.15 | — | +| Lato Product Inventory | 11.2.0 | — | + +--- + +## Setup + +### 1. Build mxcli + +```bash +make build && make test && make lint-go +``` + +### 2. Prepare test images + +```bash +mkdir -p /tmp/mxcli-test-images +convert -size 32x32 xc:red /tmp/mxcli-test-images/logo.png +convert -size 32x32 xc:blue /tmp/mxcli-test-images/icon.png +echo '' > /tmp/mxcli-test-images/vector.svg +``` + +### 3. Interactive testing + +```bash +mxcli repl -p /EnquiriesManagement.mpr +``` + +> **IMPORTANT:** Always run destructive tests against a **copy** of the project folder. +> Dropped image collections cannot be recovered. +> +> ```bash +> cp -r MyProject MyProject-test +> mxcli repl -p MyProject-test/MyProject.mpr +> ``` + +--- + +## 1. SHOW IMAGE COLLECTION + +### 1.1 List all image collections + +``` +show image collections; +``` + +**Expected:** Table with columns `Image Collection | Export Level | Images`. Summary `(N image collections)`. + +### 1.2 Filter by module + +``` +show image collections in Administration; +``` + +**Expected:** Only image collections from `Administration` module. + +### 1.3 Empty module + +``` +show image collections in NonExistentModule; +``` + +**Expected:** Empty result or `(0 image collections)`. + +--- + +## 2. DESCRIBE IMAGE COLLECTION + +### 2.1 Basic describe + +``` +describe image collection MyModule.Icons; +``` + +**Expected:** +``` +create or replace image collection MyModule.Icons export level 'Public' ( + image logo from file 'logo.png', + image icon from file 'icon.svg' +); +/ +``` + +### 2.2 Image extraction to disk + +``` +describe image collection MyModule.Icons; +``` + +**Expected:** Images extracted to `/tmp/mxcli-preview/MyModule.Icons/`. Each image file present with original format. + +### 2.3 Supported formats + +Verify `describe` correctly identifies format for each type: + +| Extension | Format | +|-----------|--------| +| `.png` | Png (default) | +| `.svg` | Svg | +| `.gif` | Gif | +| `.jpg` | Jpg | +| `.bmp` | Bmp | +| `.webp` | Webp | + +### 2.4 Non-existent image collection + +``` +describe image collection Fake.Missing; +``` + +**Expected:** Error — image collection not found. + +--- + +## 3. CREATE IMAGE COLLECTION + +### 3.1 Empty collection + +``` +create image collection MyModule.EmptyIcons; +``` + +**Expected:** `Created image collection: MyModule.EmptyIcons`. `show image collections in MyModule` lists it with `Images: 0`. + +### 3.2 Collection with images + +``` +create image collection MyModule.AppIcons export level 'Public' comment 'Application icons' ( + image logo from file '/tmp/mxcli-test-images/logo.png', + image icon from file '/tmp/mxcli-test-images/vector.svg' +); +``` + +**Expected:** Created with 2 images. `describe` shows both images with correct formats. + +### 3.3 Hidden export level (default) + +``` +create image collection MyModule.HiddenIcons ( + image logo from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** Created. `show image collections` shows `Export Level: Hidden`. + +### 3.4 Public export level + +``` +create image collection MyModule.PublicIcons export level 'Public' ( + image logo from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** Created. `show image collections` shows `Export Level: Public`. + +### 3.5 Format auto-detection from extension + +Create collections with each supported format. Verify `describe` output shows correct format. + +| File | Detected Format | +|------|-----------------| +| `image.png` | Png | +| `image.svg` | Svg | +| `image.gif` | Gif | +| `image.jpg` | Jpg | +| `image.bmp` | Bmp | +| `image.webp` | Webp | + +### 3.6 `create or replace` — new collection + +``` +create or replace image collection MyModule.Fresh ( + image a from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** Creates collection (same as without `or replace`). + +### 3.7 `create or replace` — existing collection + +``` +create or replace image collection MyModule.Fresh ( + image a from file '/tmp/mxcli-test-images/logo.png', + image b from file '/tmp/mxcli-test-images/icon.png' +); +``` + +**Expected:** Replaces existing collection. `describe` shows 2 images. + +### 3.8 Duplicate (without `or replace`) + +``` +create image collection MyModule.Fresh ( + image x from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** Error — image collection already exists. + +### 3.9 Module auto-creation + +``` +create image collection NewModule.Icons ( + image logo from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** Both module and image collection created. + +--- + +## 4. DROP IMAGE COLLECTION + +### 4.1 Drop existing collection + +``` +drop image collection MyModule.AppIcons; +``` + +**Expected:** `Dropped image collection: MyModule.AppIcons`. + +### 4.2 Drop non-existent collection + +``` +drop image collection MyModule.Fake; +``` + +**Expected:** Error — not found. + +--- + +## 5. ROUNDTRIP + +### 5.1 Create → Describe → Recreate + +1. Create collection with images (3.2) +2. `describe image collection MyModule.AppIcons` — save output +3. `drop image collection MyModule.AppIcons` +4. Execute saved describe output (change `create or replace` to `create`) +5. `describe` again + +**Expected:** Output identical between step 2 and step 5. Extracted image files match. + +--- + +## 6. FAILURE MODES + +### 6.1 Not connected + +Run any command without `-p` or `connect`. + +**Expected:** Error — not connected to a project. + +### 6.2 File not found + +``` +create image collection MyModule.Bad ( + image missing from file '/tmp/does-not-exist.png' +); +``` + +**Expected:** Error — file not found. + +### 6.3 Invalid image format + +``` +create image collection MyModule.Bad ( + image broken from file '/tmp/mxcli-test-images/notanimage.txt' +); +``` + +**Expected:** Error — unsupported image format. + +### 6.4 Backend create failure + +Simulate backend failure (e.g., corrupt MPR, disk full). + +**Expected:** Error from backend layer with descriptive message. + +### 6.5 Preview directory creation failure + +If `/tmp/mxcli-preview/` is not writable: + +``` +describe image collection MyModule.Icons; +``` + +**Expected:** Error creating preview directory. + +--- + +## 7. BOUNDARY & STRESS + +### 7.1 Large image file + +Create a collection with a large image (10 MB+ PNG). + +``` +create image collection MyModule.LargeIcons ( + image big from file '/tmp/mxcli-test-images/large-10mb.png' +); +``` + +**Expected:** Collection created. `describe` extracts image correctly. Verify file size preserved. + +### 7.2 Many images in one collection + +Create a collection with 50+ images. + +``` +create image collection MyModule.ManyIcons ( + image img001 from file '/tmp/mxcli-test-images/logo.png', + image img002 from file '/tmp/mxcli-test-images/icon.png', + ... + image img050 from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** All 50 images stored. `show image collections` reports correct count. `describe` extracts all files. + +### 7.3 Image with special characters in name + +``` +create image collection MyModule.SpecialIcons ( + image my_icon_v2 from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** Image name with underscores accepted. Verify no name sanitization issues. + +### 7.4 Empty image file + +Create a 0-byte file and attempt to import: + +```bash +touch /tmp/mxcli-test-images/empty.png +``` + +``` +create image collection MyModule.EmptyFile ( + image empty from file '/tmp/mxcli-test-images/empty.png' +); +``` + +**Expected:** Error or warning about empty/invalid image data. + +### 7.5 Duplicate image names in one collection + +``` +create image collection MyModule.DupeIcons ( + image logo from file '/tmp/mxcli-test-images/logo.png', + image logo from file '/tmp/mxcli-test-images/icon.png' +); +``` + +**Expected:** Error — duplicate image name within collection. + +### 7.6 All supported formats in one collection + +``` +create image collection MyModule.AllFormats export level 'Public' ( + image png_img from file '/tmp/mxcli-test-images/test.png', + image svg_img from file '/tmp/mxcli-test-images/test.svg', + image gif_img from file '/tmp/mxcli-test-images/test.gif', + image jpg_img from file '/tmp/mxcli-test-images/test.jpg', + image bmp_img from file '/tmp/mxcli-test-images/test.bmp', + image webp_img from file '/tmp/mxcli-test-images/test.webp' +); +``` + +**Expected:** All 6 formats accepted. `describe` shows correct format for each. + +### 7.7 Replace collection with different image set + +``` +create or replace image collection MyModule.Evolving ( + image v1 from file '/tmp/mxcli-test-images/logo.png' +); +create or replace image collection MyModule.Evolving ( + image v2a from file '/tmp/mxcli-test-images/icon.png', + image v2b from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** Second `create or replace` fully replaces. `describe` shows only `v2a` and `v2b`. + +--- + +## Test Project Coverage Matrix + +| Operation | Lato Enquiry | Evora Factory | Lato Inventory | +|-----------|:---:|:---:|:---:| +| SHOW IMAGE COLLECTION | x | x | x | +| DESCRIBE IMAGE COLLECTION | x | x | x | +| CREATE IMAGE COLLECTION | x | | | +| DROP IMAGE COLLECTION | x | | | +| ROUNDTRIP | x | | | +| BOUNDARY & STRESS | x | | | + +--- + +## Automated Test Coverage + +| Section | Automated | Manual-only | +|---------|:---------:|:-----------:| +| 1. SHOW | Mock tests | | +| 2. DESCRIBE | Mock tests | Image extraction to disk | +| 3. CREATE | Mock tests | Format auto-detection | +| 4. DROP | Mock tests | | +| 5. Roundtrip | | All manual | +| 6. Failure modes | Partial | Backend, preview dir | +| 7. Boundary & stress | | All manual | + +--- + +## Manual Test Report Template + +**Tester:** _______________ +**Date:** _______________ +**Project:** _______________ + +| # | Section | Test | Pass | Fail | Skip | Notes | +|---|---------|------|:----:|:----:|:----:|-------| +| 1.1 | SHOW | List all | | | | | +| 1.2 | SHOW | Filter module | | | | | +| 1.3 | SHOW | Empty module | | | | | +| 2.1 | DESCRIBE | Basic | | | | | +| 2.2 | DESCRIBE | Image extraction | | | | | +| 2.3 | DESCRIBE | Supported formats | | | | | +| 2.4 | DESCRIBE | Not found | | | | | +| 3.1 | CREATE | Empty collection | | | | | +| 3.2 | CREATE | With images | | | | | +| 3.3 | CREATE | Hidden (default) | | | | | +| 3.4 | CREATE | Public export level | | | | | +| 3.5 | CREATE | Format auto-detection | | | | | +| 3.6 | CREATE | create or replace (new) | | | | | +| 3.7 | CREATE | create or replace (update) | | | | | +| 3.8 | CREATE | Duplicate error | | | | | +| 3.9 | CREATE | Module auto-creation | | | | | +| 4.1 | DROP | Existing | | | | | +| 4.2 | DROP | Non-existent | | | | | +| 5.1 | ROUNDTRIP | Create → Describe → Recreate | | | | | +| 6.1 | FAILURE | Not connected | | | | | +| 6.2 | FAILURE | File not found | | | | | +| 6.3 | FAILURE | Invalid format | | | | | +| 6.4 | FAILURE | Backend create failure | | | | | +| 6.5 | FAILURE | Preview dir failure | | | | | +| 7.1 | BOUNDARY | Large image (10 MB+) | | | | | +| 7.2 | BOUNDARY | Many images (50+) | | | | | +| 7.3 | BOUNDARY | Special chars in name | | | | | +| 7.4 | BOUNDARY | Empty image file | | | | | +| 7.5 | BOUNDARY | Duplicate image names | | | | | +| 7.6 | BOUNDARY | All formats in one | | | | | +| 7.7 | BOUNDARY | Replace with different set | | | | | + +**Summary:** ___ / ___ passed | ___ failed | ___ skipped diff --git a/docs/15-testing/integration-test-cases.md b/docs/15-testing/integration-test-cases.md new file mode 100644 index 00000000..cafc54ed --- /dev/null +++ b/docs/15-testing/integration-test-cases.md @@ -0,0 +1,2019 @@ +# Integration Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#386](https://github.com/mendixlabs/mxcli/pull/386) + +Covers: OData Clients, OData Services, External Entities, Published REST Services, REST Clients, JSON Structures. + +## Test Projects + +Demo apps from [Mendix App Gallery](https://appgallery.mendixcloud.com/): + +| App | Studio Pro | Integration features | +|-----|-----------|----------------------| +| Lato Enquiry Management | 11.4.0 | OData, REST, JSON structures | +| Evora - Factory Management | 10.24.15 | OData services, external entities | +| Lato Product Inventory | 11.2.0 | Published REST, consumed OData | + +--- + +## Setup + +### 1. Download test apps + +1. Go to [Mendix App Gallery](https://appgallery.mendixcloud.com/) +2. Download each demo app listed above +3. Open each `.mpk` in Studio Pro to extract the `.mpr` file + +### 2. Build mxcli + +```bash +make build && make test && make lint-go +``` + +### 3. Smoke test + +```bash +APPS_DIR= +for mpr in "$APPS_DIR"/*/*.mpr; do + echo "=== $(basename $(dirname $mpr)) ===" + echo "show odata clients;" > /tmp/show-int.mdl + mxcli exec /tmp/show-int.mdl -p "$mpr" 2>&1 | tail -1 +done +``` + +### 4. Interactive testing + +```bash +mxcli repl -p /EnquiriesManagement.mpr +``` + +### 5. Script-based testing + +```bash +mxcli exec test-sequence.mdl -p +``` + +Write operations (CREATE, DROP, GRANT/REVOKE) modify the `.mpr` file **in place**. + +> **IMPORTANT:** Always run destructive tests against a **copy** of the project folder, +> never the original. The `.mpr` file references other files in the project directory, +> and objects that are DROPped cannot be recovered — there is no undo, no git history, +> and no Studio Pro autosave for `.mpr` files. +> +> ```bash +> # Before each destructive test session +> cp -r MyProject MyProject-test +> mxcli repl -p MyProject-test/MyProject.mpr +> ``` + +--- + +## 1. SHOW ODATA CLIENTS + +### 1.1 List all OData clients + +``` +show odata clients; +``` + +**Expected:** All consumed OData services listed. Count matches Studio Pro. + +### 1.2 Filter by module + +``` +show odata clients in MyModule; +``` + +**Expected:** Only OData clients from `MyModule`. + +### 1.3 Empty module + +``` +show odata clients in ModuleWithNoClients; +``` + +**Expected:** Empty result, no error. + +### 1.4 Non-existent module + +``` +show odata clients in NonExistentModule; +``` + +**Expected:** Empty result, no error. Consistent with other SHOW IN commands for empty modules. + +--- + +## 2. DESCRIBE ODATA CLIENT + +### 2.1 Describe existing client + +``` +describe odata client MyModule.MyODataClient; +``` + +**Expected:** Valid `create or modify odata client` MDL output. Includes Version, ODataVersion, MetadataUrl, Timeout, ProxyType, ServiceUrl, Authentication, Headers. + +### 2.2 Non-existent client + +``` +describe odata client MyModule.DoesNotExist; +``` + +**Expected:** Clear error message. + +### 2.3 All properties present + +Pick a client with all properties configured. Verify DESCRIBE output includes: + +| Property | Format | +|----------|--------| +| Version | String literal | +| ODataVersion | `3` or `4` | +| MetadataUrl | URL string | +| Timeout | Integer (seconds) | +| ProxyType | `default` / `none` / `custom` | +| ServiceUrl | `@ConstantName` reference | +| Authentication | `none` / `basic` / `custom` | +| Headers | Block with key-value pairs | + +--- + +## 3. CREATE ODATA CLIENT + +### 3.1 Full property set + +``` +create odata client MyModule.NewClient ( + Version: '1.0', + ODataVersion: 4, + MetadataUrl: 'https://api.example.com/odata/v4/$metadata', + Timeout: 300, + ProxyType: default, + ServiceUrl: @MyModule.ServiceUrlConstant, + Authentication: none, + Headers: + 'X-Custom-Header' = 'value1', + 'Accept' = 'application/json' + end headers +); +``` + +**Expected:** Created. Listed in `show odata clients`. DESCRIBE matches input. + +### 3.2 Minimal client + +``` +create odata client MyModule.MinimalClient ( + MetadataUrl: 'https://api.example.com/odata/$metadata' +); +``` + +**Expected:** Created with defaults for omitted properties. + +### 3.3 With basic authentication + +``` +create odata client MyModule.AuthClient ( + MetadataUrl: 'https://api.example.com/odata/$metadata', + Authentication: basic +); +``` + +**Expected:** Authentication set to basic. Roundtrips via DESCRIBE. + +### 3.4 Duplicate without OR MODIFY + +``` +create odata client MyModule.NewClient ( + MetadataUrl: 'https://api.example.com/$metadata' +); +create odata client MyModule.NewClient ( + MetadataUrl: 'https://api.example.com/$metadata' +); +``` + +**Expected:** Second CREATE fails with "already exists" error. + +### 3.5 Target module must exist + +``` +create odata client NewModule.TestClient ( + MetadataUrl: 'https://api.example.com/$metadata' +); +``` + +**Expected:** Error if `NewModule` does not exist. Modules must be created beforehand. + +### 3.6 Write guard + +Attempt CREATE without opening a project for writing. + +**Expected:** Error about not being connected. + +--- + +## 4. CREATE OR MODIFY ODATA CLIENT + +### 4.1 Upsert — new client + +``` +create or modify odata client MyModule.UpsertClient ( + MetadataUrl: 'https://api.example.com/$metadata', + Timeout: 60 +); +``` + +**Expected:** Created. No error. + +### 4.2 Upsert — existing client + +``` +create or modify odata client MyModule.UpsertClient ( + MetadataUrl: 'https://api.example.com/v2/$metadata', + Timeout: 120 +); +``` + +**Expected:** Updated. ID reused. DESCRIBE shows new values. + +--- + +## 5. ALTER ODATA CLIENT + +### 5.1 Set single property + +``` +alter odata client MyModule.ExistingClient set timeout = 600; +``` + +**Expected:** Only timeout changed. Other properties unchanged. + +### 5.2 Set metadata URL + +``` +alter odata client MyModule.ExistingClient set MetadataUrl = 'https://new.example.com/$metadata'; +``` + +**Expected:** URL updated. DESCRIBE confirms. + +### 5.3 Alter non-existent client + +``` +alter odata client MyModule.DoesNotExist set timeout = 60; +``` + +**Expected:** Clear error. + +--- + +## 6. DROP ODATA CLIENT + +### 6.1 Drop existing client + +``` +create odata client MyModule.ToDrop ( + MetadataUrl: 'https://example.com/$metadata' +); +drop odata client MyModule.ToDrop; +``` + +**Expected:** Removed. Not in `show odata clients`. + +### 6.2 Cascade — deletes external entities + +1. CREATE odata client +2. CREATE external entity from that client +3. DROP odata client + +**Expected:** External entities sourced from this client also removed. + +### 6.3 Drop non-existent client + +``` +drop odata client MyModule.DoesNotExist; +``` + +**Expected:** Clear error. + +### 6.4 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 7. SHOW ODATA SERVICES + +### 7.1 List all published OData services + +``` +show odata services; +``` + +**Expected:** All published OData services listed. Count matches Studio Pro. + +### 7.2 Filter by module + +``` +show odata services in MyModule; +``` + +**Expected:** Only OData services from `MyModule`. + +### 7.3 Empty module + +``` +show odata services in ModuleWithNoServices; +``` + +**Expected:** Empty result, no error. + +### 7.4 Non-existent module + +``` +show odata services in NonExistentModule; +``` + +**Expected:** Empty result, no error. Consistent with other SHOW IN commands for empty modules. + +--- + +## 8. DESCRIBE ODATA SERVICE + +### 8.1 Describe existing service + +``` +describe odata service MyModule.MyODataService; +``` + +**Expected:** Valid `create or modify odata service` MDL output. Includes Path, Version, ODataVersion, Namespace, Authentication, published entities with expose blocks. + +### 8.2 Non-existent service + +``` +describe odata service MyModule.DoesNotExist; +``` + +**Expected:** Clear error message. + +### 8.3 Published entity details + +Verify DESCRIBE output for a service with published entities includes: + +| Property | Format | +|----------|--------| +| PUBLISH ENTITY | Qualified entity name | +| EXPOSE block | Attribute list with types | +| ReadMode | `default` / `fromDatabase` / `fromMicroflow` | +| InsertMode | `default` / `fromMicroflow` | +| UpdateMode | `default` / `fromMicroflow` | +| DeleteMode | `default` / `fromMicroflow` | +| UsePaging | `true` / `false` | +| PageSize | Integer | + +--- + +## 9. CREATE ODATA SERVICE + +### 9.1 Full service with published entity + +``` +create odata service MyModule.NewService ( + Path: '/odata/v1/myservice', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example.myservice', + Authentication: none +) + publish entity MyModule.Customer + expose + Name, + Email, + PhoneNumber + end expose + ReadMode: default, + InsertMode: default, + UpdateMode: default, + DeleteMode: default, + UsePaging: true, + PageSize: 20; +``` + +**Expected:** Created. Listed in `show odata services`. DESCRIBE matches input. + +### 9.2 Multiple published entities + +``` +create odata service MyModule.MultiEntityService ( + Path: '/odata/v1/multi', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example.multi', + Authentication: none +) + publish entity MyModule.Customer + expose Name, Email end expose + ReadMode: default, + UsePaging: true, + PageSize: 50 + publish entity MyModule.Order + expose OrderNumber, TotalAmount end expose + ReadMode: default, + UsePaging: true, + PageSize: 100; +``` + +**Expected:** Both entities published. DESCRIBE lists both. + +### 9.3 Duplicate without OR MODIFY + +``` +create odata service MyModule.NewService ( + Path: '/odata/v1/svc', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example' +); +create odata service MyModule.NewService ( + Path: '/odata/v1/svc', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example' +); +``` + +**Expected:** Second CREATE fails with "already exists" error. + +### 9.4 Target module must exist + +``` +create odata service NewModule.TestService ( + Path: '/odata/v1/test', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example.test' +); +``` + +**Expected:** Error if `NewModule` does not exist. Modules must be created beforehand. + +### 9.5 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 10. CREATE OR MODIFY ODATA SERVICE + +### 10.1 Upsert — new service + +``` +create or modify odata service MyModule.UpsertService ( + Path: '/odata/v1/upsert', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example.upsert' +); +``` + +**Expected:** Created. No error. + +### 10.2 Upsert — existing service + +``` +create or modify odata service MyModule.UpsertService ( + Path: '/odata/v2/upsert', + Version: '2.0.0', + ODataVersion: 4, + Namespace: 'com.example.upsert.v2' +); +``` + +**Expected:** Updated. ID reused. DESCRIBE shows new values. + +--- + +## 11. ALTER ODATA SERVICE + +### 11.1 Set single property + +``` +alter odata service MyModule.ExistingService set version = '2.0.0'; +``` + +**Expected:** Only version changed. Other properties unchanged. + +### 11.2 Set path + +``` +alter odata service MyModule.ExistingService set path = '/odata/v2/updated'; +``` + +**Expected:** Path updated. DESCRIBE confirms. + +### 11.3 Alter non-existent service + +``` +alter odata service MyModule.DoesNotExist set version = '1.0.0'; +``` + +**Expected:** Clear error. + +--- + +## 12. DROP ODATA SERVICE + +### 12.1 Drop existing service + +``` +create odata service MyModule.ToDrop ( + Path: '/odata/v1/drop', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example.drop' +); +drop odata service MyModule.ToDrop; +``` + +**Expected:** Removed. Not in `show odata services`. + +### 12.2 Drop non-existent service + +``` +drop odata service MyModule.DoesNotExist; +``` + +**Expected:** Clear error. + +### 12.3 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 13. GRANT / REVOKE ACCESS ON ODATA SERVICE + +### 13.1 Grant to single role + +``` +grant access on odata service MyModule.MyService to MyModule.User; +``` + +**Expected:** Role granted. Verifiable via DESCRIBE or show access. + +### 13.2 Grant to multiple roles + +``` +grant access on odata service MyModule.MyService to MyModule.User, MyModule.Admin; +``` + +**Expected:** Both roles added. + +### 13.3 Idempotent grant + +Grant same role twice. + +**Expected:** No duplicate entries on second grant. + +### 13.4 Revoke from role + +``` +revoke access on odata service MyModule.MyService from MyModule.User; +``` + +**Expected:** Role removed. + +### 13.5 Grant on non-existent service + +**Expected:** Clear error. + +### 13.6 Grant with non-existent role + +**Expected:** Clear error. + +--- + +## 14. SHOW EXTERNAL ENTITIES + +### 14.1 List all external entities + +``` +show external entities; +``` + +**Expected:** All external entities listed. Count matches Studio Pro. + +### 14.2 Filter by module + +``` +show external entities in MyModule; +``` + +**Expected:** Only external entities from `MyModule`. + +### 14.3 Empty module + +``` +show external entities in ModuleWithNoExternals; +``` + +**Expected:** Empty result, no error. + +### 14.4 Non-existent module + +``` +show external entities in NonExistentModule; +``` + +**Expected:** Empty result, no error. Consistent with other SHOW IN commands for empty modules. + +--- + +## 15. SHOW EXTERNAL ACTIONS + +### 15.1 List all external actions + +``` +show external actions; +``` + +**Expected:** All external actions listed. Count matches Studio Pro. + +--- + +## 16. DESCRIBE EXTERNAL ENTITY + +### 16.1 Describe existing external entity + +``` +describe external entity MyModule.MyExternalEntity; +``` + +**Expected:** Valid MDL output. Includes source OData client, EntitySet, RemoteName, attributes, Countable, Creatable, Deletable, Updatable flags. + +### 16.2 Non-existent external entity + +``` +describe external entity MyModule.DoesNotExist; +``` + +**Expected:** Clear error message. + +### 16.3 All properties present + +Verify DESCRIBE output includes: + +| Property | Format | +|----------|--------| +| EntitySet | String | +| RemoteName | String | +| Countable | `true` / `false` | +| Creatable | `true` / `false` | +| Deletable | `true` / `false` | +| Updatable | `true` / `false` | +| Attributes | Name + type list | + +--- + +## 17. CREATE EXTERNAL ENTITY FROM ODATA CLIENT + +### 17.1 Single entity with full properties + +``` +create external entity MyModule.ExternalCustomer from odata client MyModule.MyODataClient ( + EntitySet: 'Customers', + RemoteName: 'Customer', + Countable: true, + Creatable: true, + Deletable: false, + Updatable: true +) ( + Name: String(200), + Email: String(200), + Age: Integer, + Active: Boolean +); +``` + +**Expected:** Created. Listed in `show external entities`. DESCRIBE matches input. + +### 17.2 Minimal entity + +``` +create external entity MyModule.ExternalOrder from odata client MyModule.MyODataClient ( + EntitySet: 'Orders', + RemoteName: 'Order' +) ( + OrderId: Integer, + Total: Decimal +); +``` + +**Expected:** Created with defaults for omitted boolean flags. + +### 17.3 Non-existent source client + +``` +create external entity MyModule.BadSource from odata client MyModule.NonExistentClient ( + EntitySet: 'Items', + RemoteName: 'Item' +) ( + Id: Integer +); +``` + +**Expected:** Error — source OData client not found. + +### 17.4 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 18. CREATE OR MODIFY EXTERNAL ENTITY + +### 18.1 Upsert — new entity + +``` +create or modify external entity MyModule.UpsertExternal from odata client MyModule.MyODataClient ( + EntitySet: 'Products', + RemoteName: 'Product' +) ( + ProductId: Integer, + Name: String(200) +); +``` + +**Expected:** Created. No error. + +### 18.2 Upsert — existing entity + +``` +create or modify external entity MyModule.UpsertExternal from odata client MyModule.MyODataClient ( + EntitySet: 'Products', + RemoteName: 'Product', + Countable: true +) ( + ProductId: Integer, + Name: String(200), + Price: Decimal +); +``` + +**Expected:** Updated. ID reused. DESCRIBE shows new attributes. + +--- + +## 19. CREATE EXTERNAL ENTITIES FROM CLIENT (bulk import) + +> **Not implemented (#423):** The `create external entities from odata client` syntax is not yet implemented. These test cases document the expected behavior for when it lands. + +### 19.1 Bulk import all entities + +``` +create external entities from odata client MyModule.MyODataClient; +``` + +**Expected:** All entities from the OData service metadata imported. Each listed in `show external entities`. + +### 19.2 Selective import into target module + +``` +create or modify external entities + from odata client MyModule.MyODataClient + entities (Customer, Order, Product) + into TargetModule; +``` + +**Expected:** Only Customer, Order, Product imported. Placed in `TargetModule`. + +### 19.3 Non-existent entity name in list + +``` +create or modify external entities + from odata client MyModule.MyODataClient + entities (Customer, NonExistentEntity) + into TargetModule; +``` + +**Expected:** Error for NonExistentEntity. Customer may or may not be created depending on fail-fast behavior. + +### 19.4 Non-existent source client + +``` +create external entities from odata client MyModule.NonExistent; +``` + +**Expected:** Clear error. + +--- + +## 20. SHOW PUBLISHED REST SERVICES + +> **Feature gate:** Mendix 10.0+. Test against Evora (10.24.15) and Lato Enquiry (11.4.0). + +### 20.1 List all published REST services + +``` +show published rest services; +``` + +**Expected:** All published REST services listed. Count matches Studio Pro. + +### 20.2 Filter by module + +``` +show published rest services in MyModule; +``` + +**Expected:** Only REST services from `MyModule`. + +### 20.3 Empty module + +``` +show published rest services in ModuleWithNoREST; +``` + +**Expected:** Empty result, no error. + +### 20.4 Non-existent module + +``` +show published rest services in NonExistentModule; +``` + +**Expected:** Empty result, no error. Consistent with other SHOW IN commands for empty modules. + +--- + +## 21. DESCRIBE PUBLISHED REST SERVICE + +### 21.1 Describe existing service + +``` +describe published rest service MyModule.MyRestService; +``` + +**Expected:** Valid `create or replace published rest service` MDL output. Includes Path, Version, ServiceName, resource blocks with HTTP methods and microflow references. + +### 21.2 Non-existent service + +``` +describe published rest service MyModule.DoesNotExist; +``` + +**Expected:** Clear error message. + +### 21.3 Resource detail + +Verify DESCRIBE includes for each resource: + +| Property | Format | +|----------|--------| +| Resource path | String | +| HTTP method | `GET` / `POST` / `PUT` / `PATCH` / `DELETE` | +| Microflow | Qualified `Module.Microflow` reference | +| Deprecated | `deprecated` keyword if applicable | + +--- + +## 22. CREATE PUBLISHED REST SERVICE + +### 22.1 Full service with resources + +``` +create published rest service MyModule.CustomerAPI ( + Path: '/api/v1/customers', + Version: '1.0.0', + ServiceName: 'CustomerAPI' +) + resource '/customers' { + get microflow MyModule.GetCustomers; + post microflow MyModule.CreateCustomer; + } + resource '/customers/{id}' { + get microflow MyModule.GetCustomerById; + put microflow MyModule.UpdateCustomer; + delete microflow MyModule.DeleteCustomer; + }; +``` + +**Expected:** Created. Listed in `show published rest services`. DESCRIBE matches input. + +> **Note (#429):** The `end resource` syntax causes a crash. Always use brace syntax `resource 'path' { ... }` instead. + +### 22.2 Single resource, single method + +``` +create published rest service MyModule.SimpleAPI ( + Path: '/api/v1/health', + Version: '1.0.0', + ServiceName: 'HealthCheck' +) + resource '/health' { + get microflow MyModule.HealthCheck; + }; +``` + +**Expected:** Created with minimal configuration. + +### 22.3 Deprecated resource method + +``` +create published rest service MyModule.LegacyAPI ( + Path: '/api/v1/legacy', + Version: '1.0.0', + ServiceName: 'LegacyAPI' +) + resource '/old-endpoint' { + deprecated get microflow MyModule.OldEndpoint; + }; +``` + +**Expected:** Deprecated flag preserved. DESCRIBE shows `deprecated` keyword. + +### 22.4 Duplicate without OR REPLACE + +``` +create published rest service MyModule.CustomerAPI ( + Path: '/api/v1/dup', + Version: '1.0.0', + ServiceName: 'Dup' +); +create published rest service MyModule.CustomerAPI ( + Path: '/api/v1/dup', + Version: '1.0.0', + ServiceName: 'Dup' +); +``` + +**Expected:** Second CREATE fails with "already exists" error. + +> **BUG-017:** Currently duplicate creation is allowed for published REST services. Second CREATE succeeds and creates a duplicate entry instead of returning an error. + +### 22.5 Module auto-creation + +``` +create published rest service NewModule.TestAPI ( + Path: '/api/v1/test', + Version: '1.0.0', + ServiceName: 'TestAPI' +); +``` + +**Expected:** `NewModule` created automatically if it doesn't exist. + +### 22.6 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 23. CREATE OR REPLACE PUBLISHED REST SERVICE + +### 23.1 Upsert — new service + +``` +create or replace published rest service MyModule.UpsertAPI ( + Path: '/api/v1/upsert', + Version: '1.0.0', + ServiceName: 'UpsertAPI' +) + resource '/items' { + get microflow MyModule.GetItems; + }; +``` + +**Expected:** Created. No error. + +### 23.2 Upsert — existing service + +``` +create or replace published rest service MyModule.UpsertAPI ( + Path: '/api/v2/upsert', + Version: '2.0.0', + ServiceName: 'UpsertAPI' +) + resource '/items' { + get microflow MyModule.GetItemsV2; + post microflow MyModule.CreateItem; + }; +``` + +**Expected:** Replaced. ID reused. DESCRIBE shows new values. + +--- + +## 24. ALTER PUBLISHED REST SERVICE + +### 24.1 Set property + +``` +alter published rest service MyModule.MyAPI set version = '2.0.0'; +``` + +**Expected:** Only version changed. Resources unchanged. + +### 24.2 Add resource + +``` +alter published rest service MyModule.MyAPI + add resource '/new-resource' { + get microflow MyModule.GetNewResource; + }; +``` + +**Expected:** New resource added. Existing resources unchanged. + +### 24.3 Drop resource + +``` +alter published rest service MyModule.MyAPI + drop resource '/old-resource'; +``` + +**Expected:** Resource removed. Other resources unchanged. + +### 24.4 Alter non-existent service + +``` +alter published rest service MyModule.DoesNotExist set version = '1.0.0'; +``` + +**Expected:** Clear error. + +--- + +## 25. DROP PUBLISHED REST SERVICE + +### 25.1 Drop existing service + +``` +drop published rest service MyModule.CustomerAPI; +``` + +**Expected:** Removed. Not in `show published rest services`. + +### 25.2 Drop non-existent service + +``` +drop published rest service MyModule.DoesNotExist; +``` + +**Expected:** Clear error. + +### 25.3 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 26. GRANT / REVOKE ACCESS ON PUBLISHED REST SERVICE + +### 26.1 Grant to single role + +``` +grant access on published rest service MyModule.MyAPI to MyModule.User; +``` + +**Expected:** Role granted. + +### 26.2 Grant to multiple roles + +``` +grant access on published rest service MyModule.MyAPI to MyModule.User, MyModule.Admin; +``` + +**Expected:** Both roles added. + +### 26.3 Revoke from role + +``` +revoke access on published rest service MyModule.MyAPI from MyModule.User; +``` + +**Expected:** Role removed. + +### 26.4 Grant on non-existent service + +**Expected:** Clear error. + +### 26.5 Grant with non-existent role + +**Expected:** Clear error. + +--- + +## 27. SHOW REST CLIENTS + +> **Feature gate:** Mendix 10.1+. Test against Lato Enquiry (11.4.0) and Evora (10.24.15). + +### 27.1 List all REST clients + +``` +show rest clients; +``` + +**Expected:** All consumed REST services listed. Count matches Studio Pro. + +### 27.2 Filter by module + +``` +show rest clients in MyModule; +``` + +**Expected:** Only REST clients from `MyModule`. + +### 27.3 Empty module + +``` +show rest clients in ModuleWithNoREST; +``` + +**Expected:** Empty result, no error. + +### 27.4 Non-existent module + +``` +show rest clients in NonExistentModule; +``` + +**Expected:** Empty result, no error. Consistent with other SHOW IN commands for empty modules. + +--- + +## 28. DESCRIBE REST CLIENT + +### 28.1 Describe existing client + +``` +describe rest client MyModule.MyRestClient; +``` + +**Expected:** Valid `create or modify rest client` MDL output. Includes BaseUrl, Authentication, operation blocks. + +### 28.2 Non-existent client + +``` +describe rest client MyModule.DoesNotExist; +``` + +**Expected:** Clear error message. + +### 28.3 Operation detail + +Verify DESCRIBE output for each operation includes: + +| Property | Format | +|----------|--------| +| Method | `GET` / `POST` / `PUT` / `PATCH` / `DELETE` | +| Path | String with `{param}` placeholders | +| Parameters | Name + type list | +| Query | Key-value pairs | +| Headers | Key-value pairs | +| Body | Type reference or inline | +| Timeout | Integer (seconds) | +| Response | Type reference | + +--- + +## 29. CREATE REST CLIENT (manual) + +### 29.1 Full client with operations + +> **SKIP:** The `operation...end operation` syntax below is not yet implemented in the parser. See [#429](https://github.com/mendixlabs/mxcli/issues/429). Test case retained for future validation. + +``` +create rest client MyModule.ExternalAPI ( + BaseUrl: 'https://api.example.com/v1', + Authentication: none +) + operation GetUsers + method GET + path '/users' + timeout 30 + response MyModule.UserListResponse + end operation + operation CreateUser + method POST + path '/users' + headers + 'Content-Type' = 'application/json' + end headers + body MyModule.CreateUserRequest + timeout 60 + response MyModule.UserResponse + end operation + operation GetUserById + method GET + path '/users/{userId}' + parameters + userId : String + end parameters + timeout 30 + response MyModule.UserResponse + end operation; +``` + +**Expected:** Created. Listed in `show rest clients`. DESCRIBE matches input. + +### 29.2 With basic authentication + +``` +create rest client MyModule.AuthAPI ( + BaseUrl: 'https://secure.example.com/api', + Authentication: basic +) + operation Ping + method GET + path '/ping' + timeout 10 + end operation; +``` + +**Expected:** Authentication set to basic. Roundtrips via DESCRIBE. + +### 29.3 With query parameters + +``` +create rest client MyModule.SearchAPI ( + BaseUrl: 'https://api.example.com', + Authentication: none +) + operation Search + method GET + path '/search' + query + 'q' = '$SearchTerm', + 'limit' = '10' + end query + timeout 30 + end operation; +``` + +**Expected:** Query parameters preserved in DESCRIBE. + +### 29.4 Duplicate without OR MODIFY + +``` +create rest client MyModule.ExternalAPI ( + BaseUrl: 'https://api.example.com' +); +create rest client MyModule.ExternalAPI ( + BaseUrl: 'https://api.example.com' +); +``` + +**Expected:** Second CREATE fails with "already exists" error. + +### 29.5 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 30. CREATE REST CLIENT FROM OPENAPI + +### 30.1 From local file + +``` +create rest client MyModule.PetStore + from openapi '/path/to/petstore.yaml'; +``` + +**Expected:** Client created with operations derived from OpenAPI spec. All paths and methods mapped. + +### 30.2 From URL + +``` +create rest client MyModule.PetStoreURL + from openapi 'https://petstore3.swagger.io/api/v3/openapi.json'; +``` + +**Expected:** Spec fetched and client created. Operations match spec paths. + +### 30.3 Invalid spec + +``` +create rest client MyModule.BadSpec + from openapi '/path/to/invalid.yaml'; +``` + +**Expected:** Clear error about spec parsing failure. + +### 30.4 Non-existent file + +``` +create rest client MyModule.NoFile + from openapi '/nonexistent/spec.yaml'; +``` + +**Expected:** Clear error about file not found. + +--- + +## 31. CREATE OR MODIFY REST CLIENT + +### 31.1 Upsert — new client + +``` +create or modify rest client MyModule.UpsertREST ( + BaseUrl: 'https://api.example.com', + Authentication: none +); +``` + +**Expected:** Created. No error. + +### 31.2 Upsert — existing client + +``` +create or modify rest client MyModule.UpsertREST ( + BaseUrl: 'https://api.example.com/v2', + Authentication: basic +); +``` + +**Expected:** Updated. ID reused. DESCRIBE shows new values. + +--- + +## 32. DROP REST CLIENT + +### 32.1 Drop existing client + +``` +drop rest client MyModule.ExternalAPI; +``` + +**Expected:** Removed. Not in `show rest clients`. + +### 32.2 Drop non-existent client + +``` +drop rest client MyModule.DoesNotExist; +``` + +**Expected:** Clear error. + +### 32.3 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 33. DESCRIBE CONTRACT OPERATION FROM OPENAPI + +### 33.1 Preview from local file + +``` +describe contract operation from openapi '/path/to/petstore.yaml'; +``` + +**Expected:** Lists all operations from the spec without creating anything. Shows method, path, parameters, request/response types. + +### 33.2 Preview from URL + +``` +describe contract operation from openapi 'https://petstore3.swagger.io/api/v3/openapi.json'; +``` + +**Expected:** Same as §33.1 but fetched from URL. + +### 33.3 Invalid spec + +``` +describe contract operation from openapi '/path/to/invalid.yaml'; +``` + +**Expected:** Clear error about spec parsing failure. + +--- + +## 34. SHOW JSON STRUCTURES + +### 34.1 List all JSON structures + +``` +show json structures; +``` + +**Expected:** All JSON structures listed. Count matches Studio Pro. + +### 34.2 Filter by module + +``` +show json structures in MyModule; +``` + +**Expected:** Only JSON structures from `MyModule`. + +### 34.3 Empty module + +``` +show json structures in ModuleWithNoJSON; +``` + +**Expected:** Empty result, no error. + +### 34.4 Non-existent module + +``` +show json structures in NonExistentModule; +``` + +**Expected:** Empty result, no error. Consistent with other SHOW IN commands for empty modules. + +--- + +## 35. DESCRIBE JSON STRUCTURE + +### 35.1 Describe existing structure + +``` +describe json structure MyModule.MyJsonStructure; +``` + +**Expected:** Valid `create or replace json structure` MDL output. Includes JSON snippet, folder, comment, custom name map. + +### 35.2 Non-existent structure + +``` +describe json structure MyModule.DoesNotExist; +``` + +**Expected:** Clear error message. + +--- + +## 36. CREATE JSON STRUCTURE + +### 36.1 With single-quoted snippet + +``` +create json structure MyModule.UserSchema + snippet '{"name": "string", "age": 0, "active": true}'; +``` + +**Expected:** Created. Listed in `show json structures`. DESCRIBE matches input. + +### 36.2 With dollar-quoting + +``` +create json structure MyModule.ComplexSchema + $$ + { + "users": [ + { + "id": 0, + "name": "string", + "email": "string", + "address": { + "street": "string", + "city": "string", + "zip": "string" + } + } + ], + "total": 0, + "page": 0 + } + $$; +``` + +**Expected:** Created. Complex nested structure preserved. DESCRIBE shows full JSON. + +> **Note:** Multiline `$$` quoting is not supported by the parser. Use single-line `$${"users":[...]}$$` or single-quoted JSON strings instead. + +### 36.3 With folder and comment + +``` +create json structure MyModule.DocumentedSchema + folder 'API/Schemas' + comment 'Schema for the user creation endpoint' + '{"username": "string", "password": "string"}'; +``` + +**Expected:** Folder and comment preserved in DESCRIBE. + +### 36.4 With custom name map + +``` +create json structure MyModule.MappedSchema + '{"user-name": "string", "e-mail": "string"}' + custom name map + 'user-name' as 'UserName', + 'e-mail' as 'Email' + end custom name map; +``` + +**Expected:** Custom name map preserved. DESCRIBE shows mappings. + +### 36.5 Duplicate without OR REPLACE + +``` +create json structure MyModule.UserSchema '{"a": 1}'; +create json structure MyModule.UserSchema '{"a": 1}'; +``` + +**Expected:** Second CREATE fails with "already exists" error. + +### 36.6 Module auto-creation + +``` +create json structure NewModule.TestSchema '{"test": true}'; +``` + +**Expected:** `NewModule` created automatically if it doesn't exist. + +### 36.7 Write guard + +**Expected:** Error if no project open for writing. + +### 36.8 Invalid JSON snippet + +``` +create json structure MyModule.BadJSON '{invalid json}'; +``` + +**Expected:** Clear error about JSON parsing failure. + +--- + +## 37. CREATE OR REPLACE JSON STRUCTURE + +### 37.1 Upsert — new structure + +``` +create or replace json structure MyModule.UpsertSchema + '{"value": "string"}'; +``` + +**Expected:** Created. No error. + +### 37.2 Upsert — existing structure + +``` +create or replace json structure MyModule.UpsertSchema + '{"value": "string", "count": 0}'; +``` + +**Expected:** Replaced. DESCRIBE shows new schema. + +--- + +## 38. DROP JSON STRUCTURE + +### 38.1 Drop existing structure + +``` +drop json structure MyModule.UserSchema; +``` + +**Expected:** Removed. Not in `show json structures`. + +### 38.2 Drop non-existent structure + +``` +drop json structure MyModule.DoesNotExist; +``` + +**Expected:** Clear error. + +### 38.3 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 39. ROUNDTRIP (BSON) + +### 39.1 OData Client roundtrip + +1. DESCRIBE odata client → capture MDL +2. DROP odata client +3. Execute captured MDL +4. DESCRIBE again → capture +5. Diff the two outputs + +**Expected:** Identical or cosmetic-only differences. + +### 39.2 OData Service roundtrip + +1. DESCRIBE odata service → capture MDL +2. DROP odata service +3. Execute captured MDL +4. DESCRIBE again → capture +5. Diff + +**Expected:** Identical or cosmetic-only differences. + +> **BUG-016:** OData service DESCRIBE is not fully roundtrippable. The `PUBLISH ENTITY` syntax in the output may not parse correctly when re-executed. Manual adjustment may be needed. + +1. DESCRIBE external entity → capture MDL +2. Execute captured MDL via CREATE OR MODIFY +3. DESCRIBE again → capture +4. Diff + +**Expected:** Identical output. + +### 39.4 Published REST Service roundtrip + +1. DESCRIBE published rest service → capture MDL +2. DROP published rest service +3. Execute captured MDL +4. DESCRIBE again → capture +5. Diff + +**Expected:** Identical or cosmetic-only differences. + +### 39.5 REST Client roundtrip + +1. DESCRIBE rest client → capture MDL +2. DROP rest client +3. Execute captured MDL +4. DESCRIBE again → capture +5. Diff + +**Expected:** Identical or cosmetic-only differences. + +> **Note:** REST client body/response mappings are simplified on roundtrip — complex request body and response type definitions may lose detail. This is a known data loss issue during DESCRIBE→CREATE cycles. + +1. DESCRIBE json structure → capture MDL +2. DROP json structure +3. Execute captured MDL +4. DESCRIBE again → capture +5. Diff + +**Expected:** Identical output. JSON whitespace normalization acceptable. + +### 39.7 Bulk roundtrip + +Run §39.1–§39.6 on all instances across 3 test projects. Record pass/fail counts. + +--- + +## 40. MULTI-STEP WORKFLOWS + +### 40.1 End-to-end REST integration + +1. CREATE json structure for request/response schemas +2. CREATE rest client with operations referencing the JSON structures +3. CREATE microflow calling the REST client operation +4. DESCRIBE each artifact — verify cross-references intact + +### 40.2 OData publish-consume loop + +1. CREATE odata service publishing entity `MyModule.Product` +2. CREATE odata client pointing to the service metadata URL +3. CREATE external entity from the OData client +4. DESCRIBE each — verify chain intact + +### 40.3 Scaffold module with all integration types + +1. CREATE odata client in `IntegrationModule` +2. CREATE external entities from client +3. CREATE published rest service exposing data +4. CREATE json structure for API contract +5. SHOW each type in `IntegrationModule` — verify all present +6. DESCRIBE each — verify complete MDL output + +### 40.4 Iterative service evolution + +1. CREATE published rest service with 1 resource, 1 method +2. ALTER to add second resource +3. ALTER to add method to first resource +4. ALTER to set new version +5. After each step: DESCRIBE and verify cumulative changes + +### 40.5 Drop and recreate with different configuration + +1. CREATE rest client with 2 operations +2. DROP +3. CREATE same name with 3 different operations +4. DESCRIBE — verify new configuration, no remnant of old operations + +--- + +## 41. FAILURE MODES + +### 41.1 Not connected + +Attempt each write command without opening a project: + +| Command | Expected | +|---------|----------| +| `create odata client ...` | Error: not connected | +| `create odata service ...` | Error: not connected | +| `create external entity ...` | Error: not connected | +| `create published rest service ...` | Error: not connected | +| `create rest client ...` | Error: not connected | +| `create json structure ...` | Error: not connected | + +### 41.2 Entity not found + +``` +create odata service MyModule.BadService ( + Path: '/odata/v1/bad', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example' +) + publish entity MyModule.NonExistentEntity + expose Name end expose; +``` + +**Expected:** Error — entity not found. + +### 41.3 Duplicate without OR MODIFY/REPLACE + +Test for each type. See §3.4, §9.3, §22.4, §29.4, §36.5. + +### 41.4 Invalid metadata URL + +``` +create odata client MyModule.BadUrl ( + MetadataUrl: 'not-a-url' +); +``` + +**Expected:** Error about invalid URL format or connection failure. + +### 41.5 Unsupported driver version + +Open a pre-10.0 `.mpr` file. Attempt: + +``` +show published rest services; +``` + +**Expected:** Clear error about feature gate — Published REST requires Mendix 10.0+. + +### 41.6 Feature gate violation — REST client + +Open a pre-10.1 `.mpr` file. Attempt: + +``` +show rest clients; +``` + +**Expected:** Clear error about feature gate — REST Clients require Mendix 10.1+. + +### 41.7 Invalid JSON in structure + +``` +create json structure MyModule.Bad '{not json at all'; +``` + +**Expected:** Clear error about JSON syntax. + +### 41.8 Non-existent microflow in REST service + +``` +create published rest service MyModule.BadRef ( + Path: '/api/v1/bad', + Version: '1.0.0', + ServiceName: 'BadRef' +) + resource '/items' { + get microflow MyModule.NonExistentMicroflow; + }; +``` + +**Expected:** Error — microflow not found. + +### 41.9 Double DROP + +``` +drop rest client MyModule.X; +drop rest client MyModule.X; +``` + +**Expected:** First succeeds, second gives "not found" error. + +### 41.10 Validation failure mid-batch + +``` +create json structure MyModule.Good1 '{"ok": true}'; +create json structure MyModule.Bad '{invalid}'; +create json structure MyModule.Good3 '{"ok": true}'; +``` + +**Expected:** Good1 created, Bad rejected, Good3 NOT created — batch aborts on first error. + +> Batch mode (`mxcli exec`) is fail-fast. REPL mode continues on error per-line. + +### 41.11 Error message quality + +For each error scenario, verify the message includes: +- **What** went wrong +- **Which** object (qualified name) +- **Actionable guidance** where applicable + +--- + +## Test Project Coverage Matrix + +| Category | Enquiries (11.4.0) | Evora Factory (10.24.15) | Lato Inventory (11.2.0) | +|---|---|---|---| +| SHOW OData clients | Verify count | Verify count | Verify count | +| SHOW OData services | Verify count | Verify count | Verify count | +| SHOW external entities | Verify count | Verify count | Verify count | +| SHOW published REST | Verify count | Verify count | Verify count | +| SHOW REST clients | Verify count | Verify count | Verify count | +| SHOW JSON structures | Verify count | Verify count | Verify count | +| DESCRIBE (sample 5+ each) | All 6 types | All 6 types | All 6 types | +| Roundtrip (sample 5+ each) | All 6 types | All 6 types | All 6 types | +| CREATE + DROP cycle | All 6 types | All 6 types | All 6 types | +| Feature gate (REST) | ✅ 10.0+ | ✅ 10.0+ | ✅ 10.0+ | +| Feature gate (REST client) | ✅ 10.1+ | ✅ 10.1+ | ✅ 10.1+ | +| Multi-step workflows (§40) | End-to-end | End-to-end | End-to-end | +| Security (GRANT/REVOKE) | OData + REST | OData + REST | OData + REST | + +--- + +## Automated Test Coverage + +| Area | Tests | Status | +|---|---|---| +| SHOW ODATA CLIENTS | None | **Gap** | +| DESCRIBE ODATA CLIENT | None | **Gap** | +| CREATE ODATA CLIENT | None | **Gap** | +| DROP ODATA CLIENT | None | **Gap** | +| SHOW ODATA SERVICES | None | **Gap** | +| DESCRIBE ODATA SERVICE | None | **Gap** | +| CREATE ODATA SERVICE | None | **Gap** | +| DROP ODATA SERVICE | None | **Gap** | +| GRANT/REVOKE ODATA SERVICE | None | **Gap** | +| SHOW EXTERNAL ENTITIES | None | **Gap** | +| DESCRIBE EXTERNAL ENTITY | None | **Gap** | +| CREATE EXTERNAL ENTITY | None | **Gap** | +| CREATE EXTERNAL ENTITIES (bulk) | None | **Gap** | +| SHOW PUBLISHED REST SERVICES | None | **Gap** | +| DESCRIBE PUBLISHED REST SERVICE | None | **Gap** | +| CREATE PUBLISHED REST SERVICE | None | **Gap** | +| DROP PUBLISHED REST SERVICE | None | **Gap** | +| GRANT/REVOKE REST SERVICE | None | **Gap** | +| SHOW REST CLIENTS | None | **Gap** | +| DESCRIBE REST CLIENT | None | **Gap** | +| CREATE REST CLIENT (manual) | None | **Gap** | +| CREATE REST CLIENT (OpenAPI) | None | **Gap** | +| DROP REST CLIENT | None | **Gap** | +| DESCRIBE CONTRACT OPERATION | None | **Gap** | +| SHOW JSON STRUCTURES | None | **Gap** | +| DESCRIBE JSON STRUCTURE | None | **Gap** | +| CREATE JSON STRUCTURE | None | **Gap** | +| DROP JSON STRUCTURE | None | **Gap** | +| Roundtrip (all 6 types) | None | **Gap** | +| Multi-step workflows (§40) | None | **Manual only** | +| Failure modes (§41) | None | **Manual only** | + +Manual testing priority: +1. Roundtrip all instances across 3 projects (bulk DESCRIBE→DROP→CREATE→DESCRIBE) +2. Feature gate enforcement (pre-10.0, pre-10.1 projects) +3. Multi-step workflows (§40) — highest interaction bug risk +4. Failure modes (§41) — especially §41.5, §41.6, §41.10 +5. GRANT/REVOKE on OData services and Published REST services + +--- + +## Manual Test Report Template + +Copy and fill in after running manual tests. + +```markdown +## Manual Testing + +**Date:** YYYY-MM-DD +**Build:** `make build && make test && make lint-go` — PASS + +### Test Projects + +| App | Studio Pro | SHOW counts (clients/services/entities/REST/JSON) | DESCRIBE sample | Roundtrip | +|-----|-----------|---------------------------------------------------|-----------------|-----------| +| Lato Enquiry Management | 11.4.0 | ✅ _/_/_/_/_ | ✅ _n_ tested | ✅ _n_ passed | +| Evora Factory Management | 10.24.15 | ✅ _/_/_/_/_ | ✅ _n_ tested | ✅ _n_ passed | +| Lato Product Inventory | 11.2.0 | ✅ _/_/_/_/_ | ✅ _n_ tested | ✅ _n_ passed | + +### OData Client Coverage + +| Command | Tested | Notes | +|---------|--------|-------| +| SHOW ODATA CLIENTS | ✅/❌ | | +| SHOW ODATA CLIENTS IN module | ✅/❌ | | +| DESCRIBE ODATA CLIENT | ✅/❌ | | +| CREATE ODATA CLIENT | ✅/❌ | | +| CREATE OR MODIFY ODATA CLIENT | ✅/❌ | | +| ALTER ODATA CLIENT | ✅/❌ | | +| DROP ODATA CLIENT | ✅/❌ | | + +### OData Service Coverage + +| Command | Tested | Notes | +|---------|--------|-------| +| SHOW ODATA SERVICES | ✅/❌ | | +| SHOW ODATA SERVICES IN module | ✅/❌ | | +| DESCRIBE ODATA SERVICE | ✅/❌ | | +| CREATE ODATA SERVICE | ✅/❌ | | +| CREATE OR MODIFY ODATA SERVICE | ✅/❌ | | +| ALTER ODATA SERVICE | ✅/❌ | | +| DROP ODATA SERVICE | ✅/❌ | | +| GRANT ACCESS ON ODATA SERVICE | ✅/❌ | | +| REVOKE ACCESS ON ODATA SERVICE | ✅/❌ | | + +### External Entity Coverage + +| Command | Tested | Notes | +|---------|--------|-------| +| SHOW EXTERNAL ENTITIES | ✅/❌ | | +| SHOW EXTERNAL ENTITIES IN module | ✅/❌ | | +| SHOW EXTERNAL ACTIONS | ✅/❌ | | +| DESCRIBE EXTERNAL ENTITY | ✅/❌ | | +| CREATE EXTERNAL ENTITY | ✅/❌ | | +| CREATE OR MODIFY EXTERNAL ENTITY | ✅/❌ | | +| CREATE EXTERNAL ENTITIES (bulk) | ✅/❌ | | +| CREATE OR MODIFY EXTERNAL ENTITIES (selective) | ✅/❌ | | + +### Published REST Service Coverage + +| Command | Tested | Notes | +|---------|--------|-------| +| SHOW PUBLISHED REST SERVICES | ✅/❌ | | +| SHOW PUBLISHED REST SERVICES IN module | ✅/❌ | | +| DESCRIBE PUBLISHED REST SERVICE | ✅/❌ | | +| CREATE PUBLISHED REST SERVICE | ✅/❌ | | +| CREATE OR REPLACE PUBLISHED REST SERVICE | ✅/❌ | | +| ALTER PUBLISHED REST SERVICE SET | ✅/❌ | | +| ALTER PUBLISHED REST SERVICE ADD RESOURCE | ✅/❌ | | +| ALTER PUBLISHED REST SERVICE DROP RESOURCE | ✅/❌ | | +| DROP PUBLISHED REST SERVICE | ✅/❌ | | +| GRANT ACCESS ON PUBLISHED REST SERVICE | ✅/❌ | | +| REVOKE ACCESS ON PUBLISHED REST SERVICE | ✅/❌ | | + +### REST Client Coverage + +| Command | Tested | Notes | +|---------|--------|-------| +| SHOW REST CLIENTS | ✅/❌ | | +| SHOW REST CLIENTS IN module | ✅/❌ | | +| DESCRIBE REST CLIENT | ✅/❌ | | +| CREATE REST CLIENT (manual) | ✅/❌ | | +| CREATE REST CLIENT FROM OPENAPI (file) | ✅/❌ | | +| CREATE REST CLIENT FROM OPENAPI (URL) | ✅/❌ | | +| CREATE OR MODIFY REST CLIENT | ✅/❌ | | +| DROP REST CLIENT | ✅/❌ | | +| DESCRIBE CONTRACT OPERATION FROM OPENAPI | ✅/❌ | | + +### JSON Structure Coverage + +| Command | Tested | Notes | +|---------|--------|-------| +| SHOW JSON STRUCTURES | ✅/❌ | | +| SHOW JSON STRUCTURES IN module | ✅/❌ | | +| DESCRIBE JSON STRUCTURE | ✅/❌ | | +| CREATE JSON STRUCTURE (single-quote) | ✅/❌ | | +| CREATE JSON STRUCTURE (dollar-quoting) | ✅/❌ | | +| CREATE JSON STRUCTURE (with folder/comment) | ✅/❌ | | +| CREATE JSON STRUCTURE (custom name map) | ✅/❌ | | +| CREATE OR REPLACE JSON STRUCTURE | ✅/❌ | | +| DROP JSON STRUCTURE | ✅/❌ | | + +### Roundtrip Results + +``` +OData Clients: _n_ tested, _n_ passed, _n_ failed +OData Services: _n_ tested, _n_ passed, _n_ failed +External Entities: _n_ tested, _n_ passed, _n_ failed +Published REST: _n_ tested, _n_ passed, _n_ failed +REST Clients: _n_ tested, _n_ passed, _n_ failed +JSON Structures: _n_ tested, _n_ passed, _n_ failed +``` + +### Multi-Step Workflows (§40) + +| Scenario | Result | Notes | +|----------|--------|-------| +| 40.1 End-to-end REST integration | ✅/❌ | | +| 40.2 OData publish-consume loop | ✅/❌ | | +| 40.3 Scaffold module | ✅/❌ | | +| 40.4 Iterative service evolution | ✅/❌ | | +| 40.5 Drop/recreate different config | ✅/❌ | | + +### Failure Modes (§41) + +| Scenario | Result | Notes | +|----------|--------|-------| +| 41.1 Not connected (all 6 types) | ✅/❌ | | +| 41.2 Entity not found | ✅/❌ | | +| 41.4 Invalid metadata URL | ✅/❌ | | +| 41.5 Feature gate (pre-10.0) | ✅/❌ | | +| 41.6 Feature gate (pre-10.1) | ✅/❌ | | +| 41.7 Invalid JSON | ✅/❌ | | +| 41.8 Non-existent microflow ref | ✅/❌ | | +| 41.10 Validation mid-batch | ✅/❌ | | + +### Issues Found + +1. (none / describe issues here) +``` diff --git a/docs/15-testing/mapping-test-cases.md b/docs/15-testing/mapping-test-cases.md new file mode 100644 index 00000000..8c8f7e09 --- /dev/null +++ b/docs/15-testing/mapping-test-cases.md @@ -0,0 +1,509 @@ +# Mapping & Data Transformer Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#386](https://github.com/mendixlabs/mxcli/pull/386) + +## Test Projects + +Demo apps from [Mendix App Gallery](https://appgallery.mendixcloud.com/): + +| App | Studio Pro | Import Mappings | Export Mappings | Data Transformers | +|-----|-----------|-----------------|-----------------|-------------------| +| Lato Enquiry Management | 11.4.0 | — | — | — | +| Evora - Factory Management | 10.24.15 | — | — | — | +| Lato Product Inventory | 11.2.0 | — | — | — | + +> Data Transformers require Mendix 11.9+ (feature gate: `integration/data_transformer`). Only the Lato apps (11.x) may contain them. Evora (10.24.15) must not. + +--- + +## Setup + +### 1. Download test apps + +1. Go to [Mendix App Gallery](https://appgallery.mendixcloud.com/) +2. Download each demo app listed above +3. Open each `.mpk` in Studio Pro to extract the `.mpr` file + +### 2. Build mxcli + +```bash +make build && make test && make lint-go +``` + +### 3. Smoke test + +```bash +APPS_DIR= +for mpr in "$APPS_DIR"/*/*.mpr; do + echo "=== $(basename $(dirname $mpr)) ===" + echo "show import mappings;" > /tmp/show-im.mdl + mxcli exec /tmp/show-im.mdl -p "$mpr" 2>&1 | tail -1 + echo "show export mappings;" > /tmp/show-em.mdl + mxcli exec /tmp/show-em.mdl -p "$mpr" 2>&1 | tail -1 +done +``` + +### 4. Interactive testing + +```bash +mxcli repl -p /EnquiriesManagement.mpr +``` + +### 5. Script-based testing + +```bash +mxcli exec test-sequence.mdl -p +``` + +All commands in this document are **read-only** (SHOW and DESCRIBE). No write guard warnings apply. + +--- + +## 1. SHOW IMPORT MAPPINGS + +### 1.1 List all import mappings +``` +show import mappings; +``` +**Expected:** Table with columns: Import Mapping, Name, Schema Source, Elements. Count matches Studio Pro. + +### 1.2 Filter by module +``` +show import mappings in MyModule; +``` +**Expected:** Only import mappings from `MyModule`. + +### 1.3 Empty module +``` +show import mappings in ModuleWithNoMappings; +``` +**Expected:** Empty result, no error. + +### 1.4 Non-existent module +``` +show import mappings in NonExistentModule; +``` +**Expected:** Error message. + +### 1.5 Column accuracy +Pick 5+ import mappings. Verify Schema Source and Elements columns match Studio Pro. + +--- + +## 2. DESCRIBE IMPORT MAPPING + +### 2.1 Simple import mapping +``` +describe import mapping Module.SimpleMapping; +``` +**Expected:** Valid MDL output in the form: +``` +create import mapping Module.SimpleMapping with json structure Module.Schema { + create Module.Entity { + Attr = jsonField key, + } +}; +``` + +### 2.2 Element rendering — object root +Verify the root element uses handling keyword and entity reference: +``` +create Module.Entity { + ... +} +``` +Valid handling keywords: `create` (default), `find`, `find or create`. + +### 2.3 Element rendering — nested object +Verify nested elements use association path and `jsonKey`: +``` +create Assoc/Entity = jsonKey { + ... +} +``` + +### 2.4 Element rendering — value mapping +Verify leaf values render as: +``` +Attr = jsonField key, +``` +The `key` suffix marks the key attribute. + +### 2.5 Handling keyword — find +``` +describe import mapping Module.FindMapping; +``` +**Expected:** Root element uses `find Module.Entity { ... }`. + +### 2.6 Handling keyword — find or create +``` +describe import mapping Module.FindOrCreateMapping; +``` +**Expected:** Root element uses `find or create Module.Entity { ... }`. + +### 2.7 Handling keyword — create (default) +``` +describe import mapping Module.CreateMapping; +``` +**Expected:** Root element uses `create Module.Entity { ... }`. + +### 2.8 Multiple root elements +Test import mapping with multiple top-level mapped entities. Verify all rendered. + +### 2.9 Deeply nested structure (3+ levels) +Verify correct indentation and nesting for mappings with 3+ levels of nested objects. + +### 2.10 Non-existent import mapping +``` +describe import mapping Module.DoesNotExist; +``` +**Expected:** Clear error message. + +--- + +## 3. SHOW EXPORT MAPPINGS + +### 3.1 List all export mappings +``` +show export mappings; +``` +**Expected:** Table with columns: Export Mapping, Name, Schema Source, Elements. Count matches Studio Pro. + +### 3.2 Filter by module +``` +show export mappings in MyModule; +``` +**Expected:** Only export mappings from `MyModule`. + +### 3.3 Empty module +``` +show export mappings in ModuleWithNoMappings; +``` +**Expected:** Empty result, no error. + +### 3.4 Non-existent module +``` +show export mappings in NonExistentModule; +``` +**Expected:** Error message. + +### 3.5 Column accuracy +Pick 5+ export mappings. Verify Schema Source and Elements columns match Studio Pro. + +--- + +## 4. DESCRIBE EXPORT MAPPING + +### 4.1 Simple export mapping +``` +describe export mapping Module.SimpleExport; +``` +**Expected:** Valid MDL output in the form: +``` +create export mapping Module.SimpleExport with json structure Module.Schema null values