Skip to content

docs: add manual test case documentation for all MDL command areas#386

Draft
retran wants to merge 1 commit intomendixlabs:mainfrom
retran:test-case-docs-clean
Draft

docs: add manual test case documentation for all MDL command areas#386
retran wants to merge 1 commit intomendixlabs:mainfrom
retran:test-case-docs-clean

Conversation

@retran
Copy link
Copy Markdown
Contributor

@retran retran commented Apr 29, 2026

Summary

Comprehensive manual test case documentation covering all MDL command areas. 18 test documents, 663 test cases total.

Coverage

Area File Tests
Entities & Associations entity-test-cases.md 58
Enumerations enumeration-test-cases.md 32
Microflows microflow-test-cases.md 89
Pages page-test-cases.md 72
Security security-test-cases.md 45
Navigation & Settings navigation-settings-test-cases.md 38
Organization organization-test-cases.md 28
Workflows workflow-test-cases.md 42
Integration (REST/OData) integration-test-cases.md 67
Catalog & SQL catalog-test-cases.md, sql-integration-test-cases.md 44
CLI Commands cli-commands-test-cases.md 35
Tooling (lint/report/fmt) tooling-test-cases.md 30
Business Events business-event-test-cases.md 24
Mappings mapping-test-cases.md 22
Agent/Editor agent-editor-test-cases.md 18
Images/Collections image-collection-test-cases.md 12
Session Management session-test-cases.md 7

Test Execution Results (v0.7.0-257-ga07e7e0)

  • Pass rate: 534/663 (80.5%)
  • Bugs filed: 16 confirmed issues (see comment below)
  • Test projects: Evora CRM (large), EnquiriesManagement (small)

Notes

  • All expectations corrected to match actual mxcli behavior observed during testing
  • Tests designed for manual execution by AI agents or human QA
  • Each test case includes: preconditions, steps, expected results, and cleanup

@github-actions
Copy link
Copy Markdown

AI Code Review

What Looks Good

  • The PR adds comprehensive manual test case documentation for all MDL command areas as described, with no code changes (documentation-only).
  • Each document follows the established structure from existing nanoflow test cases (numbered scenarios, MDL code blocks, expected results, failure modes, boundary/stress tests, coverage matrices, and manual test report templates).
  • The documentation covers areas that correspond to implemented features per the "Current Implementation Status" section (e.g., agent-editor, business-event, catalog, workflow, etc.), so it is not re-documenting already-shipped features as new but rather providing test documentation for existing functionality.
  • The structure is consistent and ready for QA use as stated, with clear setup instructions and test project coverage matrices.
  • No overlap with existing proposals (the proposals directory is empty) or duplication of code implementations (since no code is changed).
  • The documentation correctly uses MDL syntax patterns (colon for property definitions, qualified names, standard CRUD verbs) consistent with existing examples, and does not misuse AS in non-mapping contexts.

Recommendation

Approve the PR. The documentation is valuable for manual testing and QA processes, adheres to contribution guidelines, and introduces no risks since it contains no code changes.


Automated review via OpenRouter (Nemotron Super 120B) — workflow source

@retran retran force-pushed the test-case-docs-clean branch from c36fb9d to c361ed9 Compare April 29, 2026 20:30
@retran
Copy link
Copy Markdown
Contributor Author

retran commented Apr 29, 2026

Test Report — mxcli v0.7.0-257 (2026-04-30)

Projects: Evora CRM (Mendix 10.24.15, 708 entities), EnquiriesManagement (Mendix 11.8.0, 281 entities)
Method: AI-assisted execution of 761 test cases across 17 categories
Test docs: docs/15-testing/

Summary

Metric Count
Total test cases 761
Passed 534
Failed (confirmed) 97
Skipped (Docker/auth/TTY) 130
Pass rate (runnable) 85%
Bugs filed 30

Bugs Filed

# Sev Title Issue
1 Critical CREATE WORKFLOW user task panics (nil pointer) #403
2 High DESCRIBE ASSOCIATION emits DELETE_CASCADE but parser only accepts CASCADE #388
3 High CREATE ASSOCIATION allows duplicate qualified names #389
4 High mxcli report hangs after catalog load (all formats) #397
5 High GRANT ON entity accepts non-existent module roles #399
6 High lint hangs after catalog load #413
7 High ALTER PAGE SET only works on DynamicText/Content #414
8 High UPDATE WIDGETS syntax not implemented (parse error) #416
9 Medium DESCRIBE ENUMERATION drops documentation comments #393
10 Medium CREATE ENUMERATION accepts duplicate value names #390
11 Medium DROP ENUMERATION ambiguous name silently drops first match #391
12 Medium DESCRIBE ENUMERATION unescaped single quotes in captions #394
13 Medium SHOW CONTEXT OF Entity produces empty definition #396
14 Medium ALTER SNIPPET looks up page namespace instead of snippet #402
15 Medium DESCRIBE ODATA SERVICE output not parseable (roundtrip broken) #400
16 Medium CREATE PUBLISHED REST SERVICE allows duplicate names #401
17 Medium CREATE EXTERNAL ENTITY accepts non-existent OData client #417
18 Medium SHOW WIDGETS WHERE Name LIKE filter non-functional #418
19 Medium CATALOG.ASSOCIATIONS table does not exist #419
20 Medium CREATE OR MODIFY REST CLIENT reports "Created" for existing #421
21 Medium mxcli new exits 0 on download failure #422
22 Medium CREATE EXTERNAL ENTITIES FROM ODATA CLIENT not implemented #423
23 Low CREATE MICROFLOW accepts calls to non-existent microflows #395
24 Low Unknown attribute type silently interpreted as enum reference #392
25 Low mxcli fmt outputs to stdout, ignores invalid input #398
26 Low CREATE ASSOCIATION cross-module flaky under load #420
27 Low diff-local exits 0 on git error #424
28 Low Multiple CLI commands exit 0 on error conditions #425
29 Low rename module not implemented #426
30 Low Invalid MetadataUrl accepted for OData client #427

Results by Category

Category Total Pass Fail Skip Pass %
Entities 95 72 10 13 88%
Enumerations 30 25 5 0 83%
Microflows 62 51 4 7 93%
Pages 62 43 8 11 84%
Integration 78 61 7 10 90%
Security 45 37 8 0 82%
Navigation/Settings 38 30 8 0 79%
Workflows 42 32 10 0 76%
Organization 28 24 4 0 86%
Catalog/SQL 44 38 6 0 86%
CLI Commands 67 25 9 33 74%
Tooling 87 27 4 56 87%
Business Events 24 20 4 0 83%
Mappings 22 17 5 0 77%
Agent/Editor 18 15 3 0 83%
Images/Collections 12 10 2 0 83%
Session 7 7 0 0 100%

Blocked Tests (69)

Blocker Tests Issue
Workflow user task panic 25 #403
Agent-editor syntax (not in parser) 16
DESCRIBE/ALTER STYLING (not implemented) 10
ALTER SNIPPET namespace bug 7 #402
Report hang 5 #397
Wireframe format (not implemented) 3
SHOW LANGUAGES (not implemented) 2
LINT module filter (hang) 1 #413

Key Findings

  1. Roundtrip fidelity — DESCRIBE output often not re-parseable (DESCRIBE ASSOCIATION emits DELETE_CASCADE but parser only accepts CASCADE #388, DESCRIBE ENUMERATION emits unescaped single quotes in captions #394, DESCRIBE ODATA SERVICE output not parseable — roundtrip broken #400). This blocks automated refactoring workflows.
  2. Missing validation — CREATE commands accept invalid references, allowing corrupted models (CREATE ASSOCIATION allows duplicate qualified names #389, CREATE ENUMERATION accepts duplicate value names #390, CREATE MICROFLOW accepts calls to non-existent microflows without error #395, GRANT ... ON entity accepts non-existent module roles #399, CREATE PUBLISHED REST SERVICE allows duplicate qualified names #401, CREATE EXTERNAL ENTITY accepts non-existent OData client without validation #417).
  3. Catalog-dependent commands hang — both lint and report freeze after catalog load (mxcli report hangs indefinitely after catalog load #397, lint hangs indefinitely after catalog load #413). Likely shared root cause.
  4. ALTER PAGE too limited — SET only works on DynamicText/Content (ALTER PAGE SET only works on DynamicText/Content — fails for other widget properties #414). UPDATE WIDGETS not implemented (UPDATE WIDGETS syntax not implemented (parse error) #416). Page modification requires full CREATE OR REPLACE.
  5. Workflow authoring blocked — user task creation panics (CREATE WORKFLOW with user task panics — nil pointer dereference #403), blocking 25 tests.
  6. Exit code hygiene — multiple commands exit 0 on error (mxcli new exits 0 on download failure #422, diff-local exits 0 on git error #424, Multiple CLI commands exit 0 on error conditions #425), breaking CI/scripting.

@retran retran force-pushed the test-case-docs-clean branch from c361ed9 to 1b98697 Compare April 29, 2026 20:46
@github-actions
Copy link
Copy Markdown

AI Code Review

What Looks Good

This PR adds comprehensive manual test case documentation for all MDL command areas, which is a valuable contribution for quality assurance and testing. The documentation is well-organized by feature area (agents, business events, catalog, CLI commands, etc.) and includes:

  • Clear preconditions, steps, expected results, and cleanup instructions for each test case
  • Coverage of both read and write operations (with appropriate warnings about destructive tests)
  • Roundtrip testing validation (CREATE → DESCRIBE → CREATE from output)
  • Failure mode testing and boundary/stress tests
  • Cross-module reference validation
  • Proper version-specific considerations noted where applicable

The test cases appear to align with features marked as "Implemented" in CLAUDE.md (e.g., AI agent documents, business event services), and the documentation correctly references module requirements (like AgentEditorCommons for agents) and version constraints.

The structure follows the project's documentation conventions, placing test documentation in the appropriate docs/15-testing/ directory. The PR explicitly states that expectations were corrected to match actual mxcli behavior observed during testing, showing this documentation was validated against real implementation.

Recommendation

Approve this PR. The documentation adds significant value for manual testing and QA processes without introducing any code changes or violating project guidelines. It properly covers implemented features and provides actionable test cases for both AI agents and human testers. No changes are needed.


Automated review via OpenRouter (Nemotron Super 120B) — workflow source

@retran
Copy link
Copy Markdown
Contributor Author

retran commented Apr 30, 2026

Skipped Tests — Infrastructure Requirements

Total skipped: ~130 tests (from test session; some prereq-blocked tests counted as failures)

By Skip Reason

Docker Required (Docker Desktop must be running)

Tests requiring Docker containers for databases, MxBuild, or runtime.

Test Doc Section Test Numbers Count
cli-commands-test-cases.md §1 docker 1.1–1.10 10
cli-commands-test-cases.md §2 test 2.1–2.8 8
cli-commands-test-cases.md §3 playwright 3.1–3.4 4
sql-integration-test-cases.md §1 SQL CONNECT 1.1–1.18 18
sql-integration-test-cases.md §2 SQL DISCONNECT 2.1–2.4 4
sql-integration-test-cases.md §3 SQL CONNECTIONS 3.1–3.3 3
sql-integration-test-cases.md §4 SQL QUERY 4.1–4.10 10
sql-integration-test-cases.md §5 SHOW TABLES/VIEWS/FUNCTIONS 5.1–5.6 6
sql-integration-test-cases.md §6 SQL DESCRIBE 6.1–6.5 5
sql-integration-test-cases.md §7 GENERATE CONNECTOR 7.1–7.11 11
sql-integration-test-cases.md §8 IMPORT FROM 8.1–8.13 13
sql-integration-test-cases.md §9 CREATE DATABASE CONNECTION 9.1–9.11 11
sql-integration-test-cases.md §10 SHOW DATABASE CONNECTIONS 10.1–10.3 3
sql-integration-test-cases.md §11 DESCRIBE DATABASE CONNECTION 11.1–11.4 4
sql-integration-test-cases.md §12 MULTI-STEP WORKFLOWS 12.1–12.4 4
sql-integration-test-cases.md §13 FAILURE MODES 13.1–13.10 10
sql-integration-test-cases.md §14 BOUNDARY & STRESS 14.1–14.8 8
Subtotal 132

To unblock: Start Docker Desktop. For SQL tests, provision containers:

docker run -d --name mxcli-postgres -p 5432:5432 -e POSTGRES_PASSWORD=test postgres:16
docker run -d --name mxcli-mssql -p 1433:1433 -e ACCEPT_EULA=Y -e SA_PASSWORD='Test1234!' mcr.microsoft.com/mssql/server:2022-latest

Auth Required (Mendix SSO token)

Test Doc Section Test Numbers Count
cli-commands-test-cases.md §7 auth login/logout 7.1, 7.4 2
cli-commands-test-cases.md §8 marketplace 8.2–8.4 3
Subtotal 5

To unblock: ./mxcli auth login


TTY/Interactive Required

Test Doc Section Test Numbers Count
cli-commands-test-cases.md §15 tui 15.1–15.2 2
Subtotal 2

To unblock: Run manually in interactive terminal.


Prerequisites Not Met (git repo, OpenAPI spec, specific setup)

Test Doc Section Test Numbers Count
tooling-test-cases.md §3 DIFF LOCAL 3.2–3.8 7
integration-test-cases.md §30 CREATE REST CLIENT FROM OPENAPI 30.1–30.4 4
integration-test-cases.md §33 DESCRIBE CONTRACT FROM OPENAPI 33.1–33.3 3
cli-commands-test-cases.md §9.2 widget extract 9.2 1
cli-commands-test-cases.md §10.3 bson compare 10.3 1
Subtotal 16

To unblock:

  • DIFF LOCAL: Copy test project into git repo, make changes, commit
  • OpenAPI: Place petstore.yaml or use live Swagger URL
  • Widget extract: Provide .mpk widget file
  • BSON compare: Set up 2 project copies

Priority to unblock the most tests

Step Action Tests unblocked
1 Start Docker + provision DB containers 132
2 Set up git repo with test project 7
3 Place OpenAPI spec file 7
4 Run mxcli auth login 5
5 Manual TTY testing 2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant