JAR dependencies never reached the classpath; $Total = 5; now parses (formula1 findings §11–§13)

.claude/skills/fix-issue.md

@@ -398,3 +398,6 @@ extracting `OffsetExpression`/`LimitExpression`.
 | A `create database connection … type 'Redshift'` (or `'SQLServer'`) executes, builds **0 errors**, and the connection does not work. The skill's own table listed both, and omitted the one value that matters for an unsupported driver | mxcli passes the type string straight to BSON (`addStr(e,"DatabaseType",…)`) and **mxbuild does not validate it either** — verified on 11.12.1 — so nothing between the author and the runtime says the type is not real. Studio Pro's picker (read from `modeler/ide-client/database-connector-editor/`, identical on 11.10.0/11.12.1/11.13.0) is MSSQL, MySQL, Oracle, PostgreSQL, Snowflake, **BYOD** ("Other") — no Redshift, no SQLServer | `mdl/executor/validate_database_type.go` (`ValidateDatabaseConnectionType`, MDL-DB01), wired in `cmd/mxcli/cmd_check.go`; `.claude/skills/mendix/database-connections.md` | **A warning, not an error**: the value set is version-specific and mxcli cannot prove a string wrong on a Mendix version it has not seen — but silence is worse when the build is green and the connection is dead. **`BYOD` is the discovery worth keeping**: it forces connection-string config and *skips the driver-presence check*, so any JDBC driver Mendix has no entry for (DuckDB, SQLite, ClickHouse) works by dropping the JAR in `userlib/`. **Generalisable**: when a doc lists enum values, the shipped Studio Pro editor bundle is the authority — grep `id:"…",label:"…"` out of `ide-client/`, and diff across cached versions to see whether the set moved. Tests `validate_database_type_test.go`. mxcli-formula1 #6 |
 | `mxcli init` run from a solution root (several app folders, no `.mpr` at the root) reports success and writes tooling that points at `project.mpr` — a file that does not exist. Nobody is told which project it picked, because it did not pick one | `findMprFile` looks only in the target directory; an empty result fell through to a hardcoded `"project.mpr"` default and initialisation continued as if that were a real project | `cmd/mxcli/init.go` (`findMprFilesInSubdirs` + the three-way branch on the candidate count) | Look one level down, then branch on the **count**: 0 → warn that generated paths will be placeholders; 1 → announce the project and initialise **that** directory; 2+ → refuse, list them, and print the exact command naming one. One level only — a Mendix app keeps its `.mpr` at its own root, and walking deeper starts finding deployment copies and backups. Candidates are sorted so the refusal and its suggested command are stable rather than directory-order dependent. **Generalisable**: a "sensible default" that names a file which does not exist is not a default, it is a silent wrong answer — count the candidates and let the count choose the behaviour. Tests `cmd/mxcli/init_discover_test.go`. mxcli-formula1 #3 |
 | `alter settings configuration 'Default' …` is the write form, but `describe settings configuration 'Default'` is a **parse error** — and `show settings configurations` summarises the configuration without `ApplicationRootUrl`, so the obvious command for "did my root URL land?" cannot answer it (and renders an empty DatabaseUrl as a bare `, ,`) | The grammar's `DESCRIBE SETTINGS` alternative took no object, so the read form of a write statement simply did not exist; the summary builder listed database/port fields and never gained the root URL when that property was added | `mdl/grammar/domains/MDLCatalog.g4` (`DESCRIBE SETTINGS (CONFIGURATION STRING_LITERAL)?`), `mdl/visitor/visitor_query.go` (reuses `DescribeStmt.Qualifier`), `mdl/executor/cmd_settings.go` (`writeSettingsConfiguration`, `describeSettingsConfiguration`, summary) | **Read forms should mirror write forms** — where MDL has `alter X <selector>`, `describe X <selector>` should parse, and reaching for it and getting a parse error teaches the wrong lesson. Factor the emit into one helper so the whole-settings dump and the single-configuration dump cannot drift. An unknown name lists the ones that exist, or the user is guessing. **Watch for**: changing `describeSettings`'s signature broke three existing callers, and the same commit's `IsPartOfKey`→`KEY` change broke an OData round-trip test that only the FULL suite caught — run `go test ./mdl/...`, not just the new test. Tests `cmd_settings_configuration_test.go`. mxcli-formula1 #8 |
+| `ALTER MODULE X ADD JAR DEPENDENCY (…)` succeeds, `list jar dependencies` reports it, the build is **green** — and the runtime throws `SQLException: No JDBC driver found in app for URL`. `deployment/build.gradle` has no dependencies block and `find deployment -iname '*<artifact>*'` returns nothing | Not a bad write. Declaring and resolving are **separate steps**: the model records the coordinate, and `mx sync-java-dependencies <project.mpr>` is what downloads it into `vendorlib/`. Studio Pro runs that when you edit Module Settings; nothing headless was running it. Confirmed on 11.12.1 — a full `mxbuild --target=deploy` resolves nothing, and the sync command then fetches the jar | `cmd/mxcli/docker/javadeps.go` (`SyncJavaDependencies`, `UnvendoredJarDependencies`), `cmd/mxcli/cmd_sync_java_deps.go` (`mxcli sync-java-deps [--check]`), `cmd/mxcli/docker/runlocal.go` (vendors before boot), `mdl/executor/cmd_modules.go` (`warnUnvendoredJarDependencies`) | **How to find the missing step**: the reporter's open question was "does mxbuild skip Maven resolution, or does mxcli write it somewhere MxBuild cannot read?" — neither. `strings mx.dll | grep -i dependenc` surfaced `ISyncJavaDependenciesRunner`/`SkipManagedDependencySync`, and `mx --help` listed `sync-java-dependencies`. When a model-level write "works" but the artefact never appears, check whether the **toolset** has a separate command for it before suspecting the write. Wired at three levels so the gap cannot stay silent: the executor says so the moment it writes an unvendored coordinate, `run --local` resolves it before boot, and `--check` exits non-zero as a build gate. Resolution needs network, so every call site is best-effort with an actionable message. Tests `cmd/mxcli/docker/javadeps_test.go`. mxcli-formula1 #12 |
+| `$Total = 5;` does not parse — `no viable alternative at input '$Total=5'` — while `DECLARE $Total Integer = 0;` does, and so do `$X = HEAD($List)`, `$X = create M.E (…)` and `$X = execute database query …`. The error names the token, not the missing keyword | Assignment existed only as a **prefix** on specific activity statements (`(VARIABLE EQUALS)?` on CALL/CREATE/RETRIEVE/…), plus a `SET $Var = expression` statement. A plain value therefore required `SET`, which nothing in the error or the surrounding syntax suggested | `mdl/grammar/domains/MDLMicroflow.g4` (`setStatement : SET? …`), `cmd/mxcli/syntax/features_microflow.go` | Make the guessable form work rather than improve the error: `SET` is now optional and both spellings produce the same `MfSetStmt`/`ChangeVariableAction`. **Prove a grammar relaxation causes no regressions with a control binary, not by reading**: `git stash` the `.g4`, `make grammar`, build `bin/mxcli-control`, sweep every `mdl-examples/**/*.mdl` with both — 13 scripts fail, the *same* 13, all pre-existing. ANTLR's adaptive prediction picks the activity-prefixed alternatives over `setStatement` on its own; no ordering change was needed. Executed against a real .mpr, mxbuild reports 0 errors. Tests `mdl/visitor/visitor_microflow_bare_assign_test.go` (bare and keyword forms must agree on the AST, not merely both parse). mxcli-formula1 #13 |
+| `mxcli test tests/ -p app/App.mpr` fails with "no such file or directory" for a `tests/` that sits right next to the `.mpr` | Test paths resolved against the process CWD only. Defensible in isolation, but mxcli otherwise encourages naming the project (`-p`) rather than standing in its directory, and project auto-discovery searches outward — so the two conventions collide and the failure looks like a missing directory | `cmd/mxcli/cmd_test_run.go` (`resolveTestPaths`) | Fall back to project-relative **only when the CWD-relative path does not exist**: a `tests/` in both places must resolve to the one the user is standing in, since silently preferring the project's copy would run the wrong suite. A path that exists in neither is passed through unchanged so the error names what was typed, not a rewritten path the user never mentioned. Tests `cmd/mxcli/cmd_test_run_paths_test.go`. mxcli-formula1 #13 |

.claude/skills/mendix/database-connections.md

@@ -92,8 +92,37 @@ shipped bundle at `modeler/ide-client/database-connector-editor/`, identical on
 **`'BYOD'` — bring your own driver.** Selecting it forces connection-string
 configuration and **skips the driver-presence check**; its only validation is
 that the connection string is non-empty. That is the hook for any JDBC driver
-Mendix ships no picker entry for (DuckDB, SQLite, ClickHouse, …). Drop the
-driver JAR in `userlib/` and give the connection its JDBC URL.
+Mendix ships no picker entry for (DuckDB, SQLite, ClickHouse, …). Verified end to
+end on Mendix 11.13: a booted runtime opened `jdbc:duckdb:` through a `BYOD`
+connection and returned real rows — the runtime accepts it, not just the editor.
+
+### Getting the driver onto the classpath
+
+The driver JAR has to be *resolved*, and declaring it is not resolving it:
+
+```sql
+ALTER MODULE MyModule ADD JAR DEPENDENCY (
+  group = 'org.duckdb', artifact = 'duckdb_jdbc', version = '1.5.5.1', included = true
+);
+```
+
+writes the coordinate to the model — `list jar dependencies` will report it — and
+downloads **nothing**. MxBuild does not resolve it either: a full
+`mxbuild --target=deploy` emits a `build.gradle` with no dependencies block. The
+first symptom is a runtime `SQLException: No JDBC driver found in app for URL`,
+from a connection that looks correctly configured.
+
+Studio Pro runs the resolution when you edit Module Settings. Headless, ask for it:
+
+```bash
+mxcli sync-java-deps -p app.mpr          # download into vendorlib/
+mxcli sync-java-deps -p app.mpr --check  # report what is missing, exit 1 (build gate)
+```
+
+`mxcli run --local` does this automatically for anything not already in
+`vendorlib/`, so the warm loop works from a fresh clone. Dropping the jar into
+`userlib/` by hand works too — it is the same classpath — but then the model and
+the file system disagree about where the dependency comes from.
 
 **`'Redshift'` and `'SQLServer'` are not real values.** Both appeared in an
 earlier version of this table and neither is in the picker on any version
@@ -332,6 +361,30 @@ $ResultList = execute database query Module.Connection.QueryName
   dynamic 'SELECT id, name FROM employees WHERE active = true LIMIT 10';
 ```
 
+**A dynamic override still requires a value for every declared parameter** —
+including the ones the replacement SQL does not use. The parameter list belongs
+to the query *definition*, not to the SQL string, so Mendix asks for all of them
+whatever you substitute. Pass a placeholder for the unused ones:
+
+```sql
+-- The definition declares $driverId; this SQL ignores it, and the call still
+-- has to supply it.
+$Count = execute database query F1.DuckDB.CountAllDrivers
+  dynamic 'SELECT count(*) AS n FROM read_csv(''/data/f1db-drivers.csv'')'
+  ( driverId = 'unused' );
+```
+
+**A `{param}` placeholder can be concatenated into a path**, which is what keeps
+absolute paths out of the model — bind the data directory as one constant and
+build the file name around it:
+
+```sql
+--   read_csv({dataDir} || '/f1db-drivers.csv')
+```
+
+Verified against DuckDB through the connector on Mendix 11.13, and against a
+standalone JDBC harness before that.
+
 ### Parameterized Queries
 
 Pass values for query parameters defined with `parameter` in the query definition:

cmd/mxcli/cmd_sync_java_deps.go

@@ -0,0 +1,119 @@
+// SPDX-License-Identifier: Apache-2.0
+
+package main
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+
+	"github.com/mendixlabs/mxcli/cmd/mxcli/docker"
+	"github.com/mendixlabs/mxcli/sdk/mpr"
+	"github.com/spf13/cobra"
+)
+
+var syncJavaDepsCmd = &cobra.Command{
+	Use:   "sync-java-deps",
+	Short: "Download the project's managed Java (JAR) dependencies into vendorlib/",
+	Long: `Resolve every managed Java dependency the model declares and download it
+into the project's vendorlib/ directory.
+
+Declaring a dependency and resolving it are separate steps. MDL's
+'ALTER MODULE X ADD JAR DEPENDENCY (…)' records the coordinate in the model —
+'list jar dependencies' will report it — but nothing downloads the jar, and
+MxBuild does not resolve it either: a full build produces a build.gradle with no
+dependencies block. Studio Pro runs the resolution for you when you edit Module
+Settings; headless, this command is that step.
+
+Without it the failure is silent until runtime, as a missing-driver exception
+from code that looks correctly configured.
+
+Requires network access (Maven) and the mx binary for the project's version;
+'mxcli setup mxbuild' fetches the latter.
+
+Examples:
+  mxcli sync-java-deps -p app.mpr
+  mxcli sync-java-deps -p app.mpr --check   # report what is missing, download nothing
+`,
+	RunE: func(cmd *cobra.Command, args []string) error {
+		projectPath, _ := cmd.Flags().GetString("project")
+		checkOnly, _ := cmd.Flags().GetBool("check")
+		if projectPath == "" {
+			return fmt.Errorf("--project (-p) is required")
+		}
+
+		deps, version, err := declaredJarDependencies(projectPath)
+		if err != nil {
+			return err
+		}
+		if len(deps) == 0 {
+			fmt.Println("No managed Java dependencies declared.")
+			return nil
+		}
+
+		missing := docker.UnvendoredJarDependencies(filepath.Dir(projectPath), deps)
+		fmt.Printf("Declared: %d managed Java dependency/dependencies; %d not in vendorlib/\n",
+			len(deps), len(missing))
+		for _, m := range missing {
+			fmt.Printf("  missing  %s\n", m)
+		}
+		if checkOnly {
+			if len(missing) > 0 {
+				// A non-zero exit makes this usable as a build-gate.
+				os.Exit(1)
+			}
+			return nil
+		}
+		if len(missing) == 0 {
+			return nil
+		}
+
+		fmt.Printf("Syncing against Mendix %s...\n", version)
+		if err := docker.SyncJavaDependencies(projectPath, "", version, os.Stdout); err != nil {
+			return err
+		}
+		if still := docker.UnvendoredJarDependencies(filepath.Dir(projectPath), deps); len(still) > 0 {
+			// mx reports success even when a coordinate resolves to nothing, so
+			// check the postcondition rather than trusting the exit code.
+			return fmt.Errorf("sync finished but %d dependency/dependencies are still missing from vendorlib/: %v", len(still), still)
+		}
+		fmt.Println("All declared dependencies are in vendorlib/.")
+		return nil
+	},
+}
+
+// declaredJarDependencies reads the model's managed Java dependencies and the
+// project's Mendix version.
+func declaredJarDependencies(projectPath string) ([]docker.JarDependencyRef, string, error) {
+	reader, err := mpr.Open(projectPath)
+	if err != nil {
+		return nil, "", fmt.Errorf("opening project: %w", err)
+	}
+	defer reader.Close()
+
+	version := reader.ProjectVersion().ProductVersion
+
+	// ListModuleSettings covers every module in one read; the dependency's owner
+	// does not matter here, only whether the jar is on the classpath.
+	all, err := reader.ListModuleSettings()
+	if err != nil {
+		return nil, version, fmt.Errorf("reading module settings: %w", err)
+	}
+	var out []docker.JarDependencyRef
+	for _, ms := range all {
+		if ms == nil {
+			continue
+		}
+		for _, d := range ms.JarDependencies {
+			out = append(out, docker.JarDependencyRef{
+				Group: d.GroupID, Artifact: d.ArtifactID, Version: d.Version,
+			})
+		}
+	}
+	return out, version, nil
+}
+
+func init() {
+	syncJavaDepsCmd.Flags().Bool("check", false, "Report missing dependencies and exit non-zero; download nothing")
+	rootCmd.AddCommand(syncJavaDepsCmd)
+}

cmd/mxcli/cmd_test_run.go

@@ -5,6 +5,7 @@ package main
 import (
 	"fmt"
 	"os"
+	"path/filepath"
 	"time"
 
 	"github.com/mendixlabs/mxcli/cmd/mxcli/testrunner"
@@ -147,7 +148,7 @@ Examples:
 
 		opts := testrunner.RunOptions{
 			ProjectPath:  projectPath,
-			TestFiles:    args,
+			TestFiles:    resolveTestPaths(args, projectPath),
 			SkipBuild:    skipBuild,
 			Local:        local,
 			LegacyRunner: legacyRunner,
@@ -177,3 +178,41 @@ Examples:
 		}
 	},
 }
+
+// resolveTestPaths lets a relative test path be relative to the PROJECT as well
+// as to the working directory.
+//
+// `mxcli test tests/ -p app/App.mpr` used to fail with "no such file or
+// directory" for a tests/ that sits right next to the .mpr — the path resolved
+// against the process CWD only. That is defensible on its own, but mxcli
+// otherwise encourages naming the project rather than standing in its
+// directory, so the two conventions collide (mxcli-formula1 findings #13).
+//
+// The working directory still wins: a tests/ in both places resolves to the one
+// the user is standing in, which is what every other tool does.
+func resolveTestPaths(paths []string, projectPath string) []string {
+	if projectPath == "" || len(paths) == 0 {
+		return paths
+	}
+	projectDir := filepath.Dir(projectPath)
+	out := make([]string, 0, len(paths))
+	for _, p := range paths {
+		if filepath.IsAbs(p) {
+			out = append(out, p)
+			continue
+		}
+		if _, err := os.Stat(p); err == nil {
+			out = append(out, p)
+			continue
+		}
+		if alt := filepath.Join(projectDir, p); alt != p {
+			if _, err := os.Stat(alt); err == nil {
+				out = append(out, alt)
+				continue
+			}
+		}
+		// Neither exists: keep the original so the error names what was typed.
+		out = append(out, p)
+	}
+	return out
+}

cmd/mxcli/cmd_test_run_paths_test.go

@@ -0,0 +1,74 @@
+// SPDX-License-Identifier: Apache-2.0
+
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+// mxcli-formula1 findings #13: `mxcli test tests/ -p app/App.mpr` failed with
+// "no such file or directory" for a tests/ sitting right next to the .mpr,
+// because the path resolved against the process CWD only. Defensible alone, but
+// mxcli otherwise encourages naming the project instead of standing in its
+// directory, so the two conventions collided.
+func TestResolveTestPaths(t *testing.T) {
+	root := t.TempDir()
+	projectDir := filepath.Join(root, "app")
+	if err := os.MkdirAll(filepath.Join(projectDir, "tests"), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	projectPath := filepath.Join(projectDir, "App.mpr")
+	if err := os.WriteFile(projectPath, []byte("x"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+
+	t.Run("falls back to the project directory", func(t *testing.T) {
+		got := resolveTestPaths([]string{"tests"}, projectPath)
+		want := filepath.Join(projectDir, "tests")
+		if len(got) != 1 || got[0] != want {
+			t.Errorf("got %v, want [%s]", got, want)
+		}
+	})
+
+	t.Run("the working directory still wins", func(t *testing.T) {
+		// A tests/ in both places must resolve to the one the user is standing
+		// in — that is what every other tool does, and silently preferring the
+		// project's copy would run the wrong suite.
+		cwdTests := filepath.Join(root, "tests")
+		if err := os.MkdirAll(cwdTests, 0o755); err != nil {
+			t.Fatal(err)
+		}
+		t.Chdir(root)
+
+		got := resolveTestPaths([]string{"tests"}, projectPath)
+		if len(got) != 1 || got[0] != "tests" {
+			t.Errorf("got %v, want the CWD-relative [tests]", got)
+		}
+	})
+
+	t.Run("an absolute path is untouched", func(t *testing.T) {
+		abs := filepath.Join(projectDir, "tests")
+		got := resolveTestPaths([]string{abs}, projectPath)
+		if len(got) != 1 || got[0] != abs {
+			t.Errorf("got %v, want [%s]", got, abs)
+		}
+	})
+
+	t.Run("a path that exists nowhere keeps what was typed", func(t *testing.T) {
+		// The error must name what the user wrote, not a rewritten path they
+		// never mentioned.
+		got := resolveTestPaths([]string{"nope"}, projectPath)
+		if len(got) != 1 || got[0] != "nope" {
+			t.Errorf("got %v, want [nope]", got)
+		}
+	})
+
+	t.Run("no project means no rewriting", func(t *testing.T) {
+		got := resolveTestPaths([]string{"tests"}, "")
+		if len(got) != 1 || got[0] != "tests" {
+			t.Errorf("got %v, want [tests]", got)
+		}
+	})
+}