Implement Liquibase CLI integration

[thomhurst]

Summary

• generate strongly typed Liquibase 5.0.3 options and services for 43 commands
• parse current Picocli help, including required options, aliases, enums, numeric/boolean values, multiline descriptions, and repeatable -Dname=value properties
• generate root/global options once and emit them before Liquibase subcommands
• execute Windows .bat/.cmd CLI launchers through cmd.exe with redirected output
• mark password options as secrets, document package usage, and pin/checksum the generator workflow installer

Validation

• full solution Release build
• 383 OptionsGenerator tests
• 62 focused command parser/logger/builder/Liquibase tests
• ModularPipelines.Liquibase build
• real pinned Liquibase 5.0.3 Windows generation: 43 commands, 59 files, checksum verified

Stacked on #3032.

Closes #3005

[greptile-apps]

Greptile Summary

This PR adds strongly typed Liquibase 5.0.3 support. The main changes are:

• Generated options and services for 43 Liquibase commands.
• Parsing for global options, aliases, required values, enums, and repeatable properties.
• Secret redaction for password options.
• Windows batch and command-script launcher support.
• Package documentation and a checksum-pinned generator workflow.

Confidence Score: 5/5

This looks safe to merge.

• The Windows launcher path now resolves batch scripts and executes them through the command interpreter.
• Redirected output and cancellation behavior remain intact.
• The previously failing Liquibase generation path is covered by tests for this change.
• No blocking issue remains in the updated code.

Important Files Changed

Filename	Overview
tools/ModularPipelines.OptionsGenerator/src/ModularPipelines.OptionsGenerator/Scrapers/Cli/LiquibaseCliScraper.cs	Adds Liquibase help parsing and Windows executable resolution, including batch-launcher discovery.
src/ModularPipelines/Context/Command.cs	Adds command execution support for Windows batch and command-script launchers.
src/ModularPipelines/Context/CommandLineBuilder.cs	Adds global-option ordering and repeatable property formatting used by Liquibase commands.
tools/ModularPipelines.OptionsGenerator/src/ModularPipelines.OptionsGenerator/Generators/CodeGeneratorOrchestrator.cs	Extends option generation to emit shared global options and Liquibase command models.

_{Reviews (11): Last reviewed commit: "fix(cli): restore command factory import" | Re-trigger Greptile}

[claude]

Code review

Reviewed the Liquibase 5.0.3 CLI integration (generator, scraper, generated options/services, tests, and the pinned installer workflow).

Verified:

• Downloaded the pinned liquibase-5.0.3.tar.gz release asset and confirmed its SHA-256 matches the checksum added in .github/workflows/generate-cli-options.yml — good supply-chain hardening.
• Spot-checked the new scraper logic (ParseOptions, AccumulateMultiLineDescription, TryCreateEnumDefinition, DetermineCSharpType, the -D property handling) against the generated output (e.g. LiquibaseUpdateOptions.Generated.cs) and the new unit/scraper tests — the required/enum/numeric/boolean/multiline-description inference all line up correctly, and the -D<name>=<value> repeatable property is wired through OptionFormat.NoSeparator + AllowMultiple correctly, matching the rendering test in LiquibaseOptionsTests.cs.
• Docs page and generated services/extensions are consistent with the rest of the codebase's established generator conventions.

No high-confidence bugs or CLAUDE.md violations found. One architectural suggestion, non-blocking:

Extract Windows executable resolution into a shared base-class helper instead of duplicating it per scraper.

LiquibaseCliScraper.ResolveExecutablePath() walks PATH looking for liquibase.bat and returns its full path. This is effectively the same job GcloudCliScraper.ResolveGcloudPath() already does (searching known install locations for gcloud.cmd). Two independent, slightly-different implementations of "find the Windows script wrapper for this CLI" now exist in Scrapers/Cli/.

Why this matters: ProcessCliCommandExecutor runs with UseShellExecute = false, and Process.Start cannot launch a .bat/.cmd script directly under that mode on Windows (it needs to go through cmd.exe /c). Both scrapers' resolved paths would hit this the same way if exercised on Windows — right now generate-windows in the workflow only runs choco/winget, so this doesn't fire in CI for either tool, but it will bite the next scraper that copies this pattern (or a contributor generating locally on Windows) until it's fixed once in a shared place.

Pulling the "search PATH / known install dirs for a Windows launcher script, and shell out to cmd.exe /c <script> <args> when needed" logic into CliScraperBase (or a small WindowsScriptExecutableResolver utility) would let the fix land once and be reused by every future CLI-first scraper that needs a .bat/.cmd, rather than each new scraper re-implementing (and re-debugging) the same PATH-walk.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8a81d2b6d2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[thomhurst]

Rebased onto current stacked base d58ff39 and addressed both review threads in 0d35a34: Windows command scripts now run through cmd.exe, and generated Liquibase global options render before subcommands. Validation passed: full Release solution build, 383 generator tests, 62 focused command tests, and real pinned Liquibase 5.0.3 Windows regeneration (43 commands / 59 files).

[claude]

Code review

Re-reviewed following the new commit pushed since my last review: fix(liquibase): support global options (routes Windows command scripts through cmd.exe and emits scraped root/global options before Liquibase subcommands).

Previous suggestion addressed: My last review suggested extracting the Windows .bat/.cmd launcher handling into a shared helper instead of duplicating it per-scraper. This commit fixes it at an even better layer — ProcessCliCommandExecutor.CreateStartInfo now centrally detects .bat/.cmd targets and routes them through cmd.exe /d /s /c, so every current and future scraper gets the fix automatically rather than needing to copy a pattern.

New global-options feature, spot-checked:

• CliScraperBase.ProcessPathAsync captures GlobalOptions only from the root path (path.Length == 1), and child subcommand paths are only enqueued after the root path finishes processing — so there's no race where a subcommand worker reads GlobalOptions before it's populated.
• CodeGeneratorOrchestrator now calls CreateToolDefinition() after the scraping loop instead of before — correct fix, since GlobalOptions wouldn't be populated yet if called beforehand.
• CommandLineBuilder.IsGlobalOption correctly checks the declaring type for [CliGlobalOptions] (matching Inherited = false on the attribute), so it identifies base-declared properties regardless of the derived options type — verified against the new Build_Places_Global_Options_Before_Subcommands test and the updated LiquibaseOptionsTests fixture.
• Nice cleanup as part of this: Command.cs no longer duplicates command-line-building logic and now delegates to the existing ICommandLineBuilder, removing dead code.

No high-confidence bugs or CLAUDE.md violations found in this delta. Checked for compile-breaking issues, logic errors, and CLAUDE.md compliance (generated files are consistent with regenerating from the updated generator, not hand-edited).

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0d35a34f43

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[claude]

Code review

Re-reviewed following the latest commit: fix(liquibase): preserve performance filenames ("Include Liquibase in release project discovery").

Changes in this delta:

• FindProjectsModule.cs adds ModularPipelines_Liquibase to the release project list, consistent with the surrounding entries.
• --monitor-performance is corrected from bool? to string? on LiquibaseOptions.Generated.cs, since per Liquibase's own help text it accepts false/true or an arbitrary filename to write JFR data to — not a pure boolean. This is implemented as a new BooleanDefaultExceptions set in LiquibaseCliScraper.cs, following the same exception-list pattern already used for NumericOptions.
• The generated options file was correctly regenerated from the fixed scraper rather than hand-edited (still carries the <auto-generated> / "Do not edit this file manually" header), so this stays compliant with the CLAUDE.md rule against modifying generated options classes directly.
• New coverage added at both layers: a scraper-parsing test (Monitor_Performance_Accepts_Boolean_Or_Filename_Values) and a rendered-command test (Monitor_Performance_Renders_A_Filename), confirming the fix end-to-end.

Previous review suggestions: Both prior suggestions (extracting shared Windows .bat/.cmd launcher handling, and the global-options ordering/race check) were already addressed and verified in the previous review round — nothing new surfaced in this delta that reopens either concern.

No high-confidence bugs or CLAUDE.md violations found in this commit.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2cf9b145d1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[thomhurst]

Rebased Liquibase-only commits onto the current Argo CD base and added the two documented Databricks diff globals.

Validation:

• official Liquibase 5.0.3 GitHub asset SHA-256 digest matched
• regeneration stable: 59 generated files, zero errors
• LiquibaseCliScraperTests: 6/6 passed
• LiquibaseOptionsTests: 5/5 passed
• ModularPipelines.Liquibase and ModularPipelines.Build compiled
• git diff --check clean

Head: 083a07c

[claude]

Code review

Re-reviewed following the latest commit: fix(liquibase): add Databricks diff filters (adds --databricks-diff-tblproperties-exclude-patterns / --databricks-diff-tblproperties-ignore-all, plus a #nullable enable regeneration pass across the generated Liquibase files).

No bugs or CLAUDE.md violations found, but this delta introduces a pattern worth reconsidering before more scrapers copy it.

Unconditional hardcoded option injection can silently mask real scrape results

LiquibaseCliScraper.ParseGlobalOptions now always calls AddDocumentedDatabricksOptions, which unconditionally overwrites (AddOrReplace) whatever the real liquibase --help output produced for these two switches with a hand-authored CliOptionDefinition. The Global_Options_Include_Documented_Databricks_Diff_Filters test confirms this: it passes help text containing only the exclude-patterns line, yet asserts the ignore-all option is present too — proving both are injected regardless of what the scraper actually observed.

Why this matters: this generator's entire value proposition (per the repo's CLAUDE.md) is that options classes are regenerated from the live CLI, so they stay correct as the tool evolves — that's why hand-editing generated files is banned. This hardcoded pair breaks that invariant in the other direction: if a future Liquibase/Databricks-extension version documents these flags in --help for real (with a different value format, a short form, or as a plain flag instead of --switch=value), AddOrReplace will still clobber the freshly scraped definition with the frozen hardcoded one, every regeneration, with no signal that the override is now stale or redundant. The same risk applies to any other CLI-first scraper that copies this "manually inject documented-but-unscraped options" pattern — each one becomes a silent, unreviewable divergence from the actual tool surface.

Suggested approach: make the fallback additive-only, not an unconditional replace, e.g.

private static void AddDocumentedDatabricksOptions(List<CliOptionDefinition> options)
{
    // Supplements options the CLI doesn't yet expose via --help.
    // If a name below ever appears in `options` already, the scraper found it live —
    // remove the corresponding manual entry so behavior always favors the live scrape.
    var seen = options.Select(o => o.SwitchName).ToHashSet(StringComparer.OrdinalIgnoreCase);
    foreach (var supplement in DatabricksSupplementalOptions.Where(o => !seen.Contains(o.SwitchName)))
    {
        options.Add(supplement);
    }
}

That way, if the CLI ever starts documenting these for real, the live-scraped values automatically win instead of being silently discarded, and a future contributor auditing the generator has a clear signal ("supplement is now a no-op") for when it's safe to delete the manual entry.

Minor, non-blocking: the two hardcoded CliOptionDefinitions skip IsSecret = GeneratorUtils.IsSecretOption(...), unlike every option produced through ParseOptions. Not a real risk here since neither name matches the secret-detection keywords, but worth using the same helper for consistency if this pattern is kept.

Previously flagged items: the Windows .bat/.cmd launcher extraction and global-options ordering/race concerns from earlier rounds remain addressed; nothing regressed in this commit.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 083a07ce5a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[thomhurst]

Addressed the latest review feedback in af7f1f6:

• help-marked required command values are optional in the generated API because Liquibase can supply them through defaults files or environment variables;
• documented Databricks supplements are now additive-only, so live scraped descriptions/types win;
• stable TblProperties public property casing is normalized during live parsing;
• supplemental options use the standard secret classifier;
• updated docs and regenerated 59 Liquibase files twice with identical output.

Validation: generator tests 418/418; Liquibase runtime tests 6/6; UnitTests, ModularPipelines.Liquibase, and ModularPipelines.Build compile successfully (existing warnings only).

@codex review

[claude]

Code review

Re-reviewed the delta since my last review: two new commits, fix(liquibase): add Databricks diff filters and fix(liquibase): support defaults files (current HEAD).

Changes in this delta:

• LiquibaseCliScraper.AddDocumentedDatabricksOptions manually adds two Databricks-specific global options (--databricks-diff-tblproperties-exclude-patterns, --databricks-diff-tblproperties-ignore-all) that aren't present in Liquibase's own --help output, guarded by AddIfMissing so a future CLI version that does document them won't produce a duplicate property. This follows the same "manual exception list" pattern already established for NumericOptions/BooleanDefaultExceptions, so it's consistent with the rest of the scraper rather than a new pattern.
• Required-option detection ([REQUIRED] in the help description → constructor parameter) is removed in favor of always IsRequired = false (LiquibaseCliScraper.cs#L249-L257), since Liquibase can source --changelog-file/--url/etc. from a defaults file or environment variables instead of the CLI. Verified this is a considered, tested tradeoff rather than an oversight: LiquibaseUpdateOptions.Generated.cs correctly moved ChangelogFile/Url from positional constructor parameters to regular init properties, and the new Defaults_File_Can_Supply_Required_Update_Values test in LiquibaseOptionsTests.cs confirms a command with only DefaultsFile set still renders correctly.
• New coverage added at all the right layers: scraper test (Global_Options_Include_Documented_Databricks_Diff_Filters), rendered-command test (Databricks_Diff_Filters_Render_As_Global_Options), and the defaults-file test above. I checked the bool?-typed DatabricksDiffTblPropertiesIgnoreAll renders through CommandArgumentBuilder.GetSingleValue as lowercase true/false via the equals-separated CliOption path (not a bare flag) — this matches the explicit test assertion (IsFlag is false by design here, not a leftover from copy-pasting the flag-detection code) and the expected command string in the new render test.
• Docs page (liquibase.md) and the earlier Update_Options_Parse_As_Expected test were both correctly updated to the new initializer-property style.

No high-confidence bugs or CLAUDE.md violations found in this delta. Previous review rounds' suggestions (shared Windows .bat/.cmd launcher handling, global-options ordering/race) remain addressed with nothing new reopening either concern.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af7f1f61b6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[thomhurst]

Rebased onto current main and resolved the docs/generator-test conflicts. Also fixed the unresolved overridden-global-metadata review in 1226b4989 and applied the shared secret-classifier consistency nit for Liquibase --format. Validation: OptionsGenerator 478/478; generated/reflection metadata 12/12; command ordering 1/1; broad UnitTests 1002 passed, 6 skipped, with only the known isolated-worktree name assertion remaining; docs typecheck/build (97 docs); SourceGenerator, core, and Liquibase Release builds; scoped format and diff check. @codex review

[claude]

Code review

Re-reviewed following the latest commits (8b316bf → 1226b4989, "fix(cli): preserve overridden global options" plus the diff-format follow-ups).

Changes in this delta, verified against the code:

• CommandModelProvider.IsGlobalOption (reflection fallback) now walks the base-type chain and matches on GetBaseDefinition() equality instead of only checking the property's immediate DeclaringType. This fixes a real gap: a derived record that overrides a virtual property (e.g. Verbose) without re-declaring [CliGlobalOptions] on itself would previously lose the "global option" classification inherited from its base type, causing CommandLineBuilder to place it after the subcommand instead of before it.
• CommandOptionsGenerator.IsGlobalOption (source-generated fast path) applies the equivalent fix via IPropertySymbol.OverriddenProperty, keeping the reflection and source-gen paths consistent — important since CliOptionValuePair/CommandLineBuilder must produce identical ordering regardless of which path executes at runtime.
• Both fixes are covered by new tests: ReflectionMetadata_PreservesGlobalMetadataFromOverriddenProperties (with a ReflectionOverrideCommandDerived fixture that also overrides a private property, exercising the DeclaredOnly/NonPublic binding-flag lookup) and the updated Build_Places_Global_Options_Before_Subcommands test, which now asserts IsGlobalOption directly on the resolved model rather than only asserting on rendered output.
• LiquibaseCliScraper.cs#L365 now calls GeneratorUtils.IsSecretOption("Format", isFlag: false) instead of hardcoding IsSecret = false — this was the consistency nit flagged in the previous review round, now resolved so every generated/supplemented option goes through the same secret classifier.

No high-confidence bugs or CLAUDE.md violations found in this delta. All previously flagged items across earlier rounds (Windows launcher duplication, global-options ordering, additive-only Databricks/diff override pattern, required-option typing, secret-classifier consistency) are now addressed at this HEAD.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 1226b49891

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[thomhurst]

Rebased onto current main at 199f299. Resolved the source-generator metadata conflict by retaining both main's SecretValueKeys and Liquibase's IsGlobalOption fields, then restored the CliCommandFactory import required by the rebased Command path.

Validation: SourceGenerator and Liquibase builds pass; OptionsGenerator 485/485; LiquibaseOptions 7/7; CommandLineBuilder 8/8; GeneratedRuntimeMetadata 14/14.

[claude]

Code review

Re-reviewed at current HEAD (199f2998e4, "fix(cli): restore command factory import"), after the three most recent commits: 9a4a73232c (generate diff format option), f801aaf693 (preserve overridden global options), 199f2998e4 (restore the ModularPipelines.Helpers.Internal using dropped during the rebase onto main).

This PR has already been through many review rounds (mine, Codex, Greptile), and I independently re-verified the full diff rather than trusting prior summaries. Findings:

Verified correct:

• The pinned Liquibase installer checksum in .github/workflows/generate-cli-options.yml — downloaded liquibase-5.0.3.tar.gz from the official GitHub release myself and its SHA-256 matches 865aab391be9cbe3fee22461392d5d29aa76652596598b14700a3f5f05c9e2b7 exactly. Good hardening: pinning + verifying a release asset instead of trusting "latest".
• CliGlobalOptionsAttribute + the dual IsGlobalOption implementations (CommandOptionsGenerator.IsGlobalOption via IPropertySymbol.OverriddenProperty, and CommandModelProvider.IsGlobalOption via reflection's base-type walk + GetBaseDefinition()) produce consistent results and both correctly propagate the flag through virtual property overrides that don't redeclare the attribute. Covered by matching tests on both paths (CommandMetadata_PreservesAttributesFromOverriddenProperties, ReflectionMetadata_PreservesGlobalMetadataFromOverriddenProperties).
• Command.cs's refactor to delegate entirely to CommandLineBuilder.Build(...) is a real simplification — it deletes ~20 lines of argument-assembly logic that duplicated what CommandLineBuilder already did, removes four now-unused constructor dependencies, and leaves one code path responsible for command-line construction instead of two. This is the kind of consolidation worth calling out positively: it removes a class of "generated vs. manual command differ" bugs.
• The last commit's fix (restoring the ModularPipelines.Helpers.Internal using in Command.cs, needed for CliCommandFactory) is correct and matches the file's actual usage.
• Generated Liquibase options/services (43 commands, 59 files) follow the existing generator conventions consistently — [CliGlobalOptions] on LiquibaseOptions, secrets routed through GeneratorUtils.IsSecretOption everywhere including the two Databricks/--format supplements that earlier rounds had flagged as hardcoded.
• Test coverage is solid: 383 OptionsGenerator tests + 62 focused command/parser tests, including edge cases like the Windows .bat launcher resolution, multi-line Picocli description accumulation, and global-vs-command option separation.

One minor, non-blocking nit (cosmetic, not a bug): the generated XML doc comment for AnalyticsEnabled in LiquibaseOptions.Generated.cs reads:

Enable or disable sending product usage data and analytics to Liquibase. Learn more at https: //docs.liquibase.com/analytics.

The stray space after https: comes from the scraper's multi-line description joiner (AccumulateMultiLineDescription) rejoining a URL that Picocli's help text wrapped mid-string. It's harmless (doc comment only, not functional), but if it's easy to special-case URL continuations (e.g. don't insert a space when the previous line ends in : immediately before //), it'd be a nice polish. Not blocking.

No high-confidence bugs, security issues, or CLAUDE.md violations found. All items raised across earlier rounds (Windows launcher duplication, global-options ordering, additive-only Databricks/diff overrides, required-option typing, secret-classifier consistency, the rebase's dropped import) remain resolved at this HEAD. This looks ready to merge pending the stacked base (#3032).

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 199f2998e4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Expose the Secure license key globally

When pipelines run Liquibase Secure features and need to pass a per-invocation license key, Liquibase documents the CLI form as liquibase --license-key=... [command] (docs), but this generated global-options base has no LicenseKey property (I also checked that --license-key/LicenseKey is absent from the Liquibase package and scraper). Because CommandLineBuilder appends raw Arguments after the subcommand, typed calls such as secure JSON diff cannot emit the required global position without dropping to a custom command; add a secret global --license-key option in the scraper/generator.

Useful? React with 👍 / 👎.

[thomhurst]

The failed Ubuntu job has no downloadable log; its check annotation says the hosted runner lost communication with GitHub while Run Pipeline was still in progress. This is infrastructure failure, not a surfaced repository error. I reran the failed job once and will let CI complete.