Database Packaging

[isc-dchui]

Description

Resolves #986

Overview

• The feature adds a new lifecycle class, %IPM.Lifecycle.Database, that sits alongside the existing %IPM.Lifecycle.Module. It inherits from %IPM.Lifecycle.Base directly — not from Module — because Module has PACKAGING as a [Final] parameter and can't be subclassed to change it.
• Two new commands are exposed in %IPM.Main:
  • package-database → forces %IPM.Lifecycle.Database, routes to the Package phase
  • publish-database → same, but routes to Publish (which calls Package first, then uploads to registry)
• The -swap-db modifier is added to load/install/update to authorize the DB swap during install.

Key Details

• Package = IRIS.DAT + metadata, not source. The .tgz contains the binary IRIS.DAT, an enriched module.xml (with database and a SHA-256 ), a deps/ directory of dependency manifests, and any non-compiled resource files (wheels, FileCopy, CPF). Nothing is recompiled at install time.
• Remap-before-dismount during packaging. When building the package, the namespace is remapped to a fresh empty temp DB before the old DB is dismounted. This avoids any window where the namespace has no mounted routines DB.
• Generated resources need special handling. Classes with Generated="true" are exported from the source DB to XML before the remap, then imported into the temp DB after. Once the old DB is dismounted, compiled generated classes are inaccessible.
• Install skips Reload and Compile entirely. IsInstallContext() checks two conditions: Packaging=database AND IRIS.DAT is present in the module root. If both are true, %Reload sets SkipInvokes=1 and returns, %Compile is a no-op, and %Activate performs the swap. Pre-Activate elements are suppressed because the packaged classes don't exist until the DB is mounted.
• Swap is a rename, not a copy. PerformDatabaseSwap dismounts the existing DB, renames the current IRIS.DAT to IRIS__.DAT as a backup, renames the packaged IRIS.DAT into place, and remounts. Renames are atomic and avoid copying multi-GB files.
• Rollback is best-effort. On any failure after the swap starts, RollbackDatabaseSwap attempts dismount → delete new IRIS.DAT → rename backup back → remount. It never throws — it logs warnings and continues since it's called from a catch block.
• Update step seeding on fresh install. %Activate calls HandleAllUpdateSteps(..., seedOnly=1) on a fresh install so a later update only runs steps introduced since that version. Skipped when params("Update")=1 (an actual upgrade).
• One database package per namespace. ValidateBeforeSwap checks %IPM_Storage.ModuleItem for any other module with Packaging='database' and rejects the install if one exists.
• Python deps are on by default for package-database. Unlike package (which defaults off), package-database defaults ExportPythonDependencies=1. Both requirements.txt-based deps (resolved to wheels via ExportPythonDependencies) and explicit resources are staged into the .tgz. PythonWheel install at runtime fires during Initialize phase, which has no Database.cls override.
• Non-compiled resources are handled by resource processor hooks, not %Activate. FileCopy, WebApplication, and CPF processors run via OnBeforePhase/OnAfterPhase at the Module level before the lifecycle method is called. Overriding %Activate in Database.cls doesn't suppress them.
• publish-database forces the lifecycle class. The handler sets tCommandInfo("data","Lifecycle") = "%IPM.Lifecycle.Database", then re-routes to the existing publish command handler. Base.%Publish calls ..Package(.pParams, 0) first (which populates ..Payload), then uploads to the registry. No Database.cls override of %Publish is needed.
• dependencies.xml is actually a deps/ directory of per-dependency IRIS export XML files, each loadable with $system.OBJ.Load. RegisterDependencyMetadata loads them at install time to populate IPM storage — the code is already in the swapped-in DB, so only the IPM metadata records need registering. This is much easier to handle instead of using a fragile XSLT for a giant dependencies.xml
• ValidateIPMNotInRoutinesDB is called at two points: immediately in ShellInternal (fail-fast before any work) and again inside %Package (defense-in-depth). It checks %Library.RoutineMgr.IsMapped("%IPM.Main.CLS").
• The XSLT transform (InjectDatabasePackagingTransform XData) injects and into module.xml. The checksum placeholder REPLACECHECKSUM is substituted in ObjectScript before the stylesheet is compiled, since XSLT 1.0 has no parameterized element content.
• When installing without swap-db, only installs source-packaged modules. If none exist, but database-packaged module(s) exist, informs user. When installing with swap-db, only installs database-packaged modules.
• When publishing database-packaged modules to OCI registry, sets com.intersystems.ipm.packaging manifest annotation to "database" (for backwards compatibility, source packaged modules will not have this annotation) and use <module-version>_database__<IRIS-version> tag

Testing

• All tests are in Test.PM.Integration.DatabasePackaging. The test infrastructure uses two shared lazy-initialized namespaces to avoid ~15 namespace create/teardown cycles per run:
  • SharedNS1 (TESTDBPKGNS1): simple-db-module, module-with-invokes, module-with-dependencies
  • SharedNS2 (TESTDBPKGNS2): all-resources-module, module-with-mixed-python, module-with-requirements; also has the zot ORAS registry configured
• Per-test install namespaces are created fresh and torn down in OnAfterOneTest.

Test	What it covers
TestSimplePackageAndInstall	Package → verify IRIS.DAT + module.xml content (Packaging, checksum, SystemRequirements version) → install with load -swap-db → classes callable, module in list and history
TestPackageAndInstallWithDependencies	Package dep-module + main-with-deps; install main-with-deps → main class exists, dep appears in list
TestPackageAndInstallAllResourceTypes	Smoke test: module with class, include file, generated class, WebApp, FileCopy, PythonWheel packages and installs; include constant accessible, generated class callable, WebApp created
TestPackageAndInstallWithTestResources	-include-test-resources includes Scope=test (TestHelper) and UnitTest (Test) classes; default packaging excludes both
TestPackageWithUseCurrentDbFlag	-use-current-db completes without error and produces a package file
TestPackagingAndInstallValidation	Valid install succeeds; tampered IRIS.DAT (checksum mismatch), zero-sized IRIS.DAT, and missing deps/ directory each fail validation
TestBackupAndRollback	Backup file created in routines DB directory after swap
TestUpgradeSourceToDatabase	Source-installed module upgraded via update -path -swap-db; Packaging changes to "database"; backup confirms DB swap occurred
TestUninstallDatabasePackage	Module removed from list; backup preserved; compiled classes gone from namespace
TestMixedPackaging	Database + source package coexist in same namespace; second database package in same namespace is rejected
TestPythonDependencyPackaging	module-with-mixed-python: explicit PythonWheel (lune) + requirements.txt wheel (pycparser) both included by default, excluded with -no-export-python-deps; lune importable after install. module-with-requirements: packaging wheel included by default, excluded with flag
TestInvokeBehaviorDuringDatabaseInstall	<Invoke After="Compile"> skipped during database install; <Invoke After="Activate"> runs; global markers confirm each
TestNonCompiledResourcesAppliedDuringInstall	CPF file staged in package; FileCopy target deleted before install, confirmed recreated after install — proves resource processor hooks fire independently of %Activate override
TestUpdateStepsWithDatabasePackaging	v1 fresh install seeds Step001 (not run); v2 upgrade runs Step002 once; Step001 still not run; backup count > 1 confirms second swap
TestPublishDatabaseWithORAS	publish-database + publish to zot; source install (no -swap-db) → Packaging=module; database install (-swap-db) → Packaging=database; main-with-deps install resolves dep as source; after unpublishing source tag, install without -swap-db fails with hint to use -swap-db
TestSystemRequirementsOverwrite	package-database overwrites a stale IRIS version in <SystemRequirements> with the current version

Not Yet Handled

• Mirroring

Checklist

• This branch has the latest changes from the main branch rebased or merged.
• Changelog entry added.
• Unit (zpm test -only) and integration tests (zpm verify -only) pass.
• Style matches the style guide in the contributing guide.
• Documentation has been/will be updated
  • Source controlled docs, e.g. README.md, should be included in this PR and Wiki changes should be made after this PR is merged (add an extra issue for this if needed)
• Pull request correctly renders in the "Preview" tab.