[WIP] feat(workflows): batch entity processing — entityList-only for automated task nodes#26715
[WIP] feat(workflows): batch entity processing — entityList-only for automated task nodes#26715
Conversation
yan-3005
added
safe to test
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
...adata/service/governance/workflows/elements/nodes/automatedTask/impl/RollbackEntityImpl.java
Show resolved
Hide resolved
...rces/json/schema/governance/workflows/elements/nodes/automatedTask/dataCompletenessTask.json
Outdated
Show resolved
Hide resolved
...e/governance/workflows/elements/nodes/automatedTask/impl/CheckChangeDescriptionTaskImpl.java
Outdated
Show resolved
Hide resolved
✅ TypeScript Types Auto-UpdatedThe generated TypeScript types have been automatically updated based on JSON schema changes in this PR. |
There was a problem hiding this comment.
Pull request overview
Phase 1 of batch-entity processing for governance workflows: moves automated task nodes to consume entities exclusively via entityList (a list of entity-link strings), updates trigger/BPMN wiring to pass lists, and adds a v1130 migration to update stored workflow definitions accordingly.
Changes:
- Extend trigger outputs and workflow event variable initialization to include
entityList. - Update automated task node implementations and BPMN builders to read from
entityListand to populateentityListininputNamespaceMapby default. - Add v1130 data migration to inject
entityListinto workflow definitions and removerelatedEntityfrom batch-node namespace maps; update built-in workflow JSON and integration tests.
Reviewed changes
Copilot reviewed 30 out of 37 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/triggers/periodicBatchEntityTrigger.json | Updates trigger output schema to include entityList. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/triggers/eventBasedEntityTrigger.json | Updates trigger output schema to include entityList. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/sinkTask.json | Switches sink task schema inputs to entityList. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/setEntityAttributeTask.json | Switches set-attribute task schema inputs to entityList. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/rollbackEntityTask.json | Switches rollback task schema inputs to entityList. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/dataCompletenessTask.json | Adds entityList to schema for completeness task. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/checkEntityAttributesTask.json | Switches check-attributes task schema input to entityList and adds output list defaults. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/checkChangeDescriptionTask.json | Switches check-change-description task schema input to entityList and adds output list defaults. |
| openmetadata-service/src/main/resources/json/data/governance/workflows/GlossaryApprovalWorkflow.json | Updates built-in workflow to wire batch-capable nodes via entityList. |
| openmetadata-service/src/main/java/org/openmetadata/service/migration/utils/v1130/MigrationUtil.java | Adds v1130 migration to add entityList to triggers and batch node namespace maps; strips relatedEntity for batch nodes; redeploys workflows. |
| openmetadata-service/src/main/java/org/openmetadata/service/migration/postgres/v1130/Migration.java | Executes the new workflow migration during v1130 Postgres upgrade. |
| openmetadata-service/src/main/java/org/openmetadata/service/migration/mysql/v1130/Migration.java | Executes the new workflow migration during v1130 MySQL upgrade. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/triggers/impl/FetchEntitiesImpl.java | Builds entityToListMap so periodic trigger can pass per-entity entityList without static calls in JUEL. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/triggers/PeriodicBatchEntityTrigger.java | Updates periodic trigger call-activity input mapping to always pass entityList (full batch or per-entity list) and adjusts loop cardinality handling. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/triggers/EventBasedEntityTrigger.java | Ensures event-based trigger passes entityList into the called workflow. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/sink/SinkTaskDelegate.java | Refactors sink delegate to always read entities from entityList. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/SetEntityAttributeImpl.java | Changes set-attribute delegate to iterate over entityList. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/RollbackEntityImpl.java | Changes rollback delegate to iterate over entityList and adds BPMN error handling + exception propagation. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/DataCompletenessImpl.java | Changes completeness delegate to process entityList and emit per-band lists + results. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/CheckEntityAttributesImpl.java | Changes attribute-check delegate to partition entities via entityList. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/CheckChangeDescriptionTaskImpl.java | Changes change-description-check delegate to partition entities via entityList. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/SetEntityAttributeTask.java | Ensures BPMN builder injects entityList namespace mapping by default. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/RollbackEntityTask.java | Ensures BPMN builder injects entityList namespace mapping by default. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/DataCompletenessTask.java | Ensures BPMN builder injects entityList namespace mapping by default. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/CheckEntityAttributesTask.java | Ensures BPMN builder injects entityList namespace mapping by default. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/CheckChangeDescriptionTask.java | Ensures BPMN builder injects entityList namespace mapping by default. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/TriggerFactory.java | Forces periodic triggers to run with singleExecutionMode=false. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowEventConsumer.java | Populates both relatedEntity and entityList in event-triggered workflow variables. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/Workflow.java | Adds a constant for a “false entity list” variable name. |
| openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/WorkflowDefinitionResourceIT.java | Updates integration tests to use entityList-based configs and expected outputs. |
Comments suppressed due to low confidence (1)
openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/dataCompletenessTask.json:110
- This schema still requires
relatedEntityininputNamespaceMap(and includes it in the defaultinput), but the Java implementation now reads only fromentityListand the v1130 migration stripsrelatedEntityfrom batch node inputNamespaceMaps. As written, migrated workflows will fail validation fordataCompletenessTask. Update the schema to makeentityListthe required input/namespace and remove or deprecaterelatedEntityhere to match runtime behavior.
"input": {
"type": "array",
"items": { "type": "string" },
"default": ["relatedEntity", "entityList"],
"additionalItems": false,
"minItems": 1
},
"inputNamespaceMap": {
"type": "object",
"properties": {
"relatedEntity": {
"type": "string",
"default": "global"
},
"entityList": {
"type": "string",
"default": "global"
}
},
"additionalProperties": false,
"required": ["relatedEntity"]
},
...ice/src/main/java/org/openmetadata/service/governance/workflows/elements/TriggerFactory.java
Show resolved
Hide resolved
openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/Workflow.java
Show resolved
Hide resolved
...ervice/governance/workflows/elements/nodes/automatedTask/impl/CheckEntityAttributesImpl.java
Outdated
Show resolved
Hide resolved
...e/governance/workflows/elements/nodes/automatedTask/impl/CheckChangeDescriptionTaskImpl.java
Outdated
Show resolved
Hide resolved
...in/resources/json/schema/governance/workflows/elements/triggers/eventBasedEntityTrigger.json
Outdated
Show resolved
Hide resolved
...resources/json/schema/governance/workflows/elements/triggers/periodicBatchEntityTrigger.json
Outdated
Show resolved
Hide resolved
...json/schema/governance/workflows/elements/nodes/automatedTask/checkEntityAttributesTask.json
Show resolved
Hide resolved
🟡 Playwright Results — all passed (19 flaky)✅ 3397 passed · ❌ 0 failed · 🟡 19 flaky · ⏭️ 216 skipped
🟡 19 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
…ted task nodes
Phase 1 of batch entity processing in governance workflows. All
automated task nodes (checkEntityAttributesTask, setEntityAttributeTask,
checkChangeDescriptionTask, rollbackEntityTask, sinkTask,
dataCompletenessTask) now process a List<String> of entity links via
entityList exclusively. The relatedEntity fallback in getEntityList()
is removed — batch nodes no longer have that path.
Key changes:
- PeriodicBatchEntityTrigger (singleExecutionMode=false): each child
process now receives global_entityList via ${entityToListMap[relatedEntity]}.
FetchEntitiesImpl pre-builds entityToListMap (entity -> List.of(entity))
so the JUEL expression resolves without static class references.
- Batch node impls (6 files): removed relatedEntity fallback from
getEntityList() and the now-unused RELATED_ENTITY_VARIABLE import.
entityList is the only input path.
- BPMN builders: putIfAbsent(ENTITY_LIST_VARIABLE, GLOBAL_NAMESPACE) for
all batch-capable nodes; relatedEntity is never added to inputNamespaceMap
by builders.
- v1130 migration (addEntityListToNamespaceMap): updated to also strip
relatedEntity from batch node inputNamespaceMaps, covering both fresh
upgrades (add entityList + remove relatedEntity) and instances that
already ran the previous migration (entityList present, remove relatedEntity).
Migration remains idempotent.
- GlossaryApprovalWorkflow.json: removed relatedEntity from all 13 batch
node inputNamespaceMaps. userApprovalTask nodes keep relatedEntity.
- JSON schemas: entityList added to trigger output and batch node
inputNamespaceMap/input definitions.
- Integration tests: updated to reflect entityList-first structure across
all batch-capable workflow node configurations.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
yan-3005
force-pushed
the
ram/workflow-improvements
branch
from
ccd7f2a to
0ec257c
Compare
✅ TypeScript Types Auto-UpdatedThe generated TypeScript types have been automatically updated based on JSON schema changes in this PR. |
...in/resources/json/schema/governance/workflows/elements/triggers/eventBasedEntityTrigger.json
Outdated
Show resolved
Hide resolved
- Remove relatedEntity from checkEntityAttributesTask and dataCompletenessTask inputNamespaceMap schemas; make entityList required with default "global" - Restore hasBatchModeNodes() in TriggerFactory so sink workflows with batchMode=true run as a single batch execution instead of N parallel children - Extract duplicated getEntityList() from 6 impl files into a shared static method on WorkflowVariableHandler - Add migrateInputArray() to MigrationUtil to replace relatedEntity with entityList in stored workflow node input arrays - Fix integration test trigger outputs: remove relatedEntity from pure batch workflows; add entityList to approval workflows that were missing it; keep relatedEntity for workflows with userApprovalTask nodes (required by MainWorkflow.WorkflowGraph validation) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
✅ TypeScript Types Auto-UpdatedThe generated TypeScript types have been automatically updated based on JSON schema changes in this PR. |
...ervice/governance/workflows/elements/nodes/automatedTask/impl/CheckEntityAttributesImpl.java
Outdated
Show resolved
Hide resolved
...e/governance/workflows/elements/nodes/automatedTask/impl/CheckChangeDescriptionTaskImpl.java
Outdated
Show resolved
Hide resolved
openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/Workflow.java
Show resolved
Hide resolved
...resources/json/schema/governance/workflows/elements/triggers/periodicBatchEntityTrigger.json
Show resolved
Hide resolved
…k variables - Fix falseEntityList vs false_entityList mismatch: schema output defaults in checkEntityAttributesTask.json and checkChangeDescriptionTask.json declared "falseEntityList" (camelCase) while runtime code writes "false_entityList" (underscore via FALSE_ENTITY_LIST_VARIABLE). Update schemas and integration tests to match the runtime variable name. - Remove five write-only dead execution variables from RollbackEntityImpl (rollbackAction, rollbackFromVersion, rollbackToVersion, rollbackEntityId, rollbackEntityType) — nothing reads them; they only polluted the workflow instance state blob. Remove unused DelegateExecution param from rollbackEntity() private method. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…*_entityList routing
- MigrationUtil: build incomingEdge (target→{source,condition}) and nodeSubType maps
from workflow JSON; downstream nodes of check tasks now get condition_entityList
(e.g. true_entityList, false_entityList, gold_entityList) keyed to the check node's
namespace, instead of always mapping entityList→global
- WorkflowVariableHandler.getEntityList(): iterate all inputNamespaceMap keys ending
in _entityList first (covers arbitrary dataCompleteness band names), then fall back
to plain entityList key
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
...data-service/src/main/java/org/openmetadata/service/migration/utils/v1140/MigrationUtil.java
Outdated
Show resolved
Hide resolved
...-service/src/test/java/org/openmetadata/service/migration/utils/v1140/MigrationUtilTest.java
Show resolved
Hide resolved
...-service/src/test/java/org/openmetadata/service/migration/utils/v1140/MigrationUtilTest.java
Show resolved
Hide resolved
...data-service/src/main/java/org/openmetadata/service/migration/utils/v1140/MigrationUtil.java
Outdated
Show resolved
Hide resolved
…ismatch - MigrationUtil.addEntityListToNamespaceMap: start from deepCopy of existing inputNamespaceMap and remove relatedEntity, instead of creating a fresh empty map — preserves updatedBy and other keys for setEntityAttributeTask/rollbackEntityTask - MigrationUtilTest: update all addEntityListToNamespaceMap calls to 3-arg signature (incoming, nodeSubType); add tests for updatedBy preservation, true/false/band conditional routing; fix Phase 2 test to not assert createOrUpdate (code now calls WorkflowHandler.deploy directly) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tants, renames - Add TRUE_ENTITY_LIST_VARIABLE constant; replace "true_" + ENTITY_LIST_VARIABLE literals - Add resilience4j per-entity retry (3 attempts, 500ms) to all 5 impl nodes and SinkTaskDelegate list mode; per-entity failures never throw BpmnError — workflow always continues to completion - RollbackEntityImpl: add Approved → Draft → Deprecated fallback chain for rollback target; remove unused workflow/rollbackToStatus fields - DataCompletenessImpl: fix band iteration to store all bands (including empty); fix storeFieldList to always store List<String>; store counts only in entityResults to reduce variable bloat; downgrade per-entity LOG.info → LOG.debug, add single summary LOG.info - SinkTaskDelegate: fix entityFqn in SinkError to use entityLink.getEntityFQN() not raw link string - Rename converted → definedNamespaceMap in 6 builder task files - WorkflowVariableHandler: add warning log when entityList not found - v1140 migration (MySQL + Postgres): add initializeWorkflowHandler() before migrateWorkflowInputNamespaceMap() Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
...adata/service/governance/workflows/elements/nodes/automatedTask/impl/RollbackEntityImpl.java
Show resolved
Hide resolved
...data-service/src/main/java/org/openmetadata/service/migration/utils/v1140/MigrationUtil.java
Outdated
Show resolved
Hide resolved
...ice/src/main/java/org/openmetadata/service/governance/workflows/WorkflowVariableHandler.java
Outdated
Show resolved
Hide resolved
...json/schema/governance/workflows/elements/nodes/automatedTask/checkEntityAttributesTask.json
Show resolved
Hide resolved
...son/schema/governance/workflows/elements/nodes/automatedTask/checkChangeDescriptionTask.json
Show resolved
Hide resolved
...data-service/src/main/java/org/openmetadata/service/migration/utils/v1140/MigrationUtil.java
Outdated
Show resolved
Hide resolved
...data-service/src/main/java/org/openmetadata/service/migration/utils/v1140/MigrationUtil.java
Outdated
Show resolved
Hide resolved
- Remove unused ADMIN_USER_NAME constant from MigrationUtil - Downgrade LOG.warn -> LOG.debug in WorkflowVariableHandler.getEntityList() - Add true_entityList to output defaults in checkEntityAttributes and checkChangeDescription schemas - Change additionalProperties: false -> true in all 6 task inputNamespaceMaps to allow conditional routing keys (true_entityList, false_entityList, gold_entityList) - Fix MigrationUtil incomingEdge map to use List<String[]> to handle multi-edge nodes correctly - Remove COLLECTION_VARIABLE from PeriodicBatchEntityTrigger; use Workflow.ENTITY_LIST_VARIABLE - Fix FetchEntitiesImpl to import ENTITY_LIST_VARIABLE from Workflow instead of removed COLLECTION_VARIABLE - Update rollbackEntityTask.json description to reflect APPROVED -> DRAFT -> DEPRECATED fallback chain - Update MigrationUtil Javadoc to reference WorkflowHandler#deploy Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
✅ TypeScript Types Auto-UpdatedThe generated TypeScript types have been automatically updated based on JSON schema changes in this PR. |
addEntityListToNamespaceMap now takes List<String[]> instead of String[] to support multi-edge nodes; update test call sites accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pr-26715-review-bugs.md was a scratch file tracking review items and should not be part of the repo. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Removed |
Code Review 👍 Approved with suggestions 9 resolved / 10 findingsBatch entity processing refactor for automated task nodes now includes retry logic, consolidated field handling, and migration fixes addressing 9 previously flagged issues. Consider extracting the duplicate RetryConfig across 6 implementation classes into a shared utility to reduce maintenance burden. 💡 Quality: Duplicate retry configuration across 6 impl classes📄 openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/RollbackEntityImpl.java:62-69 📄 openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/SetEntityAttributeImpl.java:64-71 📄 openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/DataCompletenessImpl.java:63-70 📄 openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/CheckEntityAttributesImpl.java:49-56 📄 openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/CheckChangeDescriptionTaskImpl.java:51-58 📄 openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/sink/SinkTaskDelegate.java:265-272 The identical Consider extracting a shared retry factory, e.g., ✅ 9 resolved✅ Bug: RollbackEntity overwrites execution variables in batch loop
✅ Bug: dataCompletenessTask.json schema still requires relatedEntity
✅ Quality: Duplicated getEntityList() method across 6 classes
✅ Bug: Trigger output schema maxItems:2 violated by 3-item outputs
✅ Bug: Workflow.FALSE_ENTITY_LIST_VARIABLE unused; hardcoded strings used instead
...and 4 more resolved from earlier reviews 🤖 Prompt for agentsOptionsAuto-apply is off → Gitar will not commit updates to this branch. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
| "inputNamespaceMap": { | ||
| "type": "object", | ||
| "properties": { | ||
| "relatedEntity": { | ||
| "entityList": { | ||
| "type": "string", | ||
| "default": "global" | ||
| } | ||
| }, | ||
| "additionalProperties": false, | ||
| "required": ["relatedEntity"] | ||
| "additionalProperties": true, | ||
| "required": ["entityList"] | ||
| }, |
There was a problem hiding this comment.
inputNamespaceMap still has required: ["entityList"] even though the migration/engine now support conditional keys like true_entityList and false_entityList for edge-based routing. Workflows that map only true_entityList/false_entityList into this node will be rejected by schema validation. Recommend removing/adjusting the required constraint and using typed additionalProperties (string) instead of true.
| @@ -58,8 +58,8 @@ | |||
| "default": null | |||
| } | |||
| }, | |||
| "additionalProperties": false, | |||
| "required": ["relatedEntity"] | |||
| "additionalProperties": true, | |||
| "required": ["entityList"] | |||
| }, | |||
There was a problem hiding this comment.
inputNamespaceMap still requires the plain entityList key, but edge-routed workflows may pass only conditional keys like true_entityList / false_entityList / <band>_entityList. This will cause schema validation failures for migrated definitions. Recommend loosening the required constraint and typing additionalProperties as { "type": "string" } (namespace values) instead of true.
| @@ -159,8 +159,8 @@ | |||
| "default": "global" | |||
| } | |||
| }, | |||
| "additionalProperties": false, | |||
| "required": ["relatedEntity"] | |||
| "additionalProperties": true, | |||
| "required": ["entityList"] | |||
| }, | |||
There was a problem hiding this comment.
InputNamespaceMap now allows arbitrary keys via additionalProperties: true but still requires entityList. This conflicts with the new conditional namespace-map keys (e.g. true_entityList, false_entityList, gold_entityList) introduced by the v1140 migration and WorkflowVariableHandler.getEntityList(), where nodes may legitimately omit the plain entityList key. Schema validation will fail for migrated/edge-routed workflows. Suggestion: make additionalProperties a typed schema (e.g. { "type": "string" }) and relax/adjust the requirement so either entityList or a *_entityList key is allowed.
| @@ -75,8 +75,8 @@ | |||
| "default": null | |||
| } | |||
| }, | |||
| "additionalProperties": false, | |||
| "required": ["relatedEntity"] | |||
| "additionalProperties": true, | |||
| "required": ["entityList"] | |||
| }, | |||
There was a problem hiding this comment.
inputNamespaceMap is marked required: ["entityList"] but the new batch routing can legitimately provide only a conditional key (e.g. true_entityList, false_entityList, or gold_entityList) with no plain entityList. As-is, migrated workflows can fail validation. Suggest: relax the required constraint and set additionalProperties to { "type": "string" } (namespace strings) instead of untyped true.
| "inputNamespaceMap": { | ||
| "type": "object", | ||
| "properties": { | ||
| "relatedEntity": { | ||
| "entityList": { | ||
| "type": "string", | ||
| "default": "global" | ||
| } | ||
| }, | ||
| "additionalProperties": false, | ||
| "required": ["relatedEntity"] | ||
| "additionalProperties": true, | ||
| "required": ["entityList"] | ||
| }, |
There was a problem hiding this comment.
inputNamespaceMap requires entityList, but with band-based routing the downstream node may only have <band>_entityList (and no plain entityList). This makes the schema incompatible with the new routing/migration logic. Suggest relaxing required and constraining additionalProperties to { "type": "string" } so conditional *_entityList keys are still validated as namespace strings.
|
|
Issue: https://github.com/open-metadata/openmetadata-collate/issues/3363
Phase 1 of batch entity processing in governance workflows. All automated task nodes (checkEntityAttributesTask, setEntityAttributeTask, checkChangeDescriptionTask, rollbackEntityTask, sinkTask, dataCompletenessTask) now process a List of entity links via entityList exclusively. The relatedEntity fallback in getEntityList() is removed — batch nodes no longer have that path.
Key changes:
PeriodicBatchEntityTrigger (singleExecutionMode=false): each child process now receives global_entityList via ${entityToListMap[relatedEntity]}. FetchEntitiesImpl pre-builds entityToListMap (entity -> List.of(entity)) so the JUEL expression resolves without static class references.
Batch node impls (6 files): removed relatedEntity fallback from getEntityList() and the now-unused RELATED_ENTITY_VARIABLE import. entityList is the only input path.
BPMN builders: putIfAbsent(ENTITY_LIST_VARIABLE, GLOBAL_NAMESPACE) for all batch-capable nodes; relatedEntity is never added to inputNamespaceMap by builders.
v1130 migration (addEntityListToNamespaceMap): updated to also strip relatedEntity from batch node inputNamespaceMaps, covering both fresh upgrades (add entityList + remove relatedEntity) and instances that already ran the previous migration (entityList present, remove relatedEntity). Migration remains idempotent.
GlossaryApprovalWorkflow.json: removed relatedEntity from all 13 batch node inputNamespaceMaps. userApprovalTask nodes keep relatedEntity.
JSON schemas: entityList added to trigger output and batch node inputNamespaceMap/input definitions.
Integration tests: updated to reflect entityList-first structure across all batch-capable workflow node configurations.
Describe your changes:
Fixes
I worked on ... because ...
Type of change:
Checklist:
Fixes <issue-number>: <short explanation>