Add STRUCT/ARRAY/MAP nested column support to RAB (CSA-371)#2392
Merged
Add STRUCT/ARRAY/MAP nested column support to RAB (CSA-371)#2392
Conversation
Expands complex SQL types (STRUCT, ARRAY<STRUCT>, MAP<K,STRUCT>) into child Column assets linked via parentColumn hierarchy. Sub-columns are excluded from the table's flat Columns list by clearing all tableQualifiedName/tableName/table/view refs — navigation is via parentColumn chain only. New fields on sub-columns: - parentColumnQualifiedName / parentColumn / parentColumnName - columnHierarchy: newline-delimited JSON ancestor entries (enables breadcrumb display in the UI, e.g. struct_col > city) - columnDepthLevel: 1 for direct fields, 2+ for deeper nesting - nestedColumnOrder: ordinal position within parent - subType=nested QN format (matching Databricks connector): - STRUCT field: tableQN/parentCol/fieldName - ARRAY<STRUCT>: tableQN/parentCol/items/fieldName - MAP<K,STRUCT>: tableQN/parentCol/values/fieldName - Deeply nested: tableQN/parentCol/outer/inner Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: bladata1990 <balakrishnan.r@atlan.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: bladata1990 <balakrishnan.r@atlan.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: bladata1990 <balakrishnan.r@atlan.com>
The dependencies { include(project(":samples:packages:asset-import")) }
filter was accidentally removed from the shadowJar block, causing the fat
jar to bundle all transitive SDK dependencies instead of just asset-import.
This conflicted with the base container image jars and caused
ClassNotFoundException: com.atlan.pkg.rab.Importer at runtime.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: bladata1990 <balakrishnan.r@atlan.com>
Collaborator
Author
|
@cmgrote Gentle nudge on this, can you check and approve |
Collaborator
Author
|
Tested sample asset in customer environment- https://mastercard-pov.atlan.com/assets/e2b15ecb-62be-4304-b661-9cc27f5d1b18/overview |
cmgrote
requested changes
Apr 16, 2026
Collaborator
cmgrote
left a comment
cmgrote
left a comment
There was a problem hiding this comment.
Need to drop the changes to the CI (.github/workflows/merge.yml in particular) for non-main branches. Want to avoid creating a permanent footprint for this that needs to be maintained.
Also, it looks like the logic should handle new headings in the CSV (for parent column qualifiedName, etc) — but the provided test file and tests don't seem to exercise this path at all. Please extend the test file and test scenario to test these additions, too.
Collaborator
Author
|
@cmgrote We have not added any new columns to the files because the Struct itself is defined in the 'dataType' column attached a sample file |
… tag logic Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: bladata1990 <balakrishnan.r@atlan.com>
Adds ComplexTypeColumnsRABTest covering the full pipeline from CSV input to nested child columns in Atlan, verifying parentColumnQualifiedName, columnDepthLevel, synthetic QN nodes (/items/, /values/), and depth-2 recursion. Also updates assets-complex.csv with adminRoles/adminUsers so the connection can be created during test setup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: bladata1990 <balakrishnan.r@atlan.com>
Collaborator
Author
|
@cmgrote updated the E2E tests now |
…fy CI comment The compileJava task dependency on genCustomPkg is already covered by sourcesJar; removing the duplicate from asset-import and RAB builds. Also tightens the comment in custom-package-container.yml to accurately describe the branch-tag logic (manual run, not all non-main branches). Signed-off-by: Chris (He/Him) <cgrote@gmail.com>
Signed-off-by: Chris (He/Him) <cgrote@gmail.com>
Signed-off-by: Chris (He/Him) <cgrote@gmail.com>
cmgrote
approved these changes
Apr 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
parentColumnhierarchyKey design decisions
Why sub-columns don't appear in the flat Columns list:
Atlan auto-creates the
table_columnsrelationship fromtableQualifiedNameserver-side. Sub-columns have all table/view reference fields cleared (tableQualifiedName,tableName,table,viewQualifiedName,viewName,view,materializedView) so they are NOT added totable_columns. Navigation is via theparentColumnchain.QN format (matching Databricks connector):
tableQN/parentCol/fieldNametableQN/parentCol/items/fieldNametableQN/parentCol/values/fieldNametableQN/parentCol/outerField/innerFieldFields set on sub-columns:
parentColumnQualifiedName/parentColumn/parentColumnNamecolumnHierarchy: newline-delimited JSON ancestor entries (enables breadcrumb display)columnDepthLevel: 1 for direct fields, 2+ for deeper nestingnestedColumnOrder: ordinal within parentsubType=nestedFiles changed
ComplexTypeParser.kt— NEW: bracket-aware recursive parser for STRUCT/ARRAY/MAP type stringsAssetXformer.kt— added nested column fields toBASE_OUTPUT_HEADERSColumnXformer.kt— overridesmapRow()to recursively emit child column rows for complex typesComplexTypeParserTest.kt— NEW: unit tests for the parserassets-complex.csv— NEW: test fixture with STRUCT/ARRAY/MAP typesbuild.gradle.kts(RAB + AIM) — minor build fixesTest plan
ci_noqn_testtable)plain_col,struct_col,nested_struct_colappear in table's flat Columns list (3 total)city,zipappear only insidestruct_col's nested viewouter,labelappear only insidenested_struct_col's nested viewinner,countappear only insideouter's nested view (depth=2)columnHierarchybreadcrumbs display correctly in the UI./gradlew :samples:packages:relational-assets-builder:test -PpackageTests🤖 Generated with Claude Code