Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance Schema adapter to accommodate evolving struct #15295

Open
wants to merge 55 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
c8236ed
feat: implement NestedStructSchemaAdapter for handling schema evoluti…
kosiew Mar 18, 2025
afbe1ed
feat: enhance NestedStructSchemaAdapter with schema mapping capabilities
kosiew Mar 18, 2025
c774cab
test: add schema mapping test for NestedStructSchemaAdapter
kosiew Mar 18, 2025
5f5cd45
feat: implement NestedStructSchemaAdapterFactory for handling nested …
kosiew Mar 18, 2025
6065bc1
test: add unit test for NestedStructSchemaAdapterFactory to validate …
kosiew Mar 18, 2025
410f8d7
test: refactor test_create_appropriate_adapter for clarity and effici…
kosiew Mar 18, 2025
50cf134
feat: enhance create_appropriate_adapter to support nested schema tra…
kosiew Mar 18, 2025
3f52617
refactor: simplify create_appropriate_adapter logic for nested schema…
kosiew Mar 18, 2025
ad74d3a
refactor: remove redundant default adapter test in nested schema adapter
kosiew Mar 18, 2025
134dace
feat: enhance NestedStructSchemaAdapter to support additional table s…
kosiew Mar 18, 2025
aa89671
refactor: simplify test_nested_struct_evolution
kosiew Mar 18, 2025
f361311
refactor: streamline schema creation in nested schema adapter tests
kosiew Mar 18, 2025
a914a6b
Fix clippy errors
kosiew Mar 18, 2025
d8eb3eb
test: add async test for schema evolution with compaction in NestedSt…
kosiew Mar 21, 2025
1735b45
refactor: add missing imports and clean up test code in nested_schema…
kosiew Mar 21, 2025
72aee85
Rollback to before adding test_datafusion_schema_evolution_with_compa…
kosiew Mar 21, 2025
772fbce
feat: add nested_struct.rs to test nested schema evolution test with …
kosiew Mar 21, 2025
20af2c0
chore: remove nested_struct.rs example file to streamline repository …
kosiew Mar 21, 2025
3c0844c
feat: Add nested_struct.rs async function for schema evolution with c…
kosiew Mar 21, 2025
ad09e60
feat: Enhance logging in nested_struct.rs for better traceability 📜✨
kosiew Mar 21, 2025
61f1f6e
created helper functions
kosiew Mar 21, 2025
16a47d3
map batch1 to schema2
kosiew Mar 21, 2025
7b7183e
feat: Enhance NestedStructSchemaAdapter with custom schema mapping fo…
kosiew Mar 21, 2025
84ab195
feat: Add debug print statements to map_batch for tracing execution f…
kosiew Mar 21, 2025
51dacc5
fix: Refactor nested schema mapping for improved error handling and c…
kosiew Mar 21, 2025
aa5128a
refactor: Remove debug print statements for cleaner code execution 🧹✨
kosiew Mar 21, 2025
839bf61
nested_struct - plug adapter into ListingTableConfig
kosiew Mar 24, 2025
2e99158
feat: Add optional schema adapter factory to ListingTableConfig for e…
kosiew Mar 24, 2025
fe7ff84
feat: Add optional schema adapter factory to FileScanConfig for enhan…
kosiew Mar 24, 2025
3689140
feat: Enhance ListingTableConfig to support schema adapter factory fo…
kosiew Mar 24, 2025
76fbc6f
struct NestedStructSchemaMapping - remove table_schema, file_schema
kosiew Mar 25, 2025
f2d6b60
refactor: Remove nested_struct.rs example for schema evolution and co…
kosiew Mar 25, 2025
6b7fed9
style: Fix comment tests in ListingOptions documentation 📜✨
kosiew Mar 25, 2025
2cef654
Merge branch 'main' into test-merge
kosiew Mar 25, 2025
565ad5c
SchemaMapping remove table_schema, nested_schema_adapter remove map_p…
kosiew Mar 25, 2025
778da1e
docs: Update comments for schema_adapter_factory in ListingTableConfi…
kosiew Mar 25, 2025
f066e59
refactor: Extract schema adapter preservation logic into a helper fun…
kosiew Mar 25, 2025
4cc5f77
refactor: Extract schema adapter application logic into a dedicated f…
kosiew Mar 25, 2025
b6a828c
docs: Enhance adapt_fields documentation with performance considerati…
kosiew Mar 25, 2025
41fb40c
docs: Add detailed documentation for RecordBatch mapping in NestedStr…
kosiew Mar 25, 2025
3133cd7
refactor: Add missing import for FileSource in ListingTable implement…
kosiew Mar 25, 2025
5ad6287
refactor: Update license documentation comments for NestedSchemaAdapt…
kosiew Mar 25, 2025
8fa34da
refactor: Remove unused file_scan_exec.rs to clean up the codebase 🗑️✨
kosiew Mar 25, 2025
d229dd3
refactor: Remove unused file_scan_config.rs to streamline the codebas…
kosiew Mar 25, 2025
ff41c43
Moved the adapt_column method from NestedStructSchemaMapping to a sta…
kosiew Mar 25, 2025
2df74b6
Fix Clippy errors
kosiew Mar 25, 2025
bb4a5de
docs: Correct the struct names in documentation for NestedStructSchem…
kosiew Mar 25, 2025
a8cce59
Merge branch 'main' into schema-adapter
kosiew Mar 25, 2025
f547355
fix: remove unnecessary clone in create_physical_plan call for Listin…
kosiew Mar 25, 2025
fa7c17f
refactor: rename preserve_schema_adapter_factory to preserve_conf_sch…
kosiew Mar 25, 2025
e9c93d6
refactor: rename create_appropriate_adapter to create_adapter for cla…
kosiew Mar 25, 2025
64a4e3f
feature gate parquet
kosiew Mar 26, 2025
dd9f66d
Trigger CI
kosiew Mar 26, 2025
ca511df
refactor: mod tests, add user_infos
kosiew Mar 26, 2025
54590f4
feat: expose nested schema adapter and source for improved data handl…
kosiew Mar 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
refactor: mod tests, add user_infos
- Introduced helper functions `create_flat_schema` and `create_nested_schema` to streamline schema creation for tests.
- Updated `test_nested_struct_evolution` to improve clarity and structure, ensuring it verifies schema adaptation correctly.
- Enhanced assertions in schema mapping tests to provide clearer error messages and improve test reliability.
- Added detailed checks for field structures in `test_adapt_column_with_nested_struct`, ensuring all expected fields are present and correctly structured.
- Improved overall organization of test cases for better readability and maintainability.
kosiew committed Mar 26, 2025
commit ca511df51eec4da94a5401fb2d64d6b3c51655a0
Loading