Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nested structure #14

Closed
balluashok opened this issue Apr 27, 2019 · 5 comments
Closed

Nested structure #14

balluashok opened this issue Apr 27, 2019 · 5 comments
Labels
question Questions on how to use Delta Lake

Comments

@balluashok
Copy link

Is Delta supports nested structure too ?
@mukulmurthy
Copy link
Collaborator

Hi @balluashok,

By nested structure, do you mean nested columns in a Delta table? We've made an effort to have everything in Delta Lake work with nested columns; please let us know if you encounter any issues.

@balluashok
Copy link
Author

Thanks @mukulmurthy

Yest nested columns assume : Person can have multiple addresses(work address, home address), can have multiple contact numbers ( mobile, home, work).

we have to represented this relation in nested structure -> one person can have many addresses and can have many contact numbers.

And Person can change only his cell number or can change work address.. then we have to keep the history and SANPSHOT(latest data).. let me know we can achieve this use case using Delta.

@balluashok balluashok changed the title Nested structure Nested structure-question Apr 28, 2019
@balluashok balluashok changed the title Nested structure-question Nested structure Apr 28, 2019
@mukulmurthy
Copy link
Collaborator

Delta column types support all columns Spark supports, including ArrayType and StructType (nested types) if that's what you're using. Partitioning on nested columns is also supported.

@balluashok
Copy link
Author

Thanks @mukulmurthy , i will do a POC.

@tdas tdas added the question Questions on how to use Delta Lake label Apr 29, 2019
LantaoJin added a commit to LantaoJin/delta that referenced this issue Mar 24, 2020
@LantaoJin
[CARMEL-2116] Auto VACUUM (delta-io#14)
d8f698e
@tdas tdas closed this as completed Mar 26, 2020
LantaoJin added a commit to LantaoJin/delta that referenced this issue Mar 12, 2021
@LantaoJin
[CARMEL-4240] Check log exists before writing to avoid versions not c…
7b6f113 
…ontiguous (delta-io#14)
jbguerraz pushed a commit to jbguerraz/delta that referenced this issue Jul 6, 2022
@JassAbidi
Merge pull request delta-io#14 from delta-io/master
5717ee3 
update fork
@mrijken
Copy link

mrijken commented Jan 22, 2024

Partition on nested columns (ie partitionBy("struct.column")) is not possible. I assume that https://issues.apache.org/jira/browse/SPARK-18084 is the reason.

andreaschat-db added a commit to andreaschat-db/delta that referenced this issue Apr 23, 2024
@andreaschat-db
# This is a combination of 16 commits.
ad044ef 
# This is the 1st commit message:

flush

# This is the commit message delta-io#2:

flush

# This is the commit message delta-io#3:

First sane version without isRowDeleted

# This is the commit message delta-io#4:

Hack RowIndexMarkingFilters

# This is the commit message delta-io#5:

Add support for non-vectorized readers

# This is the commit message delta-io#6:

Metadata column fix

# This is the commit message delta-io#7:

Avoid non-deterministic UDF to filter deleted rows

# This is the commit message delta-io#8:

metadata with Expression ID

# This is the commit message delta-io#9:

Fix complex views issue

# This is the commit message delta-io#10:

Tests

# This is the commit message delta-io#11:

cleaning

# This is the commit message delta-io#12:

More tests and fixes

# This is the commit message delta-io#13:

Partial cleaning

# This is the commit message delta-io#14:

cleaning and improvements

# This is the commit message delta-io#15:

cleaning and improvements

# This is the commit message delta-io#16:

Clean RowIndexFilter
andreaschat-db added a commit to andreaschat-db/delta that referenced this issue Apr 26, 2024
@andreaschat-db
# This is a combination of 16 commits.
0826e53 
# This is the 1st commit message:

flush

# This is the commit message delta-io#2:

flush

# This is the commit message delta-io#3:

First sane version without isRowDeleted

# This is the commit message delta-io#4:

Hack RowIndexMarkingFilters

# This is the commit message delta-io#5:

Add support for non-vectorized readers

# This is the commit message delta-io#6:

Metadata column fix

# This is the commit message delta-io#7:

Avoid non-deterministic UDF to filter deleted rows

# This is the commit message delta-io#8:

metadata with Expression ID

# This is the commit message delta-io#9:

Fix complex views issue

# This is the commit message delta-io#10:

Tests

# This is the commit message delta-io#11:

cleaning

# This is the commit message delta-io#12:

More tests and fixes

# This is the commit message delta-io#13:

Partial cleaning

# This is the commit message delta-io#14:

cleaning and improvements

# This is the commit message delta-io#15:

cleaning and improvements

# This is the commit message delta-io#16:

Clean RowIndexFilter
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Questions on how to use Delta Lake
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tdas @mrijken @balluashok @mukulmurthy