Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Iceberg mixed partition table, amoro UI display and merged write directory error #1863

Closed
1 task done
Tracked by #2448
archer2012 opened this issue Aug 21, 2023 · 2 comments · Fixed by #2303
Closed
1 task done
Tracked by #2448
Labels
priority:critical type:bug Something isn't working

Comments

@archer2012
Copy link
Contributor

archer2012 commented Aug 21, 2023

What happened?

This table use month-day mixed partitioning.
c42459ec59d06a9b226943c83b71b58
f1d9fb44e27c871e167bca3992384a3

amoro UI display and merged write data to directory error(1970-xx-xx)
c0c99ad85398833a9387ba61830c70b

Affects Versions

0.5.x

What engines are you seeing the problem on?

AMS, Optimizer

How to reproduce

create table t1(id int, create_at timestamp) partition sep month(create_at) stored as iceberg tblpropertises('format'='v2');

insert into t1(1,'2022-01-01 01:00:00');
insert into t1(2,'2022-02-01 01:00:00');
insert into t1(3,'2022-03-01 01:00:00');
 
alter table t1 replace partition day(create_at);

insert into t1(1,'2023-01-01 01:00:00');
insert into t1(2,'2023-02-01 01:00:00');
insert into t1(3,'2023-03-01 01:00:00');

Relevant log output

No response

Anything else

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct
@archer2012 archer2012 added the type:bug Something isn't working label Aug 21, 2023
@shidayang
Copy link
Contributor

Thank you for reporting this issue. We need to confirm some details, but if I understand correctly, the problem is that after modifying the partition of Iceberg, the partition of the output data file is incorrect after self-optimizing, right?

@zhoujinsong
Copy link
Contributor

It seems that the reason for the issue is that when Iceberg contains files with multiple Partition Specs, Amoro currently uses the current Partition Spec to process all files in both file display and self-optimizing processes, which is incorrect and may cause problems that the issue described.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority:critical type:bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants