Feature: Write to branches #941

vinjai · 2024-07-18T17:04:14Z

Fixes: #306

(cherry picked from commit ad1ec67)

(cherry picked from commit 40cf10e)

(cherry picked from commit 6dbe68d)

(cherry picked from commit ad1ec67)

(cherry picked from commit 40cf10e)

(cherry picked from commit 6dbe68d)

vinjai · 2024-10-16T08:15:38Z

@sungwy @kevinjqliu
This PR is ready for review

vinjai · 2024-10-16T20:30:59Z

Fixed another bug.
Writes with same name but different ref types were being successful. This is a bug in the current release version too

Please review whenever you get some time.

kevinjqliu · 2024-10-19T18:33:44Z

Thanks for the contribution! I'll take a look.
I remember adding support for branch is complicated since we need to consider different edge cases.

vinjai · 2024-10-19T19:24:02Z

I have mostly tried to cover all edge cases.
The idea is that the branch is just another iceberg table where the snapshots append independently of the main branch.

I also agree with your concern.
If it helps, we can add more test cases

vinjai · 2024-11-07T22:23:16Z

Hey @kevinjqliu
Did you get a chance to look at this?

kevinjqliu

Thanks for the PR! And sorry for the delay, I was running the 0.8.0 release.
Generally LGTM, I left a comment about add more tests integrating with Spark

tests/integration/test_writes/test_writes.py

vinjai · 2024-11-14T01:09:08Z

Thank you for the review @kevinjqliu

vinjai · 2024-11-20T07:17:05Z

@kevinjqliu What are the next steps to get this merged?

Fokko · 2025-04-24T20:06:13Z

Thanks @dbuades for pinging me, and sorry for letting this one linger for so long. Let me review this tomorrow morning.

@vinjai Do you have time to resolve the conflicts by any chance?

vinjai · 2025-04-28T20:00:38Z

Hey @Fokko
Will try to resolve the conflicts over the weekend.

SebastienN15 · 2025-05-12T08:33:45Z

Hey @vinjai, did you get a chance to fix those conflicts? 🙂

SebastienN15 · 2025-05-16T14:26:58Z

@vinjai I fixed the conflicts on this branch, feel free to cherry pick the last commit #2009
Otherwise I can open a PR on my side and take it from there 🙂

vinjai · 2025-05-23T15:22:06Z

Thanks @SebastienN15 — I’ll review your commit and take it forward from there.
I’ll share the working PR tomorrow.

vinjai · 2025-05-26T04:30:02Z

Identified and fixed a bug related to empty tables.
Planning to add test cases to cover this scenario.

vinjai · 2025-05-27T14:28:55Z

Hey @Fokko
This PR is ready for review again

vinjai · 2025-06-01T16:45:37Z

Hey @Fokko
I noticed that the integration tests for Python 3.9 are failing.
Possibly, a runtime issue:

ServerError: NoSuchBucketException: The specified bucket does not exist (Service: S3, Status Code: 404, Request ID: 18436A88DE8BCC82, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8) (SDK Attempt Count: 1)

I ran the same tests locally with Python 3.9.6, and they passed without issues.
Is there any change required in the PR to resolve the above failure?

Fokko · 2025-06-01T19:48:48Z

@vinjai Thanks for pinging me. The issue has been fixed in #2049. Locally, you're still using an older version of minio, and therefore the tests pass. Could you pull in the latest main branch to fix the CI? Thanks!

vinjai · 2025-06-03T09:20:34Z

Hey @Fokko
I've pulled the latest changes from the main branch. Please re-trigger the workflow.

Fokko

Thanks @vinjai for working on this, and sorry for the long wait. I think this looks great, and let's move this forward 👍

Fokko · 2025-06-18T14:51:49Z

pyiceberg/table/__init__.py

+            if branch is None:
+                files = self._scan(row_filter=delete_filter, case_sensitive=case_sensitive).plan_files()
+            else:
+                files = self._scan(row_filter=delete_filter, case_sensitive=case_sensitive).use_ref(branch).plan_files()


Maybe in a subsequent PR we can pass in the ref to the constructor 👍

pyiceberg/table/__init__.py

Fokko · 2025-06-18T15:03:42Z

@vinjai I'm sorry, can you resolve the conflicts once more? I'll merge right after

SebastienN15 · 2025-06-23T14:56:38Z

Hey @vinjai, would you be able to resolve the conflicts? I would love to see this PR merged 🙏

vinjai · 2025-06-24T01:56:38Z

Hey @Fokko
I have resolved all conflicts and merged the recent changes.
While running integration tests, I am facing this error for a few tests:

java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (360448)
Allocator(toArrowBatchIterator) 0/360448/360448/9223372036854775807 (res/actual/peak/limit)

The same error is coming on the main branch too.

Any idea on how to resolve this?

vinjai · 2025-06-28T20:33:30Z

Hey @Fokko,

I tried the tests on a different machine. The tests are running fine.
Request you to run the workflow and merge this PR ahead.

Thanks

Fokko · 2025-07-02T22:50:53Z

@vinjai Thanks, let's move this forward. Thanks @SebastienN15 for pinging me on this matter :)

Fixes: apache#306 --------- Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>

kevinjqliu and others added 4 commits July 15, 2024 03:36

s/"main"/MAIN_BRANCH

d4ca653

(cherry picked from commit ad1ec67)

replace string literals

0b7aaaf

(cherry picked from commit 40cf10e)

default writes to main branch

23f04ec

(cherry picked from commit 6dbe68d)

Added some more methods for branches

af6ff9a

vinjai changed the title ~~Feature/write to branch~~ Feature: Write to branches Jul 18, 2024

kevinjqliu and others added 11 commits July 29, 2024 08:39

s/"main"/MAIN_BRANCH

6fbf3f1

(cherry picked from commit ad1ec67)

replace string literals

8ce1509

(cherry picked from commit 40cf10e)

default writes to main branch

6daf29e

(cherry picked from commit 6dbe68d)

Added some more methods for branches

09321cd

Merged with master

60fef31

Updated antries for branches

45b01a6

Resolved Merge Conflict

917108b

Fixed some bugs

917b044

Fixed bugs in delete and overwrite

398f6c0

Added tests and some refactoring

b7b8ba0

Added another integration test

ee591b4

vinjai marked this pull request as ready for review October 16, 2024 08:15

Fixed bug: concurrent same name branch and tag writes

e81907d

kevinjqliu added this to the PyIceberg 0.9.0 release milestone Oct 30, 2024

kevinjqliu reviewed Nov 10, 2024

View reviewed changes

tests/integration/test_writes/test_writes.py Outdated Show resolved Hide resolved

vinjai added 2 commits November 14, 2024 04:07

Merge with main branch

4cf9198

Added integration tests with spark

bc6fb68

vinjai requested a review from kevinjqliu November 20, 2024 10:00

dbuades mentioned this pull request Apr 24, 2025

Support writing to a branch #306

Closed

vinjai added 3 commits May 26, 2025 03:12

Merge branch 'main' into feature/write-to-branch

53a7f84

Fixed linter and code errors

e4463df

Fixed bug for empty tables

49f75b4

vinjai added 2 commits May 27, 2025 13:35

Fixed bugs and added more tests

4ed0607

changed design context for branch writes

958aac4

Merge branch 'main' into feature/write-to-branch

a0aae4d

Fokko approved these changes Jun 18, 2025

View reviewed changes

vinjai added 2 commits June 24, 2025 05:29

Merge branch 'main' into feature/write-to-branch

76249e9

Fixed linter, comments and other bugs

079802a

Usage of builder pattern

f45df8b

vinjai requested a review from Fokko June 28, 2025 06:29

Fokko approved these changes Jul 2, 2025

View reviewed changes

Fokko merged commit 33c8931 into apache:main Jul 2, 2025
10 checks passed

amitgilad3 pushed a commit to amitgilad3/iceberg-python that referenced this pull request Jul 7, 2025

Feature: Write to branches (apache#941)

3a37b52

Fixes: apache#306 --------- Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>

Feature: Write to branches #941

Feature: Write to branches #941

Uh oh!

Conversation

vinjai commented Jul 18, 2024

Uh oh!

vinjai commented Oct 16, 2024

Uh oh!

vinjai commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinjqliu commented Oct 19, 2024

Uh oh!

vinjai commented Oct 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vinjai commented Nov 7, 2024

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vinjai commented Nov 14, 2024

Uh oh!

vinjai commented Nov 20, 2024

Uh oh!

Fokko commented Apr 24, 2025

Uh oh!

vinjai commented Apr 28, 2025

Uh oh!

SebastienN15 commented May 12, 2025

Uh oh!

SebastienN15 commented May 16, 2025

Uh oh!

vinjai commented May 23, 2025

Uh oh!

vinjai commented May 26, 2025

Uh oh!

vinjai commented May 27, 2025

Uh oh!

vinjai commented Jun 1, 2025

Uh oh!

Fokko commented Jun 1, 2025

Uh oh!

vinjai commented Jun 3, 2025

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Fokko Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Fokko commented Jun 18, 2025

Uh oh!

SebastienN15 commented Jun 23, 2025

Uh oh!

vinjai commented Jun 24, 2025

Uh oh!

vinjai commented Jun 28, 2025

Uh oh!

Fokko commented Jul 2, 2025

Uh oh!

Uh oh!

Uh oh!

vinjai commented Oct 16, 2024 •

edited

Loading

vinjai commented Oct 19, 2024 •

edited

Loading