Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new notebook showing how to do WAP with branches #76

Merged
merged 4 commits into from
May 22, 2023

Conversation

rmoff
Copy link
Contributor

@rmoff rmoff commented Apr 25, 2023

This is based on the existing Iceberg - Integrated Audits Demo.ipynb, and shows how to use a branch-based approach for WAP.

@rdblue
Copy link

rdblue commented Apr 27, 2023

This looks great to me!

The only problem before I would publish this is that cherry-pick only works on one commit. You probably want to mention that where you call it out as "less elegant" (agreed!) because people may think that it would move more than one commit. Just note that it isn't a "rebase" maybe?

Also, thank you for taking the time @rmoff! Good to run into you again.

@rmoff
Copy link
Contributor Author

rmoff commented Apr 28, 2023

Good point, thanks @rdblue - I've updated it.

@chrigehr
Copy link

Thank you for the work @rmoff , very helpful.

I've been looking into branching and tagging capabilities of iceberg for some time now and was wondering if and how a wap approach could be used to handle duplicates.
You create a branch and add new records there. Now there can be duplicates in the branch related to the main branch. So far I have thought about an "Insert into ...where not exists...." from the branch to main. But maybe there are smarter alternatives?!?

But probably this is the wrong place for such a discussion, sorry for that.

@rdblue
Copy link

rdblue commented May 1, 2023

@Fokko, can you help out here? This is failing validation but it looks good to me.

@rmoff
Copy link
Contributor Author

rmoff commented May 15, 2023

Anything I can do here to help push this PR forward? :)

@rdblue
Copy link

rdblue commented May 21, 2023

@rmoff, the failure is this:

nbstripout...............................................................Failed
- hook id: nbstripout
- files were modified by this hook

Looks like you just need to run the pre-commit scripts locally to fix them. Then CI will pass because it doesn't need to make any changes.

@Fokko
Copy link
Contributor

Fokko commented May 22, 2023

@rmoff

Please run:

pip3 install pre-commit
pre-commit run --all-files

Commit the changes, and this PR is ready to go :)

@rmoff
Copy link
Contributor Author

rmoff commented May 22, 2023

@Fokko thanks - have pushed a change that should fix it.

@Fokko
Copy link
Contributor

Fokko commented May 22, 2023

I just ran the notebook and it looks great! Thanks for contributing @rmoff ! 👏🏻

@Fokko Fokko merged commit 9b915d2 into tabular-io:main May 22, 2023
1 check passed
@rdblue
Copy link

rdblue commented May 26, 2023

Thanks, @rmoff!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants