Skip to content

Add Iceberg Catalog documentation #9121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
May 29, 2025
Merged

Add Iceberg Catalog documentation #9121

merged 11 commits into from
May 29, 2025

Conversation

itaigilo
Copy link
Contributor

Adding documentation for the Iceberg Catalog (Enterprise).

@itaigilo itaigilo added docs Improvements or additions to documentation exclude-changelog PR description should not be included in next release changelog minor-change Used for PRs that don't require issue attached labels May 28, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented May 28, 2025 •
edited
Loading

♻️ PR Preview e74c8db has been successfully destroyed since this PR has been closed.

🤖 By surge-preview

itaigilo
itaigilo commented May 28, 2025
View reviewed changes
docs/howto/iceberg-catalog.md Outdated

{: .note}
> Iceberg Catalog is only available to licensed [lakeFS Enterprise]({% link enterprise/index.md %}) customers.
> [Contact us](https://info.lakefs.io/thanks-iceberg-catalog) to get started!
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already asked @keren-lakeFS about this landing page.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as long as it will return 200 for the links checker.

nopcoder
nopcoder requested changes May 28, 2025
View reviewed changes
docs/howto/iceberg-catalog.md Outdated
Comment on lines 33 to 34
- Merge changes safely with conflict detection
- Maintain a clear audit trail of table changes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure we provide these

docs/howto/iceberg-catalog.md Outdated

Local storage is currently no supported, but it will probably be in future releases.

## Future Releases
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think oz needs to review the roadmap, but from the prd it looks correct.

itaigilo
itaigilo commented May 28, 2025
View reviewed changes
itaigilo
itaigilo commented May 28, 2025
View reviewed changes
Copy link
Contributor Author

@itaigilo itaigilo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @nopcoder for the detailed review.
Made quite a few changes, PTAL again 🙏

docs/howto/iceberg-catalog.md Outdated
2. Configure your Iceberg clients to use the lakeFS REST catalog endpoint.
3. Use your lakeFS access key and secret for authentication.

#### Catalog Initialization Example (using `pyiceberg`)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can, but it'll include a decent amount of code, I think...

docs/howto/iceberg-catalog.md Outdated

</div>

<div markdown="2" id="trino">
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean, how to init the catalog?

docs/howto/iceberg-catalog.md Outdated
</div>
</div>

## Working with Namespaces and Tables
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the section title -
If it's not enough, lmk.

docs/howto/iceberg-catalog.md Outdated
#### Namespace Restrictions

- Repository and branch names must follow lakeFS naming conventions
- Namespace components cannot contain special characters except dots (.) for nesting
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This specifies that . is allowed...

docs/howto/iceberg-catalog.md Outdated

### Current Limitations

The following features are *not yet supported*, and are planned for future releases:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the text -
Not sure what you mean by "two sub-lists"...

@itaigilo itaigilo requested a review from nopcoder May 28, 2025 18:23
itaigilo and others added 3 commits May 28, 2025 19:24
@itaigilo @nopcoder
Apply suggestions from code review
e4c384d 
Co-authored-by: Barak Amar <barak.amar@treeverse.io>
@itaigilo
Update description
39234f0
@itaigilo
Merge branch 'docs/frosty-documentation' of github.com:treeverse/lake…
e8f7c76 
…FS into docs/frosty-documentation
@ozkatz
Copy link
Collaborator

ozkatz commented May 29, 2025

Thanks! Great stuff. I opened a PR on top of this one because I suggested too many changes :) most are either very minor or simply rename the feature. LMK what you think.

@itaigilo
Copy link
Contributor Author

Thanks! Great stuff. I opened a PR on top of this one because I suggested too many changes :) most are either very minor or simply rename the feature. LMK what you think.

Thanks @ozkatz for the corrections. They are looking good, and approved.

@ozkatz @itaigilo
lakeFS Tables docs (#9124)
cff4abf 
* lakeFS Tables docs

* Update docs/howto/lakefs-tables.md

Co-authored-by: itaigilo <itai.gilo@treeverse.io>

---------

Co-authored-by: itaigilo <itai.gilo@treeverse.io>
ozkatz
ozkatz approved these changes May 29, 2025
View reviewed changes
Copy link
Collaborator

@ozkatz ozkatz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@itaigilo itaigilo merged commit 04ac1ef into master May 29, 2025
42 checks passed
@itaigilo itaigilo deleted the docs/frosty-documentation branch May 29, 2025 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nopcoder nopcoder nopcoder approved these changes

@ozkatz ozkatz ozkatz approved these changes

@arielshaqed arielshaqed Awaiting requested review from arielshaqed

@itaiad200 itaiad200 Awaiting requested review from itaiad200

@talSofer talSofer Awaiting requested review from talSofer

@guy-har guy-har Awaiting requested review from guy-har

Assignees

@nopcoder nopcoder

Labels
docs Improvements or additions to documentation exclude-changelog PR description should not be included in next release changelog minor-change Used for PRs that don't require issue attached
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@itaigilo @ozkatz @nopcoder