Project Nessie is a Transactional Catalog for Data Lakes with Git-like semantics.
More information can be found at projectnessie.org.
Nessie supports Iceberg Tables/Views and Delta Lake Tables. Additionally, Nessie is focused on working with the widest range of tools possible, which can be seen in the feature matrix.
You can quickly get started with Nessie by using our small, fast docker image.
IMPORTANT NOTE Nessie moves away from
docker.io to GitHub's container registry
quay.io. Previous releases are already available on both ghcr.io and quay.io. Please
update references to
projectnessie/nessie in your code to either
docker pull ghcr.io/projectnessie/nessie docker run -p 19120:19120 ghcr.io/projectnessie/nessie
For trying Nessie image with different configuration options, refer to the templates under the docker module.
A local Web UI will be available at this point.
Then install the Nessie CLI tool (to learn more about CLI tool and how to use it, check Nessie CLI Documentation).
pip install pynessie
From there, you can use one of our technology integrations such those for
To learn more about all supported integrations and tools, check here
Have fun! We have a Google Group and a Slack channel we use for both developers and users. Check them out here.
By default, Nessie servers run with authentication disabled and all requests are processed under the "anonymous" user identity.
Nessie supports bearer tokens and uses OpenID Connect for validating them.
Authentication can be enabled by setting the following Quarkus properties:
quarkus.oidc.auth-server-url=<OpenID Server URL>
Experimenting with Nessie Authentication in Docker
One can start the
projectnessie/nessie docker image in authenticated mode by setting
the properties mentioned above via docker environment variables. For example:
docker run -p 19120:19120 \ -e QUARKUS_OIDC_CLIENT_ID=<Client ID> \ -e QUARKUS_OIDC_AUTH_SERVER_URL=<OpenID Server URL> \ -e NESSIE_SERVER_AUTHENTICATION_ENABLED=true \ --network host \ ghcr.io/projectnessie/nessie
Building and Developing Nessie
- JDK 11 or higher: JDK11 or higher is needed to build Nessie (artifacts are built for Java 8)
Clone this repository:
git clone https://github.com/projectnessie/nessie cd nessie
Then open the project in IntelliJ or Eclipse, or just use the IDEs to clone this github repository.
Refer to CONTRIBUTING for build instructions.
Nessie Iceberg's integration is compatible with Iceberg as in the following table:
|Nessie version||Iceberg version||Spark version||Hive version||Flink version||Presto version|
|0.54.0||1.2.0||3.1.x (Scala 2.12), 3.2.x (Scala 2.12+2.13), 3.3.x (Scala 2.12+2.13)||n/a||1.14.x, 1.15.x, 1.16.x||0.277, 0.278.x, 0.279|
Nessie Delta Lake's integration is compatible with Delta Lake as in the following table:
|Nessie version||Delta Lake version||Spark version|
Delta Lake artifacts
Nessie required some minor changes to Delta for full support of branching and history. These changes are currently being integrated into the mainline repo. Until these have been merged we have provided custom builds in our fork which can be downloaded from a separate maven repository.
- configuration in
- go to
To run the ui (from
- If you are running in test ensure that
setupProxy.jspoints to the correct api instance. This ensures we avoid CORS issues in testing
npm installwill install dependencies
npm run startto start the ui in development mode via node
To deploy the ui (from
npm installwill install dependencies
npm buildwill minify and collect the package for deployment in
builddirectory can be deployed to any static hosting environment or run locally as
serve -s build
Official Nessie images are built with support for multiplatform builds. But to quickly build a docker image for testing purposes, simply run the following command:
./gradlew :nessie-quarkus:clean :nessie-quarkus:quarkusBuild docker build -f ./tools/dockerbuild/docker/Dockerfile-jvm -t nessie-unstable:latest ./servers/quarkus-server
If you prefer to build a native docker image instead, add
-Pnative and use a dedicated Dockerfile:
./gradlew :nessie-quarkus:clean :nessie-quarkus:quarkusBuild -Pnative docker build -f ./tools/dockerbuild/docker/Dockerfile-native -t nessie-unstable:latest ./servers/quarkus-server
Warning: building native images this way may not work properly on macOS with Apple Silicon processors.
Check that your image is available locally:
You should see something like this:
REPOSITORY TAG IMAGE ID CREATED SIZE nessie-unstable latest 24bb4c7bd696 15 seconds ago 555MB
Once this is done you can run your image with
docker run -p 19120:19120 quay.io/nessie-unstable:latest, passing the relevant
environment variables, if any. Environment variables names must follow MicroProfile Config's mapping
You can also deploy to AWS lambda function by following the steps in
Nessie related repositories
- Nessie Demos: Demos for Nessie
- CEL Java: Java port of the Common Expression Language
- Nessie apprunner: Maven and Gradle plugins to use Nessie in integration tests.
The Nessie project uses the Google Java Code Style, scalafmt and pep8. See CONTRIBUTING.md for more information.