Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,15 @@ Apache Paimon Rust is an exciting project currently under active development. Wh
- Start discussion thread at [dev mailing list](mailto:dev@paimon.apache.org) ([subscribe](<mailto:dev-subscribe@paimon.apache.org?subject=(send%20this%20email%20to%20subscribe)>) / [unsubscribe](<mailto:dev-unsubscribe@paimon.apache.org?subject=(send%20this%20email%20to%20unsubscribe)>) / [archives](https://lists.apache.org/list.html?dev@paimon.apache.org))
- Talk to community directly at [Slack #paimon channel](https://join.slack.com/t/the-asf/shared_invite/zt-2l9rns8pz-H8PE2Xnz6KraVd2Ap40z4g).

## Documentation

The project documentation is built with [MkDocs](https://www.mkdocs.org/). See [docs/README.md](docs/README.md) for details.

```bash
pip3 install mkdocs-material
cd docs && mkdocs serve
```

## Getting help

Submit [issues](https://github.com/apache/paimon-rust/issues/new/choose) for bug report or asking questions in [discussion](https://github.com/apache/paimon-rust/discussions/new?category=q-a).
Expand Down
55 changes: 55 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Documentation

This directory contains the source files for the Apache Paimon Rust documentation site, built with [MkDocs](https://www.mkdocs.org/) and the [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/) theme.

## Prerequisites

- Python 3.8+
- pip3

## Setup

```bash
pip3 install mkdocs-material
```

## Development

Preview the docs locally with live reload:

```bash
cd docs
mkdocs serve
```

Then open [http://127.0.0.1:8000](http://127.0.0.1:8000) in your browser.

## Build

Generate the static site:

```bash
cd docs
mkdocs build
```

The output will be in the `docs/site/` directory.
65 changes: 65 additions & 0 deletions docs/mkdocs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

site_name: Apache Paimon Rust
site_description: The Rust implementation of Apache Paimon
site_url: https://apache.github.io/paimon-rust/
repo_url: https://github.com/apache/paimon-rust
repo_name: apache/paimon-rust

docs_dir: src

theme:
name: material
palette:
- scheme: default
primary: indigo
accent: indigo
toggle:
icon: material/brightness-7
name: Switch to dark mode
- scheme: slate
primary: indigo
accent: indigo
toggle:
icon: material/brightness-4
name: Switch to light mode
features:
- navigation.sections
- navigation.expand
- navigation.top
- search.suggest
- content.code.copy

nav:
- Home: index.md
- Getting Started: getting-started.md
- Architecture: architecture.md
- Releases: releases.md
- Contributing: contributing.md

markdown_extensions:
- admonition
- pymdownx.details
- pymdownx.superfences
- pymdownx.highlight:
anchor_linenums: true
- pymdownx.inlinehilite
- pymdownx.tabbed:
alternate_style: true
- toc:
permalink: true
60 changes: 60 additions & 0 deletions docs/src/architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Architecture

## Overview

Apache Paimon Rust is organized as a Cargo workspace with multiple crates, each responsible for a distinct layer of functionality.

## Crate Structure

### `crates/paimon` — Core Library

The core crate implements the Paimon table format, including:

- **Catalog** — Catalog client for discovering and managing databases and tables
- **Table** — Table abstraction for reading Paimon tables
- **Snapshot & Manifest** — Reading snapshot and manifest metadata
- **Schema** — Table schema management and evolution
- **File IO** — Abstraction layer for storage backends (local filesystem, S3)
- **File Format** — Parquet file reading and writing via Apache Arrow

### `crates/integrations/datafusion` — DataFusion Integration

Provides a `TableProvider` implementation that allows querying Paimon tables using [Apache DataFusion](https://datafusion.apache.org/)'s SQL engine.

## Data Model

Paimon organizes data in a layered structure:

```
Catalog
└── Database
└── Table
├── Schema
└── Snapshot
└── Manifest
└── Data Files (Parquet)
```

- **Catalog** manages databases and tables, accessed via REST API
- **Snapshot** represents a consistent view of a table at a point in time
- **Manifest** lists the data files that belong to a snapshot
- **Data Files** store the actual data in Parquet format
62 changes: 62 additions & 0 deletions docs/src/contributing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Contributing

Apache Paimon Rust welcomes contributions from everyone. See the full [Contributing Guide](https://github.com/apache/paimon-rust/blob/main/CONTRIBUTING.md) for detailed instructions.

## Quick Start

1. Fork the [repository](https://github.com/apache/paimon-rust)
2. Clone your fork: `git clone https://github.com/<your-username>/paimon-rust.git`
3. Create a feature branch: `git checkout -b feature/my-feature`
4. Make your changes and add tests
5. Run checks locally before submitting
6. Open a Pull Request

## Development Setup

```bash
# Ensure you have the correct Rust toolchain
rustup show

# Build the project
cargo build

# Run all tests
cargo test

# Format code
cargo fmt

# Lint (matches CI)
cargo clippy --all-targets --workspace -- -D warnings
```

## Finding Issues

- Check [open issues](https://github.com/apache/paimon-rust/issues) for tasks to work on
- Issues labeled `good first issue` are great starting points
- See the [0.1.0 tracking issue](https://github.com/apache/paimon-rust/issues/3) for the current roadmap

## Community

- **GitHub Issues**: [apache/paimon-rust/issues](https://github.com/apache/paimon-rust/issues)
- **Mailing List**: [dev@paimon.apache.org](mailto:dev@paimon.apache.org) ([subscribe](mailto:dev-subscribe@paimon.apache.org) / [archives](https://lists.apache.org/list.html?dev@paimon.apache.org))
- **Slack**: [#paimon channel](https://join.slack.com/t/the-asf/shared_invite/zt-2l9rns8pz-H8PE2Xnz6KraVd2Ap40z4g) on the ASF Slack
Loading
Loading