Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
# Changelog

## [0.2.0] - 2024-10-25

## What's Changed

- Streaming list results. `list` now returns an async or sync generator. by @kylebarron in https://github.com/developmentseed/obstore/pull/35
- Optionally return list result as arrow. The `return_arrow` keyword argument returns chunks from `list` as Arrow RecordBatches, which is faster than materializing Python dicts/lists. by @kylebarron in https://github.com/developmentseed/obstore/pull/38
- Return buffer protocol object from `get_range` and `get_ranges`. Enables zero-copy data exchange from Rust into Python. by @kylebarron in https://github.com/developmentseed/obstore/pull/39
- Add put options. Enables custom tags and attributes, as well as "put if not exists". by @kylebarron in https://github.com/developmentseed/obstore/pull/50
- Rename to obstore by @kylebarron in https://github.com/developmentseed/obstore/pull/45
- Add custom exceptions. by @kylebarron in https://github.com/developmentseed/obstore/pull/48

**Full Changelog**: https://github.com/developmentseed/obstore/compare/py-v0.1.0...py-v0.2.0

## [0.1.0] - 2024-10-21

- Initial release.
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

22 changes: 9 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,34 +1,30 @@
# obstore

[![PyPI][pypi_badge]][pypi_link]

<!-- [![Conda Version][conda_version_badge]][conda_version] -->

[pypi_badge]: https://badge.fury.io/py/obstore.svg
[pypi_link]: https://pypi.org/project/obstore/

<!-- [conda_version_badge]: https://img.shields.io/conda/vn/conda-forge/obstore.svg
[conda_version]: https://anaconda.org/conda-forge/obstore -->

A Python interface and [pyo3](https://github.com/PyO3/pyo3) integration to the Rust [`object_store`](https://docs.rs/object_store) crate, providing a uniform API for interacting with object storage services and local files.
Simple, fast integration with object storage services like Amazon S3, Google Cloud Storage, Azure Blob Storage, and S3-compliant APIs like Cloudflare R2.

Run the same code in multiple clouds via a simple runtime configuration change.

<!-- For Rust developers looking to add object_store support to their Python packages, refer to pyo3-object_store. -->

- Easy to install with no Python dependencies.
- Sync and async API.
- Streaming downloads with configurable chunking.
- Streaming `list`, with no need to paginate.
- Support for conditional put ("put if not exists"), as well as custom tags and attributes.
- Automatically supports [multipart uploads](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) under the hood for large file objects.
- Optionally return list results as [Arrow](https://arrow.apache.org/), which is faster than materializing Python `dict`/`list` objects.
- Easy to install with no required Python dependencies.
- The [underlying Rust library](https://docs.rs/object_store) is production quality and used in large scale production systems, such as the Rust package registry [crates.io](https://crates.io/).
- Support for zero-copy data exchange from Rust into Python in `get_range` and `get_ranges`.
- Simple API with static type checking.
- Helpers for constructing from environment variables and `boto3.Session` objects

Supported object storage providers include:

- Amazon S3 and S3-compliant APIs like Cloudflare R2
- Google Cloud Storage
- Azure Blob Gen1 and Gen2 accounts (including ADLS Gen2)
- Local filesystem
- In-memory storage
<!-- For Rust developers looking to add object_store support to their Python packages, refer to pyo3-object_store. -->

## Installation

Expand Down
2 changes: 1 addition & 1 deletion obstore/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "obstore"
version = "0.2.0-beta.2"
version = "0.2.0"
authors = { workspace = true }
edition = { workspace = true }
description = "A Python interface to the Rust object_store crate, providing a uniform API for interacting with object storage services and local files."
Expand Down
2 changes: 2 additions & 0 deletions obstore/python/obstore/_list.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,8 @@ def list(
overhead between Rust and Python and so this can be significantly faster for
large list operations. Defaults to `False`.

If this is `True`, the `arro3-core` Python package must be installed.

Returns:
A ListStream, which you can iterate through to access list results.
"""
Expand Down