Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/rust.yml
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ jobs:
cargo build --no-default-features --features http,rustls
cargo build --no-default-features --features ftp,http,rustls
cargo build --no-default-features --features s3,http,rustls
cargo build --no-default-features --features reqwest-gzip,rustls

- name: Build CLI feature
run: cargo build --no-default-features --features cli,http,gz,bz,rustls
Expand All @@ -137,6 +138,9 @@ jobs:
- name: Test with no features
run: cargo test --no-default-features

- name: Test reqwest-gzip feature
run: cargo test --no-default-features --features reqwest-gzip,rustls --test http_advanced_tests

- name: Test documentation
run: cargo test --doc --all-features

Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ All notable changes to this project will be documented in this file.
### Bug fixes
- Preserve leading-slash S3 object keys when using path-style endpoints such as Cloudflare R2.

### Added
- Re-exported `reqwest` as `oneio::reqwest` (under the `http` feature) so downstream crates can name HTTP types (`StatusCode`, `header`, `blocking::Response`) without declaring their own reqwest dependency and risking version skew. Note: this makes reqwest part of oneio's public API contract; a reqwest major-version bump is a breaking oneio change.
- New opt-in `reqwest-gzip` feature: advertises `Accept-Encoding: gzip` and transparently decodes `Content-Encoding: gzip` responses (e.g. ~97 MB to ~4.6 MB for `rpki.cloudflare.com/rpki.json`). Distinct from the `gz` family, which is URL-suffix-based file decompression. Off by default; no dependency-tree change unless enabled.

## v0.23.0 -- 2026-05-12

### Added
Expand Down
5 changes: 5 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,11 @@ https = ["http", "rustls"] # https needs http
ftp = ["https", "suppaftp"] # ftp needs https
s3 = ["rusty-s3", "http", "quick-xml", "percent-encoding", "dep:sha2", "dep:hmac", "dep:hex"]

# HTTP content-encoding (opt-in, additive passthrough to reqwest)
# Advertises `Accept-Encoding: gzip` and transparently decodes gzipped responses.
# Distinct from the `gz` family below, which is URL-suffix-based file decompression.
reqwest-gzip = ["http", "reqwest/gzip"]

gz = ["gz-zlib-rs"]
# internal feature to enable gzip support
any_gz = []
Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ oneio = { version = "0.23", features = ["async"] }
- `https` - HTTP/HTTPS with rustls TLS backend (equivalent to `http` + `rustls`)
- `ftp` - FTP support (requires `http` + TLS backend)
- `s3` - S3-compatible storage
- `reqwest-gzip` - Opt-in HTTP gzip content-encoding: advertises `Accept-Encoding: gzip` and transparently decodes gzipped responses (distinct from `gz`, which is URL-suffix-based file decompression)

**TLS Backends** (for HTTPS - mutually exclusive):
- `rustls` - Pure Rust TLS (use with `http`). Uses both system certificates and bundled Mozilla certificates for maximum compatibility.
Expand Down Expand Up @@ -160,7 +161,8 @@ The `OneIo` client allows you to configure headers, TLS certificates, timeouts,

```rust
use oneio::OneIo;
use reqwest::header::{HeaderName, HeaderValue};
// reqwest is re-exported for naming HTTP types without a direct reqwest dependency
use oneio::reqwest::header::{HeaderName, HeaderValue};

// Build a reusable client with custom headers and certificates
let oneio = OneIo::builder()
Expand Down
129 changes: 129 additions & 0 deletions specs/03-reqwest-reexport-http-gzip/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# Spec: reqwest Re-export and Opt-in HTTP gzip

**Status**: Complete
**Author**: Mingwei Zhang
**Created**: 2026-07-22
**Target Branch**: `dev/reqwest-reexport-http-gzip`

## 1. Overview

Re-export `reqwest` so downstream crates can name the HTTP types exposed through
oneio's public API, and add an opt-in `reqwest-gzip` feature that enables
reqwest's transparent gzip content-encoding support.

**Non-goals:**
- Changing any default behavior — the gzip feature is off unless explicitly enabled
- Brotli/deflate content-encoding — can follow the same pattern later if requested
- Replacing oneio's suffix-based file decompression (`gz` family) — orthogonal concern
- Wrapping or abstracting reqwest's API — the goal is to expose it, not hide it

**Success criteria:**
- [ ] `oneio::reqwest` resolves when the `http` feature is enabled (e.g.
`oneio::reqwest::StatusCode::NOT_MODIFIED` compiles downstream)
- [ ] With `reqwest-gzip`, requests advertise `Accept-Encoding: gzip` and
`Content-Encoding: gzip` responses are transparently decoded
- [ ] Without `reqwest-gzip`, dependency tree is unchanged (no gzip decoder crates)
- [ ] Conditional-GET workflow (send `If-None-Match`, read `304` status and
`ETag`/`Last-Modified` response headers) is possible using only oneio's API
- [ ] All quality gates pass: fmt, build, test, clippy (`-D warnings`)
Comment on lines +20 to +28

## 2. Current State

`OneIo::get_http_reader_raw()` (since 0.21) returns `reqwest::blocking::Response`
in its public signature, and `OneIo::builder()` / `OneIo::http_client()` expose
reqwest types as well. However, reqwest is an internal dependency: downstream
crates cannot `use reqwest::...` unless they declare reqwest themselves, which
risks version skew (two reqwest major versions in the tree; the `Response` they
receive is not the `Response` they imported). Working around it via type
inference (`.status().as_u16() == 304`) is fragile and implicit.

Separately, oneio's reqwest build enables `blocking`, `http2`, `charset`,
`stream` — but not `gzip`. Requests therefore do not advertise
`Accept-Encoding: gzip`, and `Content-Encoding: gzip` responses are not decoded.
oneio's own decompression is inferred from the URL suffix, so payload-gzipped
endpoints like `https://rpki.cloudflare.com/rpki.json` (~97 MB plain, ~4.6 MB
gzipped) cannot benefit.

Concrete downstream need: bgpkit-commons#33 (conditional RPKI loading) currently
adds a direct reqwest dependency solely to name these types and enable gzip.

## 3. Proposed Solution

### Re-export

```rust
#[cfg(feature = "http")]
pub use reqwest;
```

Downstream then uses `oneio::reqwest::{blocking::Client, StatusCode, header}`
guaranteed to be the exact reqwest oneio was built against.

**Key decision — full re-export vs curated module:** full `pub use reqwest`
(chosen) is simpler and the type already leaks via `get_http_reader_raw`; a
curated `oneio::http` module would not actually reduce the semver exposure.

**Semver implication (documented in the PR):** reqwest becomes part of oneio's
public API contract. A future reqwest 0.13 upgrade becomes a breaking oneio
change (same trade-off as tokio↔bytes). Acceptable: oneio is pre-1.0 and
already exposes reqwest types in signatures.

### Feature flag

```toml
reqwest-gzip = ["http", "reqwest/gzip"]
```

reqwest's `gzip` feature is purely additive: it advertises
`Accept-Encoding: gzip` and transparently decodes gzipped responses. No oneio
code changes or `#[cfg]` gating are required — the flag only forwards.

**Key decision — naming:** `reqwest-gzip` (not `gzip`) to avoid confusion with
the existing `gz`/`gz-*` family, which is suffix-based file decompression.
Pattern extends naturally to `reqwest-brotli` / `reqwest-deflate` later.

## 4. Implementation Plan

1. **Cargo.toml**: add `reqwest-gzip = ["http", "reqwest/gzip"]` feature.
Acceptance: `cargo tree --no-default-features --features reqwest-gzip` shows
a gzip decoder backend; without the feature it does not.
2. **src/lib.rs**: add the gated re-export with doc comment; add `reqwest-gzip`
row to the feature table in the crate docs.
Acceptance: doc test using `oneio::reqwest::StatusCode` compiles.
3. **tests/**: gzip integration test (mock server, gzipped body with
`Content-Encoding: gzip`, assert transparent decode) gated on
`reqwest-gzip`; conditional-GET example test using the re-export.
Acceptance: tests pass under `--features reqwest-gzip` and are absent/skipped
otherwise.
4. **Docs/CI/changelog**: README feature mention, CI build/clippy legs for the
new feature, CHANGELOG entry under Unreleased.
Acceptance: CI matrix covers `reqwest-gzip`.

## 5. Testing Strategy

- Integration test with in-process `TcpListener` mock server (pattern already
used in `tests/basic_integration.rs`):
- serve a gzip-compressed body (compressed with flate2, already in-tree via
the `gz` features) with `Content-Encoding: gzip`; assert decoded content
matches and the request carried `Accept-Encoding: gzip`
- serve `304 Not Modified` after an initial `200` with `ETag`; assert the
client can read status and validators via `oneio::reqwest` types
- Doc test on the re-export (compile-time check downstream usage works)

## 6. Risks

- **Semver coupling**: reqwest version becomes public API → documented in
CHANGELOG and PR; mitigated by oneio being pre-1.0 with reqwest already in
public signatures.
- **Feature unification surprises**: a downstream crate enabling
`reqwest/gzip` itself would silently change oneio's wire behavior (gzip is
additive) — this is standard Cargo feature semantics, noted in docs.
- **MSRV**: reqwest's gzip feature pulls `async-compression`/`flate2`; flate2
is already a oneio dependency via the `gz` features, so MSRV is unaffected.

## 7. Decision Log

- 2026-07-22: Full `pub use reqwest` over curated re-export module (the type
already leaks; curation adds maintenance without reducing exposure)
- 2026-07-22: Feature named `reqwest-gzip` to disambiguate from the
suffix-based `gz` family; passthrough-only, no cfg-gated code
1 change: 1 addition & 0 deletions specs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ This directory contains design specifications for oneio features.
|---|---|------|
| 01 | [rusty-s3-migration](rusty-s3-migration/README.md) | Complete |
| 02 | [lossy-read](02-lossy-read/README.md) | Draft |
| 03 | [reqwest-reexport-http-gzip](03-reqwest-reexport-http-gzip/README.md) | Complete |

---

Expand Down
43 changes: 43 additions & 0 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ Enable only what you need:
| `xz` | XZ compression |
| `zstd` | Zstandard compression |
| `http` | HTTP/HTTPS support |
| `reqwest-gzip` | Opt-in HTTP gzip content-encoding (advertises `Accept-Encoding: gzip`, transparently decodes responses) |
| `ftp` | FTP support |
| `s3` | S3-compatible storage |
| `async` | Async I/O support |
Expand Down Expand Up @@ -241,6 +242,48 @@ pub use builder::OneIoBuilder;
pub use client::OneIo;
pub use error::OneIoError;

/// Re-export of the exact `reqwest` crate oneio is built against.
///
/// HTTP response types (e.g. [`OneIo::get_http_reader_raw`] returning
/// `reqwest::blocking::Response`) appear in oneio's public API. This re-export
/// lets downstream crates name those types (`oneio::reqwest::StatusCode`,
/// `oneio::reqwest::header`, ...) without declaring their own reqwest
/// dependency, avoiding version skew between the reqwest oneio uses and the
/// reqwest a consumer imports.
///
/// Note: this makes reqwest part of oneio's public API contract; a reqwest
/// major-version bump is a breaking oneio change.
///
/// # Example: conditional GET with HTTP validators
///
/// ```rust,no_run
/// # #[cfg(feature = "http")]
/// # fn main() -> Result<(), Box<dyn std::error::Error>> {
/// use oneio::reqwest::StatusCode;
/// use oneio::OneIo;
///
/// let client = OneIo::builder()
/// .header_str("If-None-Match", "\"some-etag\"")
/// .build()?;
/// let response = client.get_http_reader_raw("https://example.com/data.json")?;
/// if response.status() == StatusCode::NOT_MODIFIED {
/// // cached copy is still current; skip re-processing
/// } else {
/// let etag = response
/// .headers()
/// .get("etag")
/// .and_then(|v| v.to_str().ok())
/// .map(String::from);
/// // read `response` (implements `std::io::Read`) and store `etag`
/// }
/// # Ok(())
/// # }
/// # #[cfg(not(feature = "http"))]
/// # fn main() {}
/// ```
#[cfg(feature = "http")]
pub use reqwest;

#[cfg(feature = "async")]
pub mod async_reader;
#[cfg(feature = "rustls")]
Expand Down
Loading