Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ env:
# PXF v0.72-series feature set (@<name> / @entry / @table directive
# grammar, schema validator, Result accessors, TableReader streaming)
# the Python port wraps. Bump in lockstep with cpp release cuts.
PROTOWIRE_CPP_REF: v0.75.0
PROTOWIRE_CPP_REF: v1.0.0

jobs:
# ---------------------------------------------------------------------
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ permissions:

env:
# See ci.yml for the rationale on this pin.
PROTOWIRE_CPP_REF: v0.75.0
PROTOWIRE_CPP_REF: v1.0.0

jobs:
analyze:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ env:
# feature set (@<name> / @entry / @table grammar, schema validator,
# Result accessors, TableReader streaming) that this Python port
# wraps. Bump in lockstep with cpp release cuts.
PROTOWIRE_CPP_REF: v0.75.0
PROTOWIRE_CPP_REF: v1.0.0

jobs:
# ---------------------------------------------------------------------
Expand Down
49 changes: 49 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,55 @@ format changes.

## [Unreleased]

## [1.0.0] — 2026-05-13

First major-version cut. Implements the three one-time spec changes
from the [protowire v1.0 freeze line](https://github.com/trendvidia/protowire/releases/tag/v1.0.0)
in lockstep with the other v1.0 ports, by re-pinning to
[`protowire-cpp` v1.0.0](https://github.com/trendvidia/protowire-cpp/releases/tag/v1.0.0).
**Breaking** — there is no alias period; v1.0 is itself the major
bump.

### Python API rename

- `pxf.TableDirective` → `pxf.DatasetDirective`
- `pxf.TableReader` → `pxf.DatasetReader`
- `pxf.Result.tables` → `pxf.Result.datasets`
- Tuple cell type aliases are unchanged.

Source files renamed:

- `tests/test_pxf_table_reader.py` → `tests/test_pxf_dataset_reader.py`

### Python API additions

- `pxf.ProtoDirective` (frozen dataclass with `shape`, `type_name`,
`body`) — exposed in `Result.protos`. The `shape` field is the
string literal `"anonymous" | "named" | "source" | "descriptor"`,
matching the cpp enum.
- `pxf.ProtoShape` type alias for the literal union.

### FFI shape

`_protowire.pxf_unmarshal_full` now returns a 6-tuple (raw bytes,
set_paths, null_paths, directives, datasets, **protos**). Python's
`Result` gains a `protos: tuple[ProtoDirective, ...]` field.

### Build

- `pyproject.toml` version `0.75.0` → `1.0.0`.
- `__version__` in `src/protowire/__init__.py` bumped accordingly.
- Sibling-checkout dependency on `../protowire-cpp` resolves at the
v1.0.0 tag.

### Tests

- New `tests/test_pxf_proto_directive.py` with 16 cases covering all
four `@proto` body shapes via the FFI roundtrip, multi-`@proto`,
nested-brace bodies, three error paths, parametrized reserved-
directive-name rejection, and a `ProtoDirective` dataclass check.
- pytest: 100 tests, 0 failures.

## [0.75.0] — 2026-05-12

First release after the v0.70.0 baseline. Wraps the
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ build-backend = "scikit_build_core.build"
# 2021 CLI tool). The import name stays `import protowire` — these two
# names are independent (cf. python-dateutil → import dateutil).
name = "protowire-python"
version = "0.75.0"
version = "1.0.0"
description = "Python wrapper around protowire-cpp — PXF text, SBE binary, and envelope codecs."
readme = "README.md"
requires-python = ">=3.10"
Expand Down
89 changes: 52 additions & 37 deletions src/_protowire/module.cc
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ const pbuf::Descriptor* FindDescriptor(const SchemaBundle& s,

// CellToPyTuple converts a single AST cell value (or std::nullopt for an
// absent cell) into the FFI shape consumed by pxf.py — `None` for absent,
// `(kind, value)` otherwise. Used by PxfUnmarshalFull for @table rows.
// `(kind, value)` otherwise. Used by PxfUnmarshalFull for @dataset rows.
//
// kind values mirror the AST variant tags:
// "null" → nb::none()
Expand Down Expand Up @@ -121,7 +121,7 @@ nb::object CellToPyTuple(const std::optional<protowire::pxf::ValuePtr>& cell) {
} else if constexpr (std::is_same_v<T, DurationVal>) {
return nb::make_tuple(std::string("duration"), p->raw);
} else {
// List / Block are rejected at @table cell-parse time, so this
// List / Block are rejected at @dataset cell-parse time, so this
// branch is unreachable for cells. Surface as a clean error.
return nb::make_tuple(std::string("unknown"), nb::none());
}
Expand Down Expand Up @@ -156,14 +156,19 @@ nb::bytes PxfUnmarshal(nb::bytes text, nb::bytes fds_bytes,
// Directive FFI shape: (name, prefixes, type, body, has_body, line, column).
using PyDirective = std::tuple<std::string, std::vector<std::string>, std::string,
nb::bytes, bool, int, int>;
// TableDirective FFI shape: (type, columns, rows) where rows is a list of
// DatasetDirective FFI shape: (type, columns, rows) where rows is a list of
// lists of cells (each cell None or (kind, value); see CellToPyTuple).
using PyTableDirective = std::tuple<std::string, std::vector<std::string>,
using PyDatasetDirective = std::tuple<std::string, std::vector<std::string>,
std::vector<std::vector<nb::object>>>;
// ProtoDirective FFI shape: (shape, type_name, body) where shape is one of
// "anonymous" / "named" / "source" / "descriptor" (draft §3.4.5).
using PyProtoDirective = std::tuple<std::string, std::string, nb::bytes>;

// PXF text -> (binary proto bytes, set_paths, null_paths, directives, tables).
// PXF text -> (binary proto bytes, set_paths, null_paths, directives,
// datasets, protos).
std::tuple<nb::bytes, std::vector<std::string>, std::vector<std::string>,
std::vector<PyDirective>, std::vector<PyTableDirective>>
std::vector<PyDirective>, std::vector<PyDatasetDirective>,
std::vector<PyProtoDirective>>
PxfUnmarshalFull(nb::bytes text, nb::bytes fds_bytes,
const std::string& full_name, bool discard_unknown,
bool skip_validate) {
Expand Down Expand Up @@ -193,10 +198,10 @@ PxfUnmarshalFull(nb::bytes text, nb::bytes fds_bytes,
nb::bytes(d.body.data(), d.body.size()),
d.has_body, d.pos.line, d.pos.column);
}
// Marshal tables.
std::vector<PyTableDirective> py_tables;
py_tables.reserve(r->Tables().size());
for (const auto& t : r->Tables()) {
// Marshal datasets.
std::vector<PyDatasetDirective> py_datasets;
py_datasets.reserve(r->Datasets().size());
for (const auto& t : r->Datasets()) {
std::vector<std::vector<nb::object>> py_rows;
py_rows.reserve(t.rows.size());
for (const auto& row : t.rows) {
Expand All @@ -205,13 +210,23 @@ PxfUnmarshalFull(nb::bytes text, nb::bytes fds_bytes,
for (const auto& cell : row.cells) py_cells.push_back(CellToPyTuple(cell));
py_rows.push_back(std::move(py_cells));
}
py_tables.emplace_back(t.type, t.columns, std::move(py_rows));
py_datasets.emplace_back(t.type, t.columns, std::move(py_rows));
}
// Marshal protos.
std::vector<PyProtoDirective> py_protos;
py_protos.reserve(r->Protos().size());
for (const auto& p : r->Protos()) {
py_protos.emplace_back(
std::string(protowire::pxf::ProtoShapeName(p.shape)),
p.type_name,
nb::bytes(p.body.data(), p.body.size()));
}
return {nb::bytes(out.data(), out.size()),
r->SetFields(),
r->NullFields(),
std::move(py_dirs),
std::move(py_tables)};
std::move(py_datasets),
std::move(py_protos)};
}

// PXF schema reserved-name check (draft §3.13). Returns a list of
Expand All @@ -236,21 +251,21 @@ PxfValidateDescriptor(nb::bytes fds_bytes, const std::string& full_name) {
return out;
}

// --- PyTableReader: streaming @table consumption -------------------------
// --- PyDatasetReader: streaming @dataset consumption -------------------------
//
// Wraps protowire::pxf::TableReader. The reader takes a std::istream*; we
// Wraps protowire::pxf::DatasetReader. The reader takes a std::istream*; we
// hold the istringstream alongside the reader so its lifetime is bound to
// the Python object. Input is provided as bytes (PR-2 scope); a file-like
// streambuf bridge is a possible follow-up.
class PyTableReader {
class PyDatasetReader {
public:
static std::unique_ptr<PyTableReader> FromBytes(nb::bytes data) {
auto out = std::unique_ptr<PyTableReader>(new PyTableReader());
static std::unique_ptr<PyDatasetReader> FromBytes(nb::bytes data) {
auto out = std::unique_ptr<PyDatasetReader>(new PyDatasetReader());
out->stream_ = std::make_unique<std::istringstream>(
std::string(data.c_str(), data.size()));
auto tr = protowire::pxf::TableReader::Create(out->stream_.get());
auto tr = protowire::pxf::DatasetReader::Create(out->stream_.get());
if (!tr.ok()) {
throw nb::value_error(("pxf.TableReader: " + tr.status().ToString()).c_str());
throw nb::value_error(("pxf.DatasetReader: " + tr.status().ToString()).c_str());
}
out->reader_ = std::move(*tr);
// Marshal the side-channel directives once at construction; they're
Expand All @@ -273,10 +288,10 @@ class PyTableReader {
// Raises ValueError on parse error.
nb::object NextOrNone() {
if (reader_->Done()) return nb::none();
protowire::pxf::TableRow row;
protowire::pxf::DatasetRow row;
auto s = reader_->Next(&row);
if (!s.ok()) {
throw nb::value_error(("pxf.TableReader.next: " + s.ToString()).c_str());
throw nb::value_error(("pxf.DatasetReader.next: " + s.ToString()).c_str());
}
if (reader_->Done()) return nb::none();
return RowToList(row);
Expand All @@ -285,19 +300,19 @@ class PyTableReader {
// Iterator protocol: __next__ raises StopIteration at EOF.
nb::object Next() {
if (reader_->Done()) throw nb::stop_iteration();
protowire::pxf::TableRow row;
protowire::pxf::DatasetRow row;
auto s = reader_->Next(&row);
if (!s.ok()) {
throw nb::value_error(("pxf.TableReader.next: " + s.ToString()).c_str());
throw nb::value_error(("pxf.DatasetReader.next: " + s.ToString()).c_str());
}
if (reader_->Done()) throw nb::stop_iteration();
return RowToList(row);
}

// Drains the remaining buffered + underlying bytes. Only meaningful
// after Done(); the Python wrapper exposes this as a method that
// returns bytes so callers can chain a second TableReader on
// multi-@table documents.
// returns bytes so callers can chain a second DatasetReader on
// multi-@dataset documents.
nb::bytes Tail() {
auto t = reader_->Tail();
std::ostringstream buf;
Expand All @@ -307,15 +322,15 @@ class PyTableReader {
}

private:
static nb::object RowToList(const protowire::pxf::TableRow& row) {
static nb::object RowToList(const protowire::pxf::DatasetRow& row) {
std::vector<nb::object> cells;
cells.reserve(row.cells.size());
for (const auto& cell : row.cells) cells.push_back(CellToPyTuple(cell));
return nb::cast(cells);
}

std::unique_ptr<std::istringstream> stream_;
std::unique_ptr<protowire::pxf::TableReader> reader_;
std::unique_ptr<protowire::pxf::DatasetReader> reader_;
std::vector<PyDirective> directives_;
};

Expand Down Expand Up @@ -502,16 +517,16 @@ NB_MODULE(_protowire, m) {
m.def("pxf_marshal", &PxfMarshal, "msg_bytes"_a, "fds"_a, "full_name"_a);
m.def("pxf_validate_descriptor", &PxfValidateDescriptor, "fds"_a, "full_name"_a);

nb::class_<PyTableReader>(m, "PxfTableReader")
.def_static("from_bytes", &PyTableReader::FromBytes, "data"_a)
.def_prop_ro("type", &PyTableReader::Type)
.def_prop_ro("columns", &PyTableReader::Columns)
.def_prop_ro("directives", &PyTableReader::Directives)
.def_prop_ro("done", &PyTableReader::Done)
.def("next_or_none", &PyTableReader::NextOrNone)
.def("tail", &PyTableReader::Tail)
.def("__iter__", [](PyTableReader& self) -> PyTableReader& { return self; })
.def("__next__", &PyTableReader::Next);
nb::class_<PyDatasetReader>(m, "PxfDatasetReader")
.def_static("from_bytes", &PyDatasetReader::FromBytes, "data"_a)
.def_prop_ro("type", &PyDatasetReader::Type)
.def_prop_ro("columns", &PyDatasetReader::Columns)
.def_prop_ro("directives", &PyDatasetReader::Directives)
.def_prop_ro("done", &PyDatasetReader::Done)
.def("next_or_none", &PyDatasetReader::NextOrNone)
.def("tail", &PyDatasetReader::Tail)
.def("__iter__", [](PyDatasetReader& self) -> PyDatasetReader& { return self; })
.def("__next__", &PyDatasetReader::Next);

nb::class_<SbeCodec>(m, "SbeCodec")
.def_static("create", &SbeCodec::Create, "fds"_a, "file_names"_a)
Expand Down
2 changes: 1 addition & 1 deletion src/protowire/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@
from . import envelope, pxf, sbe

__all__ = ["pxf", "sbe", "envelope"]
__version__ = "0.75.0"
__version__ = "1.0.0"
Loading
Loading