Skip to content

v0.8.0

Choose a tag to compare

@sandreae sandreae released this 09 Nov 16:48
· 23 commits to main since this release

What's new?

Spaces API: Encrypted Groups

There's no denying it...this is a big release! Our main focus over the past few months has been integrating encrypted spaces and group management into our high-level API. This release brings that work into the light of main. This has arguably been some of the most technically challenging work we've ever tackled and we're excited that it's now available for use!

The spaces API behaves a lot like our existing stream and ephemeral stream APIs; you create a space and begin publishing and receiving messages. What's new is that messages published to a space are encrypted towards a managed set of identities. Key bundles, key rotation, encryption and decryption all happen as part of the p2panda event-processing pipeline. Messages published into a space are automatically encrypted using the latest key material and group membership state, ensuring that only active members are able to decrypt those messages on the other end. As with the stream API, our new spaces API also supports replays of past events and messages; it's possible to stream all events for a space, only unacknowledged events, or all events from a custom cursor (starting point). Take a look at our group_chat example to see the spaces API in action.

🚧 A word of caution: this release contains known security vulnerabilites in spaces; we are working on fixes but they are not included in this release. While we've done our best to thoroughly test the code we're introducing here, it is still somewhat experimental and we do expect bugs to emerge. We hope to rely on your support to find and fix any bugs as they appear.

Hackable Sync API

Inspired by our collaboration with the Dash Chat team, we've introduced a new api module in p2panda-sync which includes a collection of useful methods for hacking on custom sync protocols. adz has written in-depth documentation for that module which gives a huge amount of detail on data sync (replication) in peer-to-peer contexts. We've also included three new code examples which use the sync helper methods: unicast_w_live_mode.rs - which is very similar to the approach we currently run over iroh's QUIC streams in p2panda, sneakernet.rs - e.g. sync over USB stick, and broadcast.rs.

A few more of the highlights:

  • Sync session authoriser (manage blocklists) in p2panda-net
  • Database improvements related to transactions and connection handling
  • Test-related improvements to reduce CI run-time, fix flaky behaviour and reduce friction around p2panda development
  • Upstream iroh updates which should improve performance of network connections and ephemeral streams

Added

  • node: Add custom_cursor_name optional param to Node::stream_from 1408
  • net: SyncHook intercepting sync sessions to accept or reject them #1451
  • net: Introduce an Authoriser for maintaining and enforcing allow- and blocklists #1321
  • net: Use network id to separate mDNS discovery #1511
  • node: API for promoting and demoting space members #1311
  • node: Allow graceful closure of sync sessions #1307
  • node: Associate key bundle, groups and spaces logs with log topic #1264
  • node: Automatically renew expired key-bundles and publish member messages more efficiently #1337
  • node: Cancel processed stream tasks on drop of stream handles #1300
  • node: Emit per-group and global groups events #1313
  • node: Encrypted spaces and groups management integration into high-level API #1202
  • node: Forward events resulting from local actions to app layer #1290
  • node: Handle spaces events when processing pipeline event #1276
  • node: Implement Orderer Processor for pipeline #1262
  • node: Integrate the Authoriser into the Node#1321
  • node: Membership change validation in Group API #1336
  • node: Membership change validation in Space API #1292
  • node: Task for repairing spaces #1277
  • node: Configurable egress to "publish" and "import" locally forged operations #1431
  • node: Support optionally adding spaces events when constructing processor Event #1430
  • spaces: Compute and return events from local methods #1290
  • spaces: Introduce local stream import #1296
  • spaces: Safe key bundle registration by cross-signing X3DH identity- and verifying-keys #1332
  • spaces: Support partial merge of auth state into a space #1299
  • spaces: Validate write authority when processing application messages #1295
  • store: Add resolve_topics method to TopicStore trait #1261
  • store: Introduce ProcessorStore for storing event metadata from p2panda pipeline #1262
  • store: Add method on TopicStore to get all topics - #1388
  • store: clear & clear_keys methods for OrdererStore - #1466
  • stream: Filter out spaces application messages from concurrently removed members #1291
  • stream: Hooks processor to register event callbacks in streaming pipeline #1339
  • stream: Move orderer processor from node to stream #1312
  • stream: Out-of-order buffer for validating & inserting operations #1402
  • stream: Add validate_operation method checking against store #1417
  • stream: Namespaced items to allow multiple orderer states #1460
  • node: ConnectionAuthoriserHook updating authoriser based on space membership #1359
  • node: E2EE CLI chat example using spaces API #1288
  • sync: api module with useful methods to hack your own sync protocols #1393
  • sync: Method for streaming log ranges from the store #1393
  • sync: Method to resolve log heights from topic #1405
  • sync: Re-export -stream ingest_operation in api module #1406
  • stream: Forward events of already processed operations in spaces processor #1430
  • auth: Add method for checking if actor id is a group 1483
  • node: Add groups processor to pipeline #1485
  • node: Groups hook for associating group logs with topic #1487
  • node: Debouncer w/ throttle for repair task #1499
  • node: Return PendingSpace to account for situations where it doesn't exist yet #1529
  • node: Await created, joined group events with PendingGroup #1530
  • node: Await registration of member key bundles with PendingMember #1531
  • node: Efficient encoding of authenticated member info (for QR codes, etc.) #1547
  • node: Move member log association to hook and dispatch key bundle operations for live-mode #1550
  • node: Add high-level spaces docs and introduce method-specific errors #1527

Changed

  • core: Use Signer trait instead of SigningKey #1202
  • node: Increase stream buffer size #1360
  • node: Make Group sendsync #1306
  • node: Move repair task out of stream #1347
  • node: Use random network ids in tests #1360
  • node: Use unique network ids in API tests #1425
  • spaces: Only emit membership change events if local user is space member #1304
  • spaces: Remove auth resolver generic parameter #1298
  • store: sqlx 0.9.0, use safer query builder #1322
  • core: Infallible conversion from Operation into AnyOperation #1389
  • store: Use AnyOperation on LogStore #1391
  • sync: Update examples to use new api module #1413
  • net: Separate sync- from connection authoriser #1437
  • net: Update iroh v1.0.3 -> v1.3.0 #1465
  • spaces: Return output structs from public APIs #1453
  • node: Remove redundant authoriser updates #1455
  • spaces: Make SpacesMessageStore::get_message return F::Message type #1481
  • spaces: Return all message deps from Space::repair and Space::create #1483
  • auth: Don't error when processing operation again #1483
  • stream: Namespace group state by stream id in group processor #1485
  • node: Introduce GroupSubscription and StreamEvent::Group #1489
  • store: Namespace event metadata entries #1521
  • node: Repair task re-write with hook trigger #1500
  • node: Integrate new repair task into Node, Space and Group #1498
  • net: Update iroh to v1.4.1 and iroh-gossip to v0.102.0 #1565
  • node: Remove space_id arg from Node::create_space #1574

Fixed

  • core: Make cbor_decode lenient #1429
  • node: Allow event processing to handle out-of-order buffering by separating i/o streams and preserve input ordering #1271
  • node: Unblock task tracker by introducing "pass-through" events coming from orderer #1267
  • node: Manage separate orderer state per topic stream #1461
  • node: Orderer state needs to be reset before replay #1467
  • node: Exit leaking pipeline thread on drop #1524
  • node: Fix deadlock caused by nested transactions when clearing orderer state during replay #1516
  • node: Fix lost cursor updates when acking across duplicate topic streams #1545
  • node: Fix acking space control messages #1546
  • spaces: Deterministic deserialization of SpacesArgs #1264
  • spaces: Fix calculation of historically removed members #1455
  • spaces: Fix bug instantiating space from existing groups state #1457
  • spaces: Atomic read -> modify -> write state handling for groups and spaces #1513
  • spaces: Separate key rotation and retrieval around key bundle APIs, making it more robust #1547
  • store: Use BEGIN IMMEDIATE in database transaction to allow awaiting busy write locks across processes #1502
  • net: Do not block sync actor awaiting send_after on retry #1506
  • store: Fix "write" methods not using transactions, consistent naming with _tx suffix #1515
  • node: Fix flaky spaces tests #1539
  • node: Trigger repair before any local group or space action #1541
  • encryption: Fix RUSTSEC-2026-0387 by updating hpke-rs v0.8.0 #1557
  • node: Associate group logs with topic in ingest processor #1553
  • node: Rename chat example to prevent filename collision #1560
  • node: Fix handling multiple application bodies in post pipeline logic #1567
  • store: Fix LogStore::log_entries long-lived database connection blocking pipeline #1554
  • net: Fix flaky e2e_log_sync test #1566
  • node: Add members and actors fields to StreamEvent::Group #1561

Contributors

Thank you to @matthiasbeyer, @guillemcordoba, @maackle, @jadehopepunk for code contributions, testing, feedback and care that has helped us to improve p2panda! Special thanks to the Reflection and Dash Chat teams who helped shape the new APIs with invaluable feedback over the last year 🫢