Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
222c613
Initial implementation
Meldiron Jul 29, 2026
20cf3cb
Switch fetch library to client
Meldiron Jul 29, 2026
daf09e9
Improve compatibility with CloudEvent library
Meldiron Jul 29, 2026
4d3148d
Replace event class with CloudEvent library
Meldiron Jul 29, 2026
da108fc
Rename generic adapters to journals
Meldiron Jul 29, 2026
2e1f2b7
Improve docs about each component
Meldiron Jul 29, 2026
9a5c130
Simplify readme
Meldiron Jul 29, 2026
a4cf9da
Upgrade dependencies
Meldiron Jul 29, 2026
2e95611
Fix CI base image and address review feedback
Jul 29, 2026
74f4c34
Stop a cursor position from moving backwards
Jul 29, 2026
9390ca1
Document the cursor ordering guarantee and its limit
Jul 29, 2026
60e12db
Make Redis cursor advances atomic with a one-entry stream
Jul 29, 2026
c929c91
Fix E2E assertions for the stream-backed cursor
Jul 29, 2026
e0e1999
Simplify implementation
Meldiron Jul 29, 2026
976d396
Further library simplification
Meldiron Jul 29, 2026
1e3f88b
Manual simplification
Meldiron Jul 29, 2026
3a1529f
More manual simplification
Meldiron Jul 29, 2026
583f4db
Update Protocol.php
Meldiron Jul 29, 2026
4fd2656
Introduce Appendable away from Feed
Meldiron Jul 29, 2026
f9501c5
Fix linter
Meldiron Jul 29, 2026
a1c5b2b
Update docs behaviour
Meldiron Jul 29, 2026
052b72e
Upgrade to latest cloudevent lib
Meldiron Jul 30, 2026
31289d8
Make the wire format comply with the http-feeds.org spec
Meldiron Jul 31, 2026
461a92a
Return a Batch from reads and fold Protocol out of user-facing code
Meldiron Jul 31, 2026
d90e3ab
Let a consumer start at the tip of a feed
Meldiron Jul 31, 2026
b89ff18
Add Consumer::seek() to set a consumer's position explicitly
Meldiron Jul 31, 2026
488ec87
Make the journal poll interval configurable
Meldiron Jul 31, 2026
3f122e8
Give each side of the wire its own vocabulary
Meldiron Jul 31, 2026
f23031e
Rewrite the README — simple, straight to the point, side-first
Meldiron Jul 31, 2026
703aaa3
Terminology rework
Meldiron Jul 31, 2026
5f7c0be
Manual simplification
Meldiron Jul 31, 2026
1c7afbb
Simplify classes
Meldiron Jul 31, 2026
2f6470e
Fix failing tests
Meldiron Jul 31, 2026
c3cab97
Add max site, fix concurrency
Meldiron Aug 3, 2026
efbbdf1
Document expected behaviour
Meldiron Aug 3, 2026
77b2fd0
Test rework preparations
Meldiron Aug 3, 2026
2e0e9e2
Rework tests
Meldiron Aug 3, 2026
58cf7ac
Depend on a released utopia-php/cloudevents
Meldiron Aug 3, 2026
5ae89e4
Fail loudly when the cursor cache cannot be written
Meldiron Aug 3, 2026
f2ab5c6
Keep datacontenttype across a store round trip
Meldiron Aug 3, 2026
2ba9acd
Wrap cache backend failures as Transport
Meldiron Aug 3, 2026
a6e244a
Make ext-redis a suggestion rather than a requirement
Meldiron Aug 3, 2026
d7c2fcb
Escape names when building backend keys
Meldiron Aug 3, 2026
f9fd9fc
Pin the pooled store's borrow-per-read property
Meldiron Aug 3, 2026
36bd329
Stop the cache store loading the feed on every poll tick
Meldiron Aug 3, 2026
db8e293
Run CI on main, and pin the linter's PHP version
Meldiron Aug 3, 2026
d71568c
Let seek() take the ids a third-party feed actually uses
Meldiron Aug 3, 2026
2a91682
Reject a lastEventId that is not a string in serve()
Meldiron Aug 3, 2026
3721424
Document what publish() stamps over
Meldiron Aug 3, 2026
9f8e93c
Declare the batch media type once, on Readable
Meldiron Aug 3, 2026
131e23c
Move the test suite from PHPUnit 9.6 to 13
Meldiron Aug 3, 2026
56f4c33
Test the server's timeout and limit clamps
Meldiron Aug 3, 2026
103e87b
Test the Redis adapters' Transport contract
Meldiron Aug 3, 2026
c124325
Measure and report code coverage
Meldiron Aug 3, 2026
82b07a2
Assert datacontenttype on both sides of the wire
Meldiron Aug 3, 2026
f7933cd
Set timing bounds from the regression, not the runtime
Meldiron Aug 3, 2026
5490feb
Make both decode paths filter extensions the same way
Meldiron Aug 3, 2026
0aafe6d
Move the exact-trim retention tests into the shared suite
Meldiron Aug 3, 2026
4cb4a75
Route on the feed name in the HTTP suite's endpoint
Meldiron Aug 3, 2026
0dab9bb
Restore consumer coverage over a real Redis stream
Meldiron Aug 3, 2026
cfa9496
Restore consumer coverage over a cache-fed feed
Meldiron Aug 3, 2026
0988afa
Assert full event fidelity across the HTTP wire
Meldiron Aug 3, 2026
0c80b03
Reinstate a cursor contract suite per adapter
Meldiron Aug 3, 2026
20f0d7c
Rewrite the changelog as one initial release
Meldiron Aug 3, 2026
df19717
Cut the comments down across the branch
Meldiron Aug 3, 2026
8ab8fa4
Improve redis issue behaviour
Meldiron Aug 3, 2026
5a2f228
add fallback to xRevRange
Meldiron Aug 3, 2026
33c00e9
Fix stability
Meldiron Aug 3, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions .github/ISSUE_TEMPLATE/bug.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
name: "🐛 Bug Report"
description: "Submit a bug report to help us improve"
title: "🐛 Bug Report: "
labels: [bug]
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to fill out our bug report form 🙏
- type: textarea
id: steps-to-reproduce
validations:
required: true
attributes:
label: "👟 Reproduction steps"
description: "How do you trigger this bug? Please walk us through it step by step."
placeholder: "When I ..."
- type: textarea
id: expected-behavior
validations:
required: true
attributes:
label: "👍 Expected behavior"
description: "What did you think would happen?"
placeholder: "It should ..."
- type: textarea
id: actual-behavior
validations:
required: true
attributes:
label: "👎 Actual Behavior"
description: "What did actually happen? Add screenshots, if applicable."
placeholder: "It actually ..."
- type: dropdown
id: utopia-version
attributes:
label: "🎲 Utopia Abuse version"
description: "What version of Utopia Abuse are you running?"
options:
- Version 0.6.x
- Version 0.5.x
- Version 0.4.x
- Version 0.3.x
- Version 0.2.x
- Version 0.1.x
- Different version (specify in environment)
validations:
required: true
- type: dropdown
id: operating-system
attributes:
label: "💻 Operating system"
description: "What OS is your server / device running on?"
options:
- Linux
- MacOS
- Windows
- Something else
validations:
required: true
- type: textarea
id: enviromnemt
validations:
required: false
attributes:
label: "🧱 Your Environment"
description: "Is your environment customized in any way?"
placeholder: "I use Cloudflare for ..."
- type: checkboxes
id: no-duplicate-issues
attributes:
label: "👀 Have you spent some time to check if this issue has been raised before?"
description: "Have you Googled for a similar issue or checked our older issues for a similar bug?"
options:
- label: "I checked and didn't find similar issue"
required: true
- type: checkboxes
id: read-code-of-conduct
attributes:
label: "🏢 Have you read the Code of Conduct?"
options:
- label: "I have read the [Code of Conduct](https://github.com/appwrite/appwrite/blob/HEAD/CODE_OF_CONDUCT.md)"
required: true
32 changes: 32 additions & 0 deletions .github/ISSUE_TEMPLATE/documentation.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
name: "📚 Documentation"
description: "Report an issue related to documentation"
title: "📚 Documentation: "
labels: [documentation]
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to make our documentation better 🙏
- type: textarea
id: issue-description
validations:
required: true
attributes:
label: "💭 Description"
description: "A clear and concise description of what the issue is."
placeholder: "Documentation should not ..."
- type: checkboxes
id: no-duplicate-issues
attributes:
label: "👀 Have you spent some time to check if this issue has been raised before?"
description: "Have you Googled for a similar issue or checked our older issues for a similar bug?"
options:
- label: "I checked and didn't find similar issue"
required: true
- type: checkboxes
id: read-code-of-conduct
attributes:
label: "🏢 Have you read the Code of Conduct?"
options:
- label: "I have read the [Code of Conduct](https://github.com/appwrite/appwrite/blob/HEAD/CODE_OF_CONDUCT.md)"
required: true
40 changes: 40 additions & 0 deletions .github/ISSUE_TEMPLATE/feature.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
name: 🚀 Feature
description: "Submit a proposal for a new feature"
title: "🚀 Feature: "
labels: [feature]
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to fill out our feature request form 🙏
- type: textarea
id: feature-description
validations:
required: true
attributes:
label: "🔖 Feature description"
description: "A clear and concise description of what the feature is."
placeholder: "You should add ..."
- type: textarea
id: pitch
validations:
required: true
attributes:
label: "🎤 Pitch"
description: "Please explain why this feature should be implemented and how it would be used. Add examples, if applicable."
placeholder: "In my use-case, ..."
- type: checkboxes
id: no-duplicate-issues
attributes:
label: "👀 Have you spent some time to check if this issue has been raised before?"
description: "Have you Googled for a similar issue or checked our older issues for a similar bug?"
options:
- label: "I checked and didn't find similar issue"
required: true
- type: checkboxes
id: read-code-of-conduct
attributes:
label: "🏢 Have you read the Code of Conduct?"
options:
- label: "I have read the [Code of Conduct](https://github.com/appwrite/appwrite/blob/HEAD/CODE_OF_CONDUCT.md)"
required: true
25 changes: 25 additions & 0 deletions .github/workflows/analysis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: "Static analysis"

on:
pull_request:
push:
branches: [main]

jobs:
check:
name: PHPStan
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0

# Run against the test image rather than a bare composer container: the
# Redis stream calls only resolve where ext-redis is actually installed.
- name: Build
run: |
docker compose build tests
docker compose up -d tests

- name: Run PHPStan
run: docker compose exec -T tests composer check
25 changes: 25 additions & 0 deletions .github/workflows/linter.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: "Linter"

on:
pull_request:
push:
branches: [main]

jobs:
lint:
name: Linter
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0

# The project's own image, as the analysis job uses: pint needs a PHP
# new enough to parse 8.5 syntax, which `composer:latest` does not pin.
- name: Build
run: |
docker compose build tests
docker compose up -d tests

- name: Run Linter
run: docker compose exec -T tests composer lint
54 changes: 54 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
name: "Tests"

on:
pull_request:
push:
branches: [main]

jobs:
tests:
name: "PHP ${{ matrix.php-versions }} · ${{ matrix.suite }}"
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
# Add a PHP version here and nowhere else — the image is built from one
# parameterized Dockerfile.
php-versions: ['8.5']
# One job per adapter: each runs the same behaviour scenarios
# (tests/Feed/*/Base.php), so a green suite means that adapter honours
# the whole contract. See phpunit.xml for what each suite contains.
suite: [unit, memory, cache, http, redis, pool]

env:
PHP_VERSION: ${{ matrix.php-versions }}

steps:
- name: Checkout repository
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0

- name: Build
run: |
docker compose build
docker compose up -d

- name: Run ${{ matrix.suite }} tests
run: docker compose exec -T tests vendor/bin/phpunit --testsuite ${{ matrix.suite }}

coverage:
name: Coverage
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0

- name: Build
run: |
docker compose build
docker compose up -d

# Every suite, so the Redis and pooled adapters count too. No threshold
# yet — the number is there to be read.
- name: Report coverage
run: docker compose exec -T tests composer coverage:all
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
/vendor/
/.idea/
coverage.xml
.phpunit.cache/
.phpunit.result.cache
.DS_Store
107 changes: 107 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# Changelog

## 0.1.0 — Initial release

Pull-based HTTP event feeds ([http-feeds.org](https://www.http-feeds.org/)) for
PHP. A producer writes events to an ordered log; each consumer polls for what
happened after the last event it handled, and keeps that position itself.

Requires PHP 8.5. `ext-redis` is a suggestion, needed only by the Redis and
pooled adapters.

### Producing

- `Producer` — `produce(type, data, subject)` for the common case, and
`publish(CloudEvent)` for a prepared event. Both return the event's id, which
is also its position in the feed. The producer stamps its own `source`, the
store assigns the `id`, and a missing `time` becomes now; everything else is
published as prepared.
- Events are [utopia-php/cloudevents](https://github.com/utopia-php/cloudevents)
`CloudEvent` objects. This library defines no event type of its own, so a feed
event is accepted anywhere a `CloudEvent` is, and `datacontenttype`,
`dataschema` and extension attributes survive an append and a read.

### Serving

- `Server` — the feed endpoint. `serve(array $query)` is the whole HTTP request
in one call: it reads `lastEventId`, `limit` and `timeout` from the raw query
parameters, coerces and clamps them, and rejects a `lastEventId` that is
neither a position nor the tip sentinel with `Exception\Invalid`. `read()`,
`poll()` and `tip()` are there for a route that wants the pieces.
- `Batch` — one read of a feed. Counts and iterates as its events, and carries
the limit it was actually built with, so `cacheControl()` cannot be handed a
number the read did not use: a full batch is settled history and cacheable
forever, anything shorter is `no-store`. `toArray()` is the wire encoding and
`lastId()` the position a stateless relay tracks.

### Consuming

- `Consumer` — the pull loop, at-least-once, with a durable position.
`consume(callable)` handles a batch and commits after the last event that
succeeded; a failing handler blocks the events behind it by design.
- Built over a local `Readable` for a feed the same service produces, or
straight over a [utopia-php/client](https://github.com/utopia-php/client)
whose base URI points at the feed endpoint, plus the feed's name.
- `position()`, `reset()` and `seek(eventId)` move the position by hand.
`seek()` is the escape hatch for a poison event: seek to the failing event's
own id to step past it. Both are safe to call from inside a handler.
- `Consumer::START_TIP` starts a consumer with no stored position at the tip of
the feed instead of at the oldest retained event. It rides a protocol
extension — the `lastEventId` value `$`, which the producer resolves to its
newest event as the request arrives — so skipping the backlog costs no extra
round trip and still delivers what lands mid-poll.

### Stores and cursors

- `Store\Redis`, `Store\Pool` — a capped Redis stream, directly or over a
[pooled](https://github.com/utopia-php/pools) connection. The pooled store
borrows per read, so a held long poll never ties up a connection.
- `Store\Cache` — the feed on a [Utopia cache](https://github.com/utopia-php/cache),
for a service that already carries one. The whole feed lives under one key,
so an append rewrites it (last-writer-wins — run one producing process) and
retention is also the cost of producing an event, which is why its default
`maxSize` is 1 000 rather than the 100 000 the Redis store keeps. The newest
id is kept under a second, tiny key so a caught-up long poll does not load
the feed to learn nothing.
- `Store\Memory` — tests and single-process development. `Store\None` — no
backend configured; throws on use, so a misconfigured service fails loudly
instead of dropping events.
- `Cursor\Cache`, `Cursor\Redis`, `Cursor\Pool`, `Cursor\Memory`, `Cursor\None`
— where a consumer's position lives, keyed by feed and consumer name. The
stored form is deliberately plain (`feed:<feed>:cursor:<consumer>` holding
the id as a string), so positions carry across upgrades and an operator can
answer "where is this consumer?" from a shell. A run's save is conditional
on the position it started from (read-compare-write), so an instance that
fell behind cannot undo newer progress, a seek, or a reset — it adopts the
newer position instead.
- Retention (`maxSize`) and the long-poll read interval (`pollInterval`, in
milliseconds, default 500) are constructor options on every store.

### The wire

- A batch is the plain JSON array of CloudEvents the spec defines — no
envelope. An empty array means the consumer is caught up. The media type is
`Readable::MEDIA_TYPE`, `application/cloudevents-batch+json`, aliased as
`Batch::MEDIA_TYPE` for the serving side and `Remote::MEDIA_TYPE` for the
consuming one.
- `Readable` is the contract both sides share — `read`, `poll`, `tip`,
`getName`, plus `TIP`, `MAX_BATCH` (1000) and `MAX_TIMEOUT` (30s).
`Appendable` is the contract of a store that owns its events;
`Remote` deliberately does not implement it.
- `Remote` — another service's feed over HTTP, a `Readable` in its own right
rather than a store. Decoding is strict about `id`, because that is the
consumer's position, and tolerant of everything else, so a consumer older
than the producer keeps working: unknown attributes ride along as
extensions, and an entry that cannot be read ends the batch early rather
than discarding the usable events before it — unless it is the first, where
there is no progress to keep and the read fails loudly instead.
- `Id` — feed positions (`{ms}-{seq}`) and the arithmetic for paging past one.
`Key` shapes the backend keys, escaping names so a feed and a cursor cannot
collide in one keyspace.

### Errors

Everything this library raises extends `Utopia\Feed\Exception`:
`Exception\Invalid` for something the caller handed over, `Exception\Transport`
for a backend or network failure, `Exception\Unsupported` for something a
backend cannot do.
Loading
Loading