DEPRECATION NOTICE:
rest-feeds is deprecated and will not be further developed. Use http-feeds instead.
Asynchronous data replication and event streaming with plain REST/HTTP.
A REST feed provides access to resources in a chronological sequence of changes.
GET /movies
Accept: application/json
200 OK
Content-Type: application/cloudevents-batch+json
[{
"specversion" : "1.0",
"type" : "org.example.movie",
"source" : "/movies",
"id": "11b592ae-490f-4c07-a174-04db33c2df70",
"time": "2019-12-16T08:41:519Z",
"subject": "18",
"data": {
"original_title":"The Fifth Element",
"popularity":26.163
}
},{
"specversion" : "1.0",
"type" : "org.example.movie",
"source" : "/movies",
"id": "64e11a7a-0e40-426c-8d81-259d6f6ab74e",
"time": "2019-12-16T09:12:421Z",
"subject": "12",
"data": {
"original_title":"Finding Nemo",
"popularity":23.675
}
}]
The number of items returned by the server is limited, e. g. to 1000 items per request.
Repeat the request with the id that has been processed as lastEventId query parameter.
Note that this is an example of an Aggregate Feed where resources can be updated and deleted:
GET /movies?lastEventId=64e11a7a-0e40-426c-8d81-259d6f6ab74e
Accept: application/json
200 OK
Content-Type: application/cloudevents-batch+json
[{
"specversion" : "1.0",
"type" : "org.example.movie",
"source" : "/movies",
"id": "756e21c9-4ebd-4354-8f7d-85cd7d2bc4ec",
"time": "2019-12-17T11:09:122Z",
"subject": "18",
"data": {
"original_title":"The Fifth Element",
"popularity":27.011
}
},{
"specversion" : "1.0",
"type" : "org.example.movie",
"source" : "/movies",
"id": "e510d24e-bf06-4f6a-b6db-5744f6ff2591",
"subject": "12",
"time": "2019-12-18T17:00:786Z",
"method": "DELETE"
}]
The server implements long polling. When there are no newer items, the server holds the connection open until new data or a timeout arrives.
Example for no new data after 5 seconds:
GET /movies?lastEventId=e510d24e-bf06-4f6a-b6db-5744f6ff2591
Accept: application/json
200 OK
Content-Type: application/cloudevents-batch+json
[]
The client continues polling this link until new items are received.
REST feeds provide access to resources in a chronological sequence of changes using plain HTTP(S).
A REST feed complies to these principles:
- HTTP(S) as transfer protocol
- Clients poll the feed endpoint for new items
- Batched results with
lastEventIdto scroll through further items - Content-Negotiation, with
application/cloudevents-batch+jsonas default - Feed items are immutable.
- Items are always appended.
REST feeds enable asynchronously decoupled systems without shared infrastructure.
REST feeds can be used for data replication (aggregate feeds) and event streaming (event feeds).
Aggregate feeds are used to share aggregates (aka master data, domain objects) with other systems for data replication and to trigger actions.
Every update to the resource leads to a new entry of the full current state in the feed.
An aggregate feeds must contain every resource (identified through its subject) at least once.
Event feeds are used to publish domain events that have happened in a domain.
Event feeds often contain different entry types.
Events may be deleted once they are outdated or useless.
The feed endpoint must return items in a strictly ascending order of addition to the feed.
The feed endpoint must support fetching items without any query parameters, starting with the first items.
The feed endpoint may choose to limit the number of items returned in a response to a subset of the whole set available.
The feed endpoint must must support the lastEventId query parameter.
It must only returns items after this event id.
The feed endpoint must implement long polling. If the server has no newer items, the server holds the request open until new items arrive and then immediately sends the response.
The server should send an empty response if no new item arrived after N seconds (5 seconds recommended).
The initially stored link is the feed endpoint URL without query parameters.
Pseudocode:
link = "https://feed.example.com/movies"
while true:
try:
response = GET link
for item in response:
process item
link = link + setQueryParameter "lastEventId" to value item.id
except:
wait N secondsThe client must persist the id of the last processed item.
The client's item handling must be idempotent (at-least-once delivery semantic).
The id may be used for idempotency checks.
The client must implement an exception handler that delays the next request, to protect the server in case of connection or processing errors.
The response contains an array of items. The response must comply with the CloudEvents Specification.
| Field | Type | Mandatory | Description |
|---|---|---|---|
specversion |
String | Mandatory | The currently supported CloudEvents specification version. |
id |
String | Mandatory | A unique value (such as a UUID) for this item. It can be used to implement deduplication/idempotency handling in downstream systems. |
type |
String | Mandatory | The type of the item. May be used to specify and deserialize the payload. A feed may contain different item types, especially if it is an event feed. It SHOULD be prefixed with a reverse-DNS name. |
subject |
String | Optional | Key to identify the aggregate, the item refers to. It doesn't have to be unique within the feed. This should include a business key such as an order number. |
method |
String | Optional | The HTTP equivalent method type that the feed item performs on the subject. PUT indicates that the resource was created or updated. DELETE indicates that the subject was deleted. Defaults to PUT. |
time |
String | Mandatory | The item addition timestamp. ISO 8601 UTC date and time format. |
data |
Object | Optional | The payload of the item in JSON. May be missing, e.g. when the method was DELETE. |
Further metadata may be added, e.g. for traceability.
Every consumer and provider must support the media type application/cloudevents-batch+json as defined by the CloudEvents JSON Batch Format.
It is the default and used when the Accept header is missing, defined as plain application/json or not supported.
Further media types may be used, when supported by both, client and server:
text/htmlto render feed in browserapplication/atom+xmlto support feed readerstext/event-streamfor server-sent eventsapplication/x-ndjsonto support streamingmultipart/*to have a more HTTP native representationapplication/x-protobufto minimize traffic- any other
Feed endpoints may be protected with HTTP authentication.
The most common authentication schemes are Basic and Bearer.
The server may filter feed items based on the principal. When filtering is applied, caching may be unfeasible.
When feed items include the full current state of the resource, older feed items for the same resource may be obsolete.
Items may be deleted from the feed when another item was added to the feed with the same resource URI.
It is good practice to keep the feed small to enable a quick synchronization of new clients.
The server must handle next links, when the requested item has been deleted by returning the next higher items.
When a resource was deleted, the server must append a DELETE item with the resource URI to delete.
Clients must delete this resource or otherwise handle the removal.
{
"specversion" : "1.0",
"type" : "org.example.movie",
"source" : "/movies",
"id": "8a22af5e-d9ea-4e7f-a907-b5e8687800fd",
"time": "2019-12-17T07:07:777Z",
"subject": "777777",
"method": "DELETE"
}
The server should start a compaction run afterwards to delete previous items for the same resource.
Feed endpoints may set appropriate response headers, such as Cache-Control: public, max-age=31536000, when a page is full and will not be modified anymore.
- restfeed-server-java
- restfeed-server-spring
- restfeed-server-spring-example
- restfeed-client-java
- restfeed-client-spring
2021-11-29: Comply with CloudEvents
- Data Model: Add field
specversion - Data Model: Remove field
next. Specified query parameterlastEventIdinstead. - Data Model: Rename
timestamptotime - Data Model: Rename
resourcetosubject - Data Model:
typeshould be reverse-DNS name. - Media Type: Respond with
application/cloudevents-batch+json