Introducing "subject" #406

clemensv · 2019-03-19T09:27:24Z

Based on last week's debate on a missing context qualifier and the Knative project apparently now running into similar implementation issues for generic dispatch infra that we're having in Azure Event Grid, I am herewith reintroducing "subject" for consideration.

The (extensive) argument for the "subject" field is in the (closed) issue #112. I'm also talking through it in the audio track attached to this presentation https://github.com/cloudevents/spec/blob/master/share/2018-03-22-TopicsAndSubjects.pptx and the presentation itself illustrates the topic ("source") and subject split.

Signed-off-by: Clemens Vasters clemensv@microsoft.com

n3wscott

I like it. Nits over the text needs to match the other texts in the spec.

n3wscott · 2019-03-19T14:05:00Z

spec.md

+  a simple and efficient string-suffix filter for that subset of events.
+* Constraints:
+  * OPTIONAL
+  * MUST be a non-empty string


If Present, MUST be a non-empty string

n3wscott · 2019-03-19T14:07:05Z

spec.md

+  a subscriber will typically subscribe to events emitted by a `source`,
+  but the `source` identifier alone might not be sufficient as a qualifier for
+  any specific event if the `source` context has internal sub-structure.
+* Example: A subscriber might register interest for when new blobs are created


In other fields, Example is after the Constrains section.

I can shuffle that

If the event source has an internal sub-structure, then the event consumer could be interested in any part of the sub-structure. How could the producer know which part of the sub-structure the consumer is interested in, thus which part to put in the subject? Also since it is an internal sub-structure of the source, how could the consumer know how to interpret the semantics of the subject?

That is up to the pub/sub infrastructure to provide. In Event Grid we provide filters that let you run a string expression over the subject to filter that down. The publisher doesn’t know, it just publishes for the scope it was asked to deliver events for.

n3wscott · 2019-03-19T14:07:46Z

spec.md

@@ -215,6 +215,30 @@ help intermediate gateways determine how to route the events.
    * urn:event:from:myapi/resourse/123
    * mailto:cncf-wg-serverless@lists.cncf.io

+### subject
+* Type: `String`


Perhaps Subject is a URIRef? and it relates to the source uri?

It's some sort of subject identifier. I can be a filename or just some name of a thing that the source knows. I wouldn't constrain that as much as source.

URIRef does all:

this is a valid uri ref

http://so.is/this

n3wscott · 2019-03-19T14:14:18Z

spec.md

+  distinguish separate occurrences of a same-named blog having been created;
+  the name of the newly created blob is carried in `subject`.
+
+  Identifying the subject of the event in context metadata (opposed to only in


For source, we have these examples:

Examples
* Source as a URI #123
* /Source as a URI #123
* urn:event:from:myapi/resourse/123
* mailto:cncf-wg-serverless@lists.cncf.io

Would using subject turn them into:

* source: https://github.com/cloudevents/spec/ * subject: pull/123 * source: /cloudevents/spec/pull/123 * subject: pull/123 * source: urn:event:from:myapi/resourse/123 * subject: UPDATE `<-- ? picked a verb` * source: mailto:cncf-wg-serverless@lists.cncf.io * subject: Help

The last example bring the point home rather nicely. That's exactly what subject is.

JemDay · 2019-03-19T19:09:31Z

I think it might be clearer to have some richer use-case examples added to the 'example' section as opposed to embedding them inside the description of each context attribute.

Does subject now become part of the idempotency set ?

duglin · 2019-03-19T19:54:39Z

I believe we currently have "subject" as part of "source" (even if it's just implied by what's in the spec). Is this PR suggesting we pull it out because in order for generic middleware (e.g. filters) to do their job we can't count on them to understand the format of the source URI to know which parts to extract? And that the filter expression would be too hard to construct even if the entity creating the filter expression did know the format of the source URI?

Not pushing back, I just want to fully understand the reason for the splitting up of 'source' - plus this would all be good info to put into the Primer.

mrutkows · 2019-03-19T21:02:44Z

Before adding additional fields one should clearly identify the set of use cases it intends to solve to as well exhibit what the field would be used for. If we are merely separating some partial path from a URI for convenience, it does not seem like an additional field is warranted. URI (sources) that share the same paths (up to a point) indicate same source and as paths are added/they represent and distinguish sub-areas (components or subject areas) within the same "family" of sources.

evankanderson · 2019-03-20T07:19:47Z

I believe the intent of this is to make it clear what portions of the current source field SHOULD be used to identify a unique producer (see also discussion in #391 and #403 about uniqueness of id being on a per-event-producer basis).

Three issues I'm aware of with the current definition of source:

Middleware requires some sort of omniscience (knowledge of all event producers and the schema of their source attributes) in order to be able to determine the event producer given a source URL. In some cases, the source may represent a specific object, like a pull request or an individual object in an object store, but the producing system is GitHub or the object store implementation. In some cases, this may align on hostname boundaries, but in other cases, multiple APIs may be served from a single hostname, or the same API may be served from different hostnames (see Slack).
Consumers who wish to consume events from a particular producer need a way to indicate what producer they are interested in (related to point 1) Again, if the producer uses a specific source URI of a particular resource in the system (as suggested by our examples), the consumer is then required to specify "the set of sources I'm interested in" as some sort of glob/pattern match/prefix match, depending on the system. If the source URI particularly represents the event producing system, and the subject represents the individual resource described by the event. Side note: I have a slight concern about conflating subject in this context with subject in an access control context; I might prefer producer and source or service and resource.
Currently, source is allowed to be a relative URI ref, but it's not clear what that URI ref is intended to be relative to. If we separate source and subject, subject could logically be a reference relative to the source. I realize that this isn't directly in scope of this PR, but it is related. Similarly, we might benefit by suggesting that each event producer should have a (single) unique source URI which could be used by Clarify scope of eventID's uniqueness #391 and Add some additional design points for ID #403 to allow an external observer to make assumptions about the scope of uniqueness of id values.

clemensv · 2019-03-20T10:42:41Z

@duglin The "subject" is currently necessarily embedded in "source", and since that's a URI it's hard for us to give hard rules for where the context given by the "subject" sits in that URI and therefore filtering is difficult. In Event Grid, we're making a product decision to put "subject" behind the '#' in the URI and thus effectively treat it as a distinct field. In pub/sub, you subscribe to events from a source and that source itself may raise various types of events on behalf of concepts associated with it, and the "subject" identifies those concepts.

clemensv · 2019-03-20T10:44:52Z

@JemDay it's not part of the idempotency set. For that, 'source' and 'id' remain sufficient. I agree that we need more examples and in-depth explanation of how the attributes hang together.

clemensv · 2019-03-20T10:48:31Z

@mrutkows @evankanderson If you haven't done so, please read through issue #112 which lays out in lot of detail why I'm reintroducing the field and it also enumerates a lot of examples of prior art in existing messaging and eventing infrastructure.

cneijenhuis

To build on Cathys comment, I am also struggling with how the subject is supposed to look like when we have several containers, and some things can sometimes be the subject, and sometimes a container.

Event Type	Source
`com.github.Org.Created`	`/cloudevents`
`com.github.Repo.Created`	`/cloudevents/cobol-sdk`
`com.github.PR.Created`	`/cloudevents/spec/pull/123`
`com.github.PR.Review.Created`	`/cloudevents/spec/pull/123/review/456`
`com.github.PR.Review.Comment.Created`	`/cloudevents/spec/pull/123/review/456/comment/789`

How would these look like with source + subject? It isn't clear whether the last one should be /comment/789 or /pull/123/review/456/comment/789.

(This isn't a theoretical issue for me, I can also provide some commerce use cases where we have objects that also act as containers)

What I see as positive is that some (existing) middleware allows filtering on metadata attributes, but only via a prefix. Splitting the source ideally makes it easier to filter. E.g. if the DCO checker should run for all CloudEvents repos, I logically want to match /cloudevents/*/pull/*, but I can't do that with a prefix alone. If it is split into /cloudevents/spec/ and /pull/123, I can match for /cloudevents/* and /pull/*.

However it depends on the event producer splitting the source in the "correct" way for the consumer. It reminds me of the issue with the partition key - the "correct" split can be different per consumer.

The perfect solution IMO is to have advanced matching on a URI-reference, but giving existing middleware this may still be an iterative improvement.

spec.md

duglin · 2019-04-04T02:46:42Z

@clemensv can you sign your commits?

clemensv · 2019-04-04T07:21:16Z

@duglin @cneijenhuis used a novel feature which i haven't yet found on Github myself where you can contribute little patches to the patch, and accepting those signs differently:

commit 13241fb (HEAD -> patch-subject, origin/patch-subject)
Author: Christoph Neijenhuis christoph.neijenhuis@commercetools.de
Date: Wed Apr 3 18:34:19 2019 +0200

Update spec.md

Co-Authored-By: clemensv <clemens@vasters.com>

I did a soft reset and re-committed, but the DCO checker should accept these or we should rule out that feature.

cneijenhuis · 2019-04-04T09:29:27Z

Sorry for that. I like the suggested changes feature, because a) the diff view is really nice and b) in theory it should help save time to incorporate suggestions.
I'm guessing you could manually add the Signed-off-by ... to the commit message in the GitHub UI, but it makes the process more clunky.

(The problem exists with all edits through the GitHub UI unfortunately...)

duglin · 2019-04-04T14:13:35Z

rebase needed

duglin · 2019-04-04T14:15:09Z

spec.md

@@ -1,4 +1,4 @@
-# CloudEvents - Version 0.2
+# CloudEvents - Version 0.2


The updated version adds a extra whitespace to the end of 0.2.

duglin · 2019-04-04T14:27:01Z

spec.md

+  the `data` payload) is particularly helpful in generic subscription filtering
+  scenarios where middleware is unable to interpret the `data` content. In the
+  above example, the subscriber might only be interested in blobs with names
+  ending with '.jpg' or '.jpeg' and the subject attribute allows for 


s/subject/subject/

I like this paragraph. Would it be helpful to add something about how putting this info into the "source" URI is just as hard to parse/filter-on ? And that's why we creating this property. Before this property I think people were pushed towards merging it into 'source' - if they wanted it outside of 'data'. Or, if this is too much text for the spec, perhaps in the primer?

duglin · 2019-04-04T14:39:47Z

spec.md

+* Constraints:
+  * OPTIONAL
+  * MUST be a non-empty string
+* Example:


I like this example because it shows how all of the key properties relate to each other, and better explains what people should expect in each. So, can we move this example out from under the definition of this one property so it can be view in the bigger context? And then show "id" and "type" too - or even a total CE.

duglin · 2019-04-04T18:41:42Z

@clemensv can you rebase this - then I can merge it.

Approved on the 4/4 call.

Signed-off-by: Clemens Vasters <clemensv@microsoft.com>

n3wscott · 2019-04-10T18:57:53Z

spec.md

-  - /cloudevents/spec/pull/123
-  - urn:event:from:myapi/resourse/123
-  - mailto:cncf-wg-serverless@lists.cncf.io
+* Type: `URI-reference`


- is more clear than * that it is suppose to be a list. The rest of the spec uses -

n3wscott reviewed Mar 19, 2019

View reviewed changes

evankanderson mentioned this pull request Mar 20, 2019

Clarify scope of eventID's uniqueness #391

Merged

cneijenhuis mentioned this pull request Mar 29, 2019

Topics/Subject Prior Art and Example #112

Closed

cneijenhuis reviewed Apr 2, 2019

View reviewed changes

spec.md Outdated Show resolved Hide resolved

deissnerk mentioned this pull request Apr 3, 2019

Registry Design knative/eventing#1015

Merged

clemensv force-pushed the patch-subject branch from 13241fb to ef50c12 Compare April 4, 2019 07:10

duglin reviewed Apr 4, 2019

View reviewed changes

This was referenced Apr 8, 2019

Add subject to 0.3 cloudevents/sdk-go#103

Closed

Add subject. cloudevents/sdk-go#104

Merged

Clemens Vasters added 3 commits April 9, 2019 18:57

Introducing "subject"

6a0014b

Signed-off-by: Clemens Vasters <clemensv@microsoft.com>

typo fix

41c5d14

Signed-off-by: Clemens Vasters <clemensv@microsoft.com>

fixup of source/subject references

d394d69

Signed-off-by: Clemens Vasters <clemensv@microsoft.com>

clemensv force-pushed the patch-subject branch from ea71e2e to d394d69 Compare April 9, 2019 17:00

duglin merged commit ef193d5 into cloudevents:master Apr 10, 2019

n3wscott reviewed Apr 10, 2019

View reviewed changes

jroper mentioned this pull request Jun 12, 2020

WIP - Eventing support cloudstateio/cloudstate#353

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introducing "subject" #406

Introducing "subject" #406

clemensv commented Mar 19, 2019

n3wscott left a comment

n3wscott Mar 19, 2019

clemensv Mar 20, 2019

n3wscott Apr 5, 2019

n3wscott Mar 19, 2019

clemensv Mar 20, 2019

cathyhongzhang Mar 28, 2019

clemensv Apr 3, 2019

n3wscott Mar 19, 2019

clemensv Mar 20, 2019

n3wscott Apr 5, 2019

n3wscott Mar 19, 2019

clemensv Mar 20, 2019

JemDay commented Mar 19, 2019

duglin commented Mar 19, 2019

mrutkows commented Mar 19, 2019

evankanderson commented Mar 20, 2019

clemensv commented Mar 20, 2019

clemensv commented Mar 20, 2019

clemensv commented Mar 20, 2019

cneijenhuis left a comment

duglin commented Apr 4, 2019

clemensv commented Apr 4, 2019

cneijenhuis commented Apr 4, 2019

duglin commented Apr 4, 2019

duglin Apr 4, 2019

n3wscott Apr 5, 2019

duglin Apr 4, 2019

duglin Apr 4, 2019

duglin Apr 4, 2019

duglin commented Apr 4, 2019

n3wscott Apr 10, 2019

		@@ -1,4 +1,4 @@
		# CloudEvents - Version 0.2
		# CloudEvents - Version 0.2

Introducing "subject" #406

Introducing "subject" #406

Conversation

clemensv commented Mar 19, 2019

n3wscott left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JemDay commented Mar 19, 2019

duglin commented Mar 19, 2019

mrutkows commented Mar 19, 2019

evankanderson commented Mar 20, 2019

clemensv commented Mar 20, 2019

clemensv commented Mar 20, 2019

clemensv commented Mar 20, 2019

cneijenhuis left a comment

Choose a reason for hiding this comment

duglin commented Apr 4, 2019

clemensv commented Apr 4, 2019

cneijenhuis commented Apr 4, 2019

duglin commented Apr 4, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duglin commented Apr 4, 2019

Choose a reason for hiding this comment