Create a space for specs for proprietary protocols

[rachelmyers]

This PR proposes that proprietary protocols and encodings can submit specs to a new proprietary_specs folder. The specs are identical to specs for other protocols or encodings, with the additional caveat that include information about how widely it is used and for what use cases.

It asserts that proprietary specs will receive less scrutiny because we will not maintain the spec. Because we are explicitly not maintaining proprietary specs, if, over time, they drift from CloudEvents, we can mark a spec as deprecated or remove it entirely.

This PR punts on important questions. For instance, if two, conflicting specs are submitted for the same protocol, CloudEvents doesn't want to arbitrate between them.

In other words, this PR puts almost all of the responsibility on the people submitting the proprietary specs.

[duglin]

@rachelmyers can you sign the commits - the DCO checker isn't happy.

[duglin]

Thanks @rachelmyers, et al ... for putting this together!

[cneijenhuis]

Thanks for getting this started!

In general, I'm very supportive of allowing people to choose the technology that works best for them. Working for an ISV, I'd like that our customers technology choices and ours can interoperate without either side having to jump through too many hoops.
In terms of messaging-systems, this is, in my experience, currently not the case. Either we have to write and maintain an integration (we currently support AWS SQS, SNS, Lambda; Azure ServiceBus, EventGrid, Functions; GCP Pub/Sub, Functions; IronMQ... 9 integrations!), or they have to.

The fear that I have is that a given proprietary product will simply stop at defining a spec, and make no further attempt to interoperate with the rest of the ecosystem, but claim they support CloudEvents. The majority of the work of making interoperability happen hasn't been done.

We could demand that, along with the spec, a working proof that their protocol can interoperate from and to a qualified protocol has to be submitted. The working proof must be independently verifiable. A simple verification is:

1. Send CloudEvent via qualified protocol, retrieve in proprietary protocol.
2. Send the output of 1. via the proprietary protocol, retrieve in qualified protocol.
3. Verify that the output of 2. is equivalent to the input of 1.

For a vendor/OSS org submitting a proprietary protocol, the preferred way IMO is that their product offers a qualified protocol out of the box. E.g. RocketMQ could implement the HTTP transport binding, and then one has a choice between their proprietary protocol and an interoperable protocol.
For a third party that doesn't have control over the product, they can submit an open-source adapter. E.g. if someone wants to make a spec for the AWS SQS "protocol", they could implement two Lambda functions: One that implements the HTTP transport binding as an input to SQS, the other that implements an HTTP webhook as the output.

In other words: There MUST be a reference implementation showing interoperability. It SHOULD be part of the main implementation of the protocol. It MAY be closed source if it is part of a proprietary product.

@rachelmyers Do you think that is a fair ask from the people submitting a proprietary spec?

[duglin]

ping @zongtanghu @duhengforever any comments?

[cathyhongzhang]

Not sure if we should require an implementation. But we may need some verifiable proof that the proprietary protocol is compliant with the CLoudEvents spec.

[duglin]

@clemensv I believe you were going to add a comment here with a proposed middle-ground position - just a reminder...

[duhenglucky]

@duglin Sorry for the delayed reply due to the Chinese New Year holiday, and thanks for @cneijenhuis @cathyhongzhang @rachelmyers comments, we will dig into this further, and will comment in the next few days.

[clemensv]

I have no objection to us hosting specifications for protocols exclusively used by products that are "proprietary".

My definition for "proprietary" here is that the protocol is exclusive to a product controlled by a single corporate entity or a single project consortium (which includes OSS project teams in Apache/Eclipse/etc).

Whether the product implementation code and process is closed or open ultimately doesn't even matter, in my view. What matters to me is that everything that exists in the CloudEvents repository fulfills an interoperability purpose.

A specification cannot fulfill an interoperability purpose, if

• [ ? ] there is no clear, complete, and implementable wire-level specification for the proprietary protocol
• [ ? ] there are no license or patent inhibitions for implementing the proprietary protocol
• [ ? ] there is no intent by any party except the owning company/consortium to implement the specification

As a strawman, let me present the [Microsoft Message Queue (MSMQ)] protocol.

MSMQ is proprietary and closed source. Based on the then effective MS/DOJ consent decree, Microsoft has fully documented the MSMQ protocol with an external auditor verifying completeness of the specification. The specification is available under the Microsoft Open Specification Promise, which grants patent rights. This checks off the first two boxes.

Nevertheless, it's hard to see why anyone would create a clean-room MSMQ server or client implementation and then would then need a CloudEvents mapping for binding CloudEvents to that clean-room MSMQ implementation to ensure interoperability. Which means that MSMQ, in spite of having extensive and thoroughly audited wire protocol specification, wouldn't check the third box for me.

A specification that doesn't fulfill the purpose of factually and verifiably enabling a party that is completely disassociated from the product/project owners to legally and independently implement the base protocol and the overlaid CloudEvents binding is in effect nothing more than a "name-drop" marketing document and doesn't belong into this repository.

That does NOT preclude having a plain list in a place like the primer document that enumerates "products using CloudEvents" with links, but those products will not be able and should not be able to claim conformance with the interoperability specifications we develop here.

Here's a rule proposal:

The proprietary_specs folder holds CloudEvents transport bindings for protocols that are exclusively used by a single product/project, are not the result of multilateral standardization and do not fulfill the criteria set by the CloudEvents group for "de facto" standards.

To ensure that such submissions fulfill a factual interoperability purpose, a CloudEvents binding for a protocol exclusively used by a single product/project MUST NOT be submitted and controlled by a party affiliated with (or compensated by) the owners of the project/product, and the binding MUST refer to complete, current, and publicly available wire-level protocol documentation and there MUST be clean-room reference implementation built on the specification and not sharing 1st party assets with the referenced product/project. For such a specification to be admissible, the protocol MUST also be free from license or patent royalty claims by the product/project owners. Should any of these conditions change for the conditions no longer be met, including through mergers or acquisitions, protocol revisions, or license changes, the specification MUST be removed from the CloudEvents repository.

[duglin]

@clemensv The first two bullet points you mentioned make sense to me. The third one makes me wonder though.... is the question really one of intent, or is it more a matter of ability? No one can control whether someone else implements a spec, the best spec authors can hope for is to be open about it and hope other people adopt their work. Take CloudEvents as an example. If no one adopts it then despite us doing all of the right things w.r.t. openness, there might not be any implementations out there - but it's not proprietary. So, to me it's about whether people can implement the spec, not whether people have done so.

However, even with that, I can't help but feel that we're veering some complex legalese with all of this. The 2nd paragraph of your proposal scares me and makes it sound like we're trying to protect ourselves from some serious maleficence - maybe we are :-) I don't know. But, if we end up with something that onerous then I tend to think that we should avoid the entire issue by just doing what you (and I think someone else mentioned on the call) and just allow for people to include a link to their spec from our repo rather than include the spec itself. Does anyone really want to revisit these specs periodically to make sure they still adhere to that 2nd paragraph?

@rachelmyers what are your thoughts on this?

[clemensv]

@duglin @rachelmyers The second paragraph is indeed defensive in the extreme and to counter "stuffing the committees" attempts by players who try to open-wash their proprietary wares. We might not need to be quite as strict, but I wanted to put a stake in the ground on that side of the spectrum.

[duglin]

All, on today's call we were not able to come to a decision so we're going do a vote. The choices are:
1 - do nothing, close this PR with no action
2 - include just links in our repo/docs to other specs
3 - include the other specs in our repo - just specs
4 - include the other specs in our repo + TCK/some kind of verification would be required for any spec

We will use a ranked voting system ( https://civs.cs.cornell.edu/ ) so please indicate your vote by listing the three choices in your preferred order - you can choose to leave an item out of the list. E.g. 1,3 will mean you want "1" as your first choice, "3" as your 2nd choice, and do not want "2" or "4" at all.

The following 21 entities are eligible to vote: Adobe, commercetools, ControlBEAM, Enterprise Holdings, Google, Huawei, IBM, Itau Unibanco, Microsoft, NATS/Synadia, Operiant, Oracle, PayPal, Red Hat, RSA, RX-M, SAP, Solace, Splunk, Vlad Ionescu, Tapani Moilanen

The vote closes at the start of next week's call - 12pm ET. Please cast your vote by adding a comment to this PR.

Note that none of these are a vote for this PR as-is. If either 2, 3 or 4 wins then this PR will be modified to align with the results of the vote. So this vote is more about setting our preferred direction rather than voting on the exact text of a proposal.

[cneijenhuis]

commercetools: 4, 2, 3

[clemensv]

Microsoft: 2, 1

[deissnerk]

SAP: 2,4,3

[cathyhongzhang]

huawei: 4,2,3

[duglin]

IBM: 2,4,3,1

[rachelmyers]

Google: 3,4,2

Our first choice is that people using and supporting CloudEvents in their own serializations and protocols be able to post that in this repo for others in the most lightweight way. Interoperability is about systems with different protocols all using a common serialization, CloudEvents, to interface with each other.

Our second choice is 4, that any proprietary spec pass a basic conformance test. This isn’t my first choice because practically, when accepting a new spec, we won’t have that protocol running, or be allowed to run the protocol in order to confirm that the conformance tests pass. The conformance test is more of a tool for developers creating a new spec rather than an actual test that we can confirm.

Our third option is that we collect a list of links to CloudEvent implementations. When all we have is a link, their spec could change or rot, and the link we have would stay the same. In practice I don’t think we would check and prune outdated specs as well as we would if they were immediately visible (and grep-able) in the repository itself.

[rperelma]

Adobe: 2, 1

[erikerikson]

Operiant: 2, 4, 3

[Vlaaaaaaad]

Vlad: 2, 4, 1, 3

[ColinSullivan1]

NATS: 2,4,3

In the interest of driving adoption/growth today, 2 seems to be lowest friction. Perhaps 4 could be added later as CE matures; verified protocols would be nice for end users.

[VictorItau]

Itaú: 2, 4, 1

[carimura]

Oracle: 2,1

[Tapppi]

Tapani Moilanen: 2,4,3

[duglin]

Vote is now closed - official results will be posted soon...

[duglin]

Raw votes:

Adobe: 2, 1
Commercetools: 4, 2, 3
Google: 3,4,2
Huawei: 4,2,3
IBM: 2,4,3,1
Itaú: 2, 4, 1
Microsoft: 2, 1
NATS: 2,4,3
Operiant: 2, 4, 3
Oracle: 2,1
SAP: 2,4,3
Tapani Moilanen: 2,4,3
Vlad: 2, 4, 1, 3

Converted into Condorcet input data:

2,1,-,- # Adobe
-,2,3,1 # commercetools
-,3,1,2 # Google
-,2,3,1 # Huawei
4,1,3,2 # IBM
3,1,-,2 # Itau Unibanco
2,1,-,- # Microsoft
-,1,3,2 # NATS/Synadia
-,1,3,2 # Operiant
2,1,-,- # Oracle
-,1,3,2 # SAP
-,1,3,2 # Tapani
3,1,4,2 # Vlad

Results

from https://civs.cs.cornell.edu/civs_create.html

2 - Just a list  (Condorcet winner: wins contests with all other choices)
- - -
4 - spec + TCK  loses to 2 - Just a list by 7–3
Tied:
  1 - CWNA  loses to 2 - Just a list by 6–0, loses to 4 - spec + TCK by 3–0
  3 - spec  loses to 2 - Just a list by 8–1, loses to 4 - spec + TCK by 8–1

Option 2 (just a list of pointers to external specs) has won.

@rachelmyers Can you please update the PR to align with that decision.

Thanks to everyone for their patience and votes.

[duglin]

hmmm link checker doesn't like this one. Perhaps just have it point to itself?

[duglin]

Aside from the href check, LGTM
thanks @rachelmyers !

[duglin]

All, I pushed 2 new commits:

• to fix the typo noticed on the call yesterday
• added the new file to the README, modified the mention of the file to be an href to it, and s/_/-/ in the file name to be consistent with the other files

We approved this PR in the call yesterday (with the typo fix) but before I merge it can I get someone to look over my edits to make sure I didn't introduce any new typos? Then I can merge it once the CI is done.

[duglin]

The CI failure of:
./README.md: Can't load url: https://github.com/cloudevents/spec/blob/master/proprietary-specs.md

is ok and will be fixed once we merge it.

[cathyhongzhang]

LGTM

[duglin]

thanks @cathyhongzhang
merging