Approaches to tracking implementation interop and deployment

[tfpauly]

The IETF working group chairs list has had a recent thread around hosting implementations in a working group. Doing that in something like a static archive might be problematic, but having a way for groups and protocol developers to track what’s built can be of great value for protocol authors, chairs, and implementers to gauge status.

Some groups are already doing this, so gathering that experience is a good first step.

• How are the working groups you’re involved with already tracking implementation interop and deployments? (Include links where possible)
• How should implementation status feed back into protocol development? How about after an RFC ships?
• What are the challenges you’ve seen with tracking implementations?

[cabo]

It is usually not the working group, but individuals in the WG doing the tracking. This allows these trackers to be opinionated.

One challenge can be the number of implementations and the difficulty of eliciting commensurable information about them (worse so when new criteria evolve).

[mirjak]

I would say the main challenge might be for people who are not deeply involved in the wg to actually find where implementations are tracked: sometimes it's part of a draft but removed on publication as RFC, some groups have it in the wiki, some on GitHub. Having one clear way to find implementation and understand their status (active, production) also for people who are not active IETF Pparticipants but rather IETF "consumers". Given that not everybody knows the Datatracker we would probably also need something on the ietf.org page.

[aretana]

The idr WG has a formal requirement of two implementations before advancing anything -- in fact, they were the main reason the "Waiting for Implementation" sub-state was put into the datatracker. The WG used to publish implementation report RFCs (for example, rfc4276: "BGP-4 Implementation Report"); one problem is that the content becomes out of date quickly. The WG now uses its wiki (https://trac.ietf.org/trac/idr/wiki/Protocol%20implementations%20Reports) to document the implementations needed for advancement. An important advantage is that the information can be kept up to date (new implementations can be added, for example); the downside is that no one takes the time for it.

Other WGs in RTG have implementation requirements, but they are usually not documented: there's a simple question on the list about known implementations...maybe included (and removed) in the draft...

[tfpauly]

A Wiki certainly seems preferable to trying to capture implementation status in a draft or RFC itself. Having a link between these would be nice, though!

[cabo]

On 2020-08-31, at 23:52, Tommy Pauly ***@***.***> wrote: A Wiki certainly seems preferable to trying to capture implementation status in a draft or RFC itself.

Something between a wiki page and an RFC is a markdown file on a github repository. That allows people to submit PRs instead of trampling all over the existing page, which (needing to trample) makes it hard for a conscientious individual to BE BOLD [1] and in the end may lead to fewer updates even though the threshold may nominally be lower. (Of course, actually *using* a Wikipedia page also is a viable way to run this.)

Having a link between these would be nice, though!

The elephant in this room is what is the process about updating the page. Seriously, where are the appeals going to be sent to? Grüße, Carsten [1]: https://en.wikipedia.org/wiki/Wikipedia:Be_bold

[tfpauly]

Yes, a markdown in a repo is often my preferred tool these days, but we could imagine options with similar properties (pull requests / addition requests).

For either datatracker or a GitHub page, chairs could manage and moderate the content. But that brings up the question of what occurs after a group closes—an experts group?

[cabo]

On 2020-09-01, at 00:54, Tommy Pauly ***@***.***> wrote: For either datatracker or a GitHub page, chairs could manage and moderate the content.

Are they signing up for that? Could make finding chairs more difficult.

But that brings up the question of what occurs after a group closes—an experts group?

Indeed. And there we are again at the general question of how we manage live documents over time, which is coming up in multiple venues right now. Grüße, Carsten

[tfpauly]

@cabo Yes, the very fact that this is coming up in so many conversations is why we formed this program to discuss.

[cabo]

Finding this information is a classical IR problem. You want both recall (finding all resources) and precision (finding only the ones that actually help). The first one could be solved by defining a search keyword in the RFC (which some protocols implicitly do a bit already by using a somewhat unique protocol name). The second requires ongoing curation, for which we'd need to run a process.

[cabo]

(and all this not only relates to whole protocols but also to specific features. Searchable feature names...)

[tfpauly]

Relevant to this discussion:

https://github.com/eckelcu/draft-eckel-edm-find-code
https://www.ietf.org/id/draft-eckel-edm-find-code-00.html