Test report?

[astearns]

The implementation report link in the spec leads to README.md here, and the the 'report' link in README.md leads back to the spec. Is the implementation report merely a list of reference implementations? Is there a report showing test results and/or interoperability between the reference implementations?

[danielkhan]

The reference implementation section only shows open-source implementations as proof of work. There are no reports on interop or test results. This would in my opinion also go beyond the scope if a standard.

What's your use-case that requires such a report?

[astearns]

I do see a bit on test results, where it's asserted that the simple regex implementation has “100% compliance to the test suite.” That's great, and I'm looking for evidence that the other implementations are also compliant. It's definitely in the scope of a standard, as that is usually how standards demonstrate they are correctly and interoperably implementable.

The relevant part of the W3C process is in https://w3c.github.io/w3process/#adequate-implementation . This goes over some of the requirements for taking a specification to Recommendation status. The first two bullets there are:

• is each feature of the current specification implemented, and how is this demonstrated?
• are there independent interoperable implementations of the current specification?

I'm assuming that the OpenTelemetry/OpenCensus/Elastic/LightStep implementations are independent, so I think you're covered on that point. What I'm looking for is a demonstration that each feature in the specification is implemented and in an interoperable way. In other working groups this often takes the form of a test report showing each test passing in more than one independent implementation.

[plehegar]

So, the transition request pointed to the list of implementations in the README, as we ll as the notes from last year meeting:
[[
Sergey: OpenTelemetry, several Microsoft / Azure services (azure functions, API gateway), ASP.Net

Philippe: Spec will be final mid January.

If we want to start a V2 specification - we can start publishing it whenever we want. Example may be introducing the response headers.

Morgan: Google: x-google-cloud-trace header will be replaced with traceparent, Work on introducing headers in Envoy and Istio. With small config change it is easy to switch to this header. Cloud services is a bigger priority.

Dynatrace, Christoph - OneAgent supports this header, demand wasn’t huge, also waiting for cloud services to catch up. Customers needs to be educated.

Morgan: still a lot of confusion with customers. Explaining all the cases is hard.

Justin + Vic: NewRelic - two languages prepared to be released - starting to have conversation with customers. Victor: needs to be careful with backwards compatibility support. About to release beta.

Christoph: either use old or use both. No option currently for using just trace context headers

Matthew: Lightstep is switching to OpenTelemetry. For the time being switching LS tracers to w3c. Lightstep is a primary, second is Legacy b3, not a single header B3.

Anssi - x-amzn header is being used. Blockers: want specification to be final. Implementation on streams and queues is a question, as there’s a concern about large header sizes impacting performance on extremely small requests.

AppDynamics - not much to share. Customers are confused, especially with modern stacks. No committment yet.

Yuri: maintainer of Jaeger, tech lead of tracing @ Uber. There is a Java PR in Jaeger to support Trace Context, it’s not finished… Sunsetting Jaeger client for OpenTelemetry and with the switch everything will just work.
]]

There are tests but keep in mind that those heades are not exposed to the Open Web, only within the clouds. Thus not possible to run those tests from outside. And the bulks of the specs is to define the grammar/syntax of the headers. The semantic cannot be checked by program.

[astearns]

I'm reading the above as more detail about the implementations, but I'm still not seeing anything that demonstrates that each feature in the spec is included in more than one implementation and works interoperably. The assertion that the simple regex implementation passes 100% of the tests is a good start. Another assertion that implementation X or Y passes 100% of the tests would be sufficient. As someone on the outside looking in, I can't tell whether any of the implementations are complete and/or agree with each other.

[danielkhan]

Right now, these implementations are vendor to vendor and not assertable from the outside. Vendors have to ensure that they can produce and consume standard-compliant data and this is nothing that can be asserted well on a per implementation basis. We agree that interoperability tests are desirable and we will consider them as soon as the spec also contains response headers. Closing for now.