-
Notifications
You must be signed in to change notification settings - Fork 132
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
What to do about Span.info() / Span.error()? #30
Comments
I'll note that we only have this discussion because we decided to partially implement a logging api. In java, there isn't a single answer, either. Some logging is printf format, others are different. |
@adriancole noted re the discussion itself... no argument that we're incurring a cost by discussing logging, though IMO it is a worthwhile one given the (tangible, demonstrated) benefits of span-granularity logging data at google and in my conversations about tracing with numerous other companies in the past year or so. |
@yurishkuro upon some reflection, here's a concrete proposal... the goals are to present people with platform-idiomatic and straightforward signatures for vanilla message logging, the opportunity to easily mark errors vs info (but nothing more from a severity standpoint), to allow for the logging of potentially large payloads (which may be truncated or ignored by the Tracer), and – if possible – to also support "event"-style logging, where it's still a string at the end of the day, but more of a stable string that marks a specific semantic moment... e.g., Possibility A (combine event and payload):
Possibility B (split out event and payload):
Possibility C (don't support event):
Possibility D (don't support message logging):
Possibility E (don't support message logging and combine event and payload):
Possibility F (don't support message logging or payloads):
My favorite is A. |
@bensigelman My preference would be A as well (no surprise there). It still leaves us with a task of clarifying the One other question with option A is handling exceptions in the |
I have a bias towards supporting existing ops like Trace.record, add_event(String, timestamp?) or similar meets that requirement. if "log" |
@adriancole re your suggestion of |
The most common case is searching for annotations like as "finagle.retry", Hope this helps |
|
Got it... Yeah, I'm not suggesting it makes sense; I just wanted to make Ben (mobile) On Monday, January 18, 2016, Adrian Cole notifications@github.com wrote:
Sent via mobile |
(Long thread here, so my apologies if I missed anything...and apologies for making it longer!!!) I want to share my thoughts, which I think are most similar to bhs' "Possibility A" -- but a little different in that it would introduce the notion of a The OpenTracing platform implementations are required to support a spanHandle.rawLog({
timestamp : <time_in_micros>,
type : <constant indicating INFO, ERROR, or EVENT>,
message : <a post-formatted message string || an event name>,
payload : <payload object>,
}); (or perhaps...) spanHandle.rawLog({
timestamp : <time_in_micros>,
event_name : <stable string name>,
message : <a post-formatted message string>,
error_flag : <boolean>,
payload : <payload object>,
}); Any implementation of OpenTracing must be able to gracefully handle any combination of the options, though it is allowed to ignore data it does not consider valid (e.g. lack of a timestamp or, say, both an This To address the need to make the common case convenient, the following APIs should be supported in a manner that's most idiomatic to the platform (to be explicit: this does mean per-platform variations in the signatures of the below are allowed if it promotes convenience for the platform in question).
In terms of promoting ease-of-use, the following sort of variations are allowed. This is not an exhaustive list:
As the exact signature and behavior of That's my two-cents. It was a bit on the thinking aloud side of things, so again apologies if this is missing anything fundamental! |
@bcronin I would be ok with this (maybe s/rawLog/log) @bensigelman to clarify the difference between event(str) and info(str) - the intention is that the arguments to event() are not only low-cardinality, but are also standardized with clear semantic meaning, similar to standardizing on what the tag name should be for http.url or http return code. The cs/cr annotations are not actually the primary use case for event(), as the user never needs to set these when using OT, but more for the other "standard" events known to Zipkin like wire send/recv, fragements, etc. And the expectation from the tracing system is that it might do something special based on events that it wouldn't do purely based on info logs, e.g. index/aggregate them. |
@yurishkuro makes sense re the mental model for I also like @bcronin's proposal (very much, in fact)... I am doing a bunch of non-coding work right now but later today I will try to send out a slightly more concrete (but still RFC-status) PR for the golang API and reference it on this thread. |
Per ^^^, here's a PR illustrating what this might look like in Go for starters: |
Thinking about the
|
@dankosaur thank you for the feedback. A few followups: Re bullet point 1: I'm not sure I fully understand what you're suggesting... can you express it in code/pseudocode (can be non-Go if you want, whatever works)? Re bullet points 2+3: we could replace And a note about |
@bensigelman re: question/suggestion in bullet 1, I was thinking about the following use-case, say in Python:
It's nice to be able to group on the class of an error, not just the message, which may contain unique strings that make it impossible to group. For that reason, I suggested extending the re: the |
Yes, that's what I had in mind... In your python example, I guess I would have imagined something like |
There haven't been any marked objections to the proposal outlined in opentracing/opentracing-go#38 . If we have converged, I can update the semantics doc accordingly and close this issue. |
Please don't introduce the info/error methods. That said, I think this use-case demonstrates the need for the 'payload'. (I wrote this quickly in lei of a longer comment, because of your previous latest comment @bensigelman ) |
Sorry, which "one specific use case"? This sort of logging is totally garden-variety behavior within Dapper (and was widely used because it is essential when trying to make sense of a trace). |
Fair enough. "some specific use cases" then. For the initial OpenTracing API i'd argue for the most simple plumbing api possible. Option (E) seems to be the simplest solution, and an appropriate plumbing API, that solves all the use-cases raised. |
The proposal has quite a few methods and clearly looks like a logger.
My comments about requiring a span to also be a half, or, now full logger
haven't changed. It is a lot more responsibility to take on.
No one else who is participating seems to see adding a logging interface to
the duties of all OpenTracing impls a concern. Resistance at this point is
a fools errand for me.
All that said, I am glad that it is possible to have something not muck
with existing use-case of timestamped events.
Buried in the new methods is a way to do simple string, event + timestamp.
This matches use in a lot of tracers. This is progress and I'm grateful for
that.
|
Hardly. We could start with a thinner interface that just does event+payload logging with the understanding that we will most likely (but not definitely) add logging in a not-so-distant OT API. I can live with that if there is strong support for the more conservative approach. |
Well said. It's a pragmatic and incremental step that doesn't step on anyone's toes, while still keeping it possible for all platforms to do what they need to. I'd be more than, and i'd expect everyone to be, happy to re-approach the issue with a clean slate once we've had a little more experience with the plumbing api. |
In PHP you have a popular interface for loggers: Should the PHP implementation of Open Tracing API:
|
@wladekb, if you interpose on Another approach would be to have the |
@bensigelman Unlike in Go or Python |
@wladekb sorry, I don't think I was clear... I understand that |
Yes, I thought about interface compatibility only. Making sure the logging goes to the appropriate |
@wladekb gotcha. Yeah, it's tricky... if we take the union of all logging libraries it will be nutso, but it's also nice to "blend into the surroundings" of whatever platform OT is integrating with (in this case PHP). I'd want to sleep on this but am curious to know where other people land on the topic. |
Proposal: we add info() and error() logging using platform-idiomatic conventions as a starting place. Remaining log levels are somewhat in limbo right now since we haven't resolved @wladekb's concerns about PHP, but IIUC there are no objections to info() and error() per se. I'd suggest we move forward with the above if there aren't objections within ~72 hours. |
On the 22nd January (see comments above) it was agreed that for the sake of incremental development and safety and simplicity to the initial api delivered that info/error would be left for another day. Now without any reasons stated there's a proposal to return to info/error. The majority of community effort in OpenTracing is a dog chasing its own tail. Please stop this. |
@michaelsembwever I am baffled by your tone... IIRC both you and @adriancole were uneasy about this feature and wanted to back it out in Jan. I wrote the following in response.
The reason I reopened discussion on this was because the community had come to consensus about the basic approach to propagation, which, per the above, was the reason I thought it best to defer discussion about this (less complex) piece of the OT API. In other words, the "later" in "safely added later" is now. In my understanding of well-known tracing systems, intra-span debug logging is the one feature that's still missing from OT. That's why I want to include it, especially since it's structurally simple. The most upsetting aspect of your message above is the implication that I'm going against the wishes of the community... On that note, I'd be curious to get a quick straw poll from those who've been contributing lately or registered an opinion above... @tschottdorf @yurishkuro @bg451 @adriancole @bcronin @dkuebric @slimsag: any prefs about this? If I misread the situation and there's a consensus that debug logging is either unimportant or undesirable, by all means I will close this issue. My previous understanding was that it's both important and desirable but not worth wasting time on until the juicier topics were resolved. |
I'm probably going to stay away from anything that's not vanilla Log-style logging and do all the special casing I need through a payload; maybe that's what people ought to be doing, at least for v1 - logging just has no common denominator. I could be interested in marking a trace as erroneous (as opposed to individual log messages) as a whole, but that can be done through a custom sampling priority already. My gut feeling is to stay with the simple API we have now, but I don't have a too strong opinion about it. |
@tschottdorf FYI, the logging API at head is not vanilla Log-style logging... that's what I'm proposing we add, basically. Per http://opentracing.io/spec/:
|
No. I'm implying that too many issues and conversations within the community don't land, and then re-launch off, with clearly stated reasons. The lack of compartmentalization, and clear counter arguments to previous agreed upon decisions looking to justify undoing past decisions, make understanding how the community actually got it to where it is increasingly difficult. If you had closed this issue. Then re-opened a new issue with a description on the past decision made and why it should be, or is time to be, undone. I wouldn't have reacted. Even if the the comment you made two days ago, better referenced a shift in consensus in the community and/or the rationale relative to past discussions. I wouldn't have reacted. And if this wasn't a symptom that I see generally repeating in OpenTracing. I wouldn't have reacted. |
@michaelsembwever I don't think I feel the same way about the progress of discussions in OpenTracing as you do. The topic is complex and often different use cases have overlaps, which cause discussions to go wider than we would like in a single issue. We've tried our best to capture those occasions and fork the related topics into their own issues, as evidenced by this issue here, which was a fork. I think there's been very good progress, evidenced by RC1 being available for people to implement against. The issue of logging methods has been on hold since we decided to go with a simpler initial API for RC1. I don't see anything wrong with resuming that discussion at this point. @bensigelman The way I would approach the topic of logging methods is the same as I was insisting on with all other API methods - we start with the use cases.
I want to emphasize point (2). I strongly believe the API Spec should be prescribing how end users are expected to solve specific use cases. Responses like "do it with a tag" or "special case the payload" may be perfectly acceptable forms of prescribing, but they have to be concrete, so that all end users do it in a similar manner, and all tracing systems know exactly what the end user's intention is (whether the tracers choose to implement it or not is a separate matter). I think following the above procedure would also address some of @michaelsembwever 's concerns of reopening discussions etc. Documenting use cases in the spec captures the decisions made and the reasons why certain API features exist. Lastly, I do realize that sometimes things are more complicated than that. For example, one currently unsolved use case is indicating that an error occurred during the span execution. This could be solved in isolation by using a common NB: all this rant above probably should go into its own issue, for a new section of the docs on the "process". Guilty as charged. |
Just responding to my ping on this..
I still view logging with levels as function names as a typical logging
api, or at least a slippery slope towards one. I don't think that baking
log level into the name of a log operation makes it any less a log level
than if it were a parameter, in other words.
having logEvent(String, payload) (especially wrt latter), was a concession
to avoid complete lock-up in the past. in other words, I didn't really like
the payload, and felt anxious about a future where we'd have to discuss
making this a log api again... but I was happy that we would have a chance
to get a version of opentracing (1.0) which did not have to include a
typical logging api.
So, where we are today is that we are a more progressed version of 1.0?
like 1.0: revenge of log.info? :P
since asked, yeah I still don't like info(), error() for the same reasons I
don't like log(Log.Info, message as a core api regardless of disclaimers
about dropping data. I don't like the idea of making all tracers also a log
api or a slippery slope towards one. I've written too many paragraphs to
restate again here. I don't feel I should be a blocker on this debate, as I
really have no desire to re-enter it.
|
For an example of building the right community see Apache's practices on lazy consensus and vetos in https://www.apache.org/foundation/voting.html This approach is important, because at different times different users (and possibly companies behind them) can easily stack a "vote" in their favour. The idea with lazy consensus and vetos moves the community away from quick votes and toward discussing things thoroughly through. And things are discussed through better when long discussions can be broken up into concrete steps with intermediate summaries. This minimises what's involved for newcomers to join, and better prevents the discussion chasing its tail. |
I was recently working on upgrading some instrumentation libraries and again ran into the need for some sort of an
A similar, but slightly different use case:
If we can resolve these scenarios without an From earlier discussions, I think the following can be introduced:
|
1: Of course the 2: The 3: I am ok with the guidance for instrumentation but I'd leave it at that. I don't think it's OT's business to automatically set Span errors. Deviation from error-logging per se: I was not and am not thrilled about Since the original spirit of this issue (#30) related to the above, I would like to make a concrete proposal: namely, that we create a sibling to |
Strawman: we officially don't-support If there are no objections, I'll close this in several days. |
From my pov the KV logging will provide the underlying mechanism to support all of these types of logs. As for closing this or not, it depends on whether we want to do KV logging API change with a small set of standard log tags or do the tags separately. In the latter case this issue is a good placeholder. |
Hi, best |
... continued at opentracing/specification#7. |
To fork from a different discussion in #26, the questions here are:
message
argument toinfo()
anderror()
methods to be parameterized by referring to portions of the payload, just like the logging APIs do?span.Info("oops we found %+v", payload)
in Go vs.span.info("oops we found {}", payload)
in Java?The text was updated successfully, but these errors were encountered: