Define Log Data model #97
Conversation
tigrannajaryan
requested review from
arminru,
bogdandrutu,
c24t,
carlosalberto,
iredelmeier,
jmacd,
reyang,
SergeyKanzhelev,
tedsuo and
yurishkuro
as code owners
tigrannajaryan
force-pushed
the
feature/tigran/log-data-model
branch
2 times, most recently
from
269a3da
to
e3be95e
Compare
|
@open-telemetry/specs-approvers please review. The initial wave of discussion happened in the Google Doc. I believe it is time to move to refinements and eventual approval. |
tigrannajaryan
force-pushed
the
feature/tigran/log-data-model
branch
from
e3be95e
to
a1c863d
Compare
tigrannajaryan
force-pushed
the
feature/tigran/log-data-model
branch
from
a07b657
to
66c2d9e
Compare
tigrannajaryan
force-pushed
the
feature/tigran/log-data-model
branch
from
66c2d9e
to
3720df0
Compare
|
@open-telemetry/specs-approvers please review. |
|
thank you for the hard work on this @tigrannajaryan and others. I understand the wish to keep up the momentum and move forward on this. One point that recently came up was whether we need to consider how the data model can be made to encompass not just logs, but also tracing events and metrics or at least how it will interoperate with these. cc'ing @tedsuo because I believe we discussed it in a recent call. |
|
one of the requirements for this model is that is can be unambiguously mapped to from common models. It doesn't look like we've been through that exercise for the Elastic Common Schema (ECS) and it is not present here. I'm happy to take a swing at it, if that sounds appropriate. |
@roncohen Yes, definitely, please do. Happy to add it to this OTEP or reference it from this OTEP if you post it elsewhere. |
These are just examples to help clarify the meaning of the fields for people who are familiar with one or another format. This document does not claim or attempt to standardize any vendor-specific formats. Ultimately vendors are free to decide how (and if) they want to map their formats to this model. |
I'm not sure why they belong in the spec if that's the case. If we want to include example uses such as these vendor specifics then they belong in a different file/location versus in the appendix. It seems like free advertising for commercial solutions which doesn't belong in this forum. If those are to be included then we should open that up to all vendor solutions who want to be included in the specification. |
I'll leave up to OpenTelemetry admins to decide if having this is appropriate in an OTEP. I believe examples help clarify the OTEP and should stay. |
No problem, I'll submit a PR for a logz.io example section to be added in fairness if that's how we're going to proceed. Personally, I do not believe this is the right forum for vendor names to be included. |
tigrannajaryan
force-pushed
the
feature/tigran/log-data-model
branch
from
ca19785
to
75bd614
Compare
This is a proposal of a data model and semantic conventions that allow to represent logs from various sources: application log files, machine generated events, system logs, etc. Existing log formats can be unambiguously mapped to this data model. Reverse mapping from this data model is also possible to the extent that the target log format has equivalent capabilities. The purpose of the data model is to have a common understanding of what a log record is, what data needs to be recorded, transferred, stored and interpreted by a logging system.
tigrannajaryan
force-pushed
the
feature/tigran/log-data-model
branch
from
75bd614
to
33bcdf0
Compare
tigrannajaryan
force-pushed
the
feature/tigran/log-data-model
branch
from
33bcdf0
to
9b721ee
Compare
Fixes #597 ## Changes - Add a section for "generic attributes" to the log semconv - Add an attribute `log_record.id` making use of ULID as discussed in #597 Some additional notes: - I kept the PR small, so I left out some other potential attributes, e.g. something for pre-existing ID (like windows event logs) or for storing the used logging/eventing system or even something like a "signature" that might be worth discussing, etc. - I followed the structure of "generic attributes" from the spans semconv - I took some of the existing wording from #597 & open-telemetry/oteps#97 (comment) to describe the field --------- Signed-off-by: svrnm <neumanns@cisco.com> Co-authored-by: Joao Grassi <joao@joaograssi.com> Co-authored-by: jack-berg <34418638+jack-berg@users.noreply.github.com> Co-authored-by: Tigran Najaryan <4194920+tigrannajaryan@users.noreply.github.com>
Fixes #597 ## Changes - Add a section for "generic attributes" to the log semconv - Add an attribute `log_record.id` making use of ULID as discussed in #597 Some additional notes: - I kept the PR small, so I left out some other potential attributes, e.g. something for pre-existing ID (like windows event logs) or for storing the used logging/eventing system or even something like a "signature" that might be worth discussing, etc. - I followed the structure of "generic attributes" from the spans semconv - I took some of the existing wording from #597 & open-telemetry/oteps#97 (comment) to describe the field --------- Signed-off-by: svrnm <neumanns@cisco.com> Co-authored-by: Joao Grassi <joao@joaograssi.com> Co-authored-by: jack-berg <34418638+jack-berg@users.noreply.github.com> Co-authored-by: Tigran Najaryan <4194920+tigrannajaryan@users.noreply.github.com>
Fixes #597 ## Changes - Add a section for "generic attributes" to the log semconv - Add an attribute `log_record.id` making use of ULID as discussed in #597 Some additional notes: - I kept the PR small, so I left out some other potential attributes, e.g. something for pre-existing ID (like windows event logs) or for storing the used logging/eventing system or even something like a "signature" that might be worth discussing, etc. - I followed the structure of "generic attributes" from the spans semconv - I took some of the existing wording from #597 & open-telemetry/oteps#97 (comment) to describe the field --------- Signed-off-by: svrnm <neumanns@cisco.com> Co-authored-by: Joao Grassi <joao@joaograssi.com> Co-authored-by: jack-berg <34418638+jack-berg@users.noreply.github.com> Co-authored-by: Tigran Najaryan <4194920+tigrannajaryan@users.noreply.github.com>
This is a proposal of a data model and semantic conventions that allow to
represent logs from various sources: application log files, machine
generated events, system logs, etc.
Existing log formats can be unambiguously mapped to this data model.
Reverse mapping from this data model is also possible to the extent that
the target log format has equivalent capabilities.
The purpose of the data model is to have a common understanding of what a
log record is, what data needs to be recorded, transferred, stored and
interpreted by a logging system.