-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New component json log exporter #10836
Comments
Should this be part of a "file exporter" that allows to export any signal in proto/json or any "marshaler" that we support? @djaglowski do we have such thing? |
Not to my knowledge. There would be a lot of overlap between exporting signals as json to a file and exporting signals as json to an endpoint. Seems like they'd share code but probably still should be separate exporters. |
@djaglowski Thanks for replying. |
See #7840 as well. |
Is this some specific JSON format that is different from OTLP/JSON? If it is OTLP/JSON then I think the best place for it is |
The format we use for logz.io single log entry is essentially a Our general approach is to separate Take this JSON representation of {
"resourceLogs": [
{
"resource": {
"attributes": [
{
"key": "resource-attr",
"value": {
"stringValue": "resource-attr-val-1"
}
}
]
},
"instrumentationLibraryLogs": [
{
"instrumentationLibrary": {},
"logs": [
{
"timeUnixNano": "1581452773000000789",
"severityNumber": "SEVERITY_NUMBER_INFO",
"severityText": "Info",
"name": "logA",
"body": {
"stringValue": "This is a log message"
},
"attributes": [
{
"key": "app",
"value": {
"stringValue": "server"
}
},
{
"key": "instance_num",
"value": {
"intValue": "1"
}
}
],
"droppedAttributesCount": 1,
"traceId": "08040201000000000000000000000000",
"spanId": "0102040800000000"
}
]
}
]
}
]
} Will be converted to this format: {
"@timestamp": "1581452773000000789",
"traceId": "08040201000000000000000000000000",
"spanId": "0102040800000000",
"level": "Info",
"resource-attr": "resource-attr-val-1",
"app": "server",
"instance_num": "1",
"message": "This is a log message"
} If the |
This conversion loses data (e.g. instrumentationLibrary/Scope data is missing) and has ambiguities (what happens if I have a LogRecord attribute named "resource-attr" or named "spanId"?). I do not see the justification why this format needs to be chosen as a canonical format endorsed by Otel Collector. IMO, if we want to have a exporter that is given a generic name "jsonexporter" then it needs to be:
For now I don't see see 1-4 fulfilled. My preference to move forward with this initiative would be to:
|
Sure that makes sense. |
+1, in particular it's critical that using the json exporter retains full data fidelity and is lossless.
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping |
This issue has been closed as inactive because it has been stale for 120 days with no activity. |
The purpose and use-cases of the new component
This exporter lets you send log records as JSON bytes to a generic HTTP(S) endpoint.
Will be similar to fluentbit and fluentd HTTP outputs.
An example use case can be forwarding logs to log management backends via logstash (example1, example2)
Our team at logz.io already implemented such an exporter in our vendor-specific component
logzioexporter
(ref from my fork) to support logging with open telemetry collector. We thought others in the community could find it useful so we decided to contribute the log exporter as a non-specific vendor component if the community will approve it.Example configuration for the component
Plus exporterhelper options
Telemetry data types supported
Logs
The text was updated successfully, but these errors were encountered: