[receiver/kafka] Allow Kafka Receiver to accept Azure Resource Log format #18210
Comments
cparkins
added
enhancement
atoulme
added
receiver/kafka
receiver/azureeventhub
and removed
needs triage
|
@MovieStoreGuy we can make our codecs shared across kafka and azureeventhub so users can reuse the logic. Azure event hub looks very similar to kafka, to the point it can be exposed as a kafka endpoint apparently. |
|
This is the formatter @cparkins references fwiw: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/azureeventhubreceiver/azure_formatter_impl.go |
|
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
Pinging code owners for receiver/kafka: @pavolloffay @MovieStoreGuy. See Adding Labels via Comments if you do not have permissions to add labels yourself. |
I had a proposal some time ago about having a an encoding package that provides a factory that can be passed around since other components like file receiver | exporter, kinesis exporter, kafka receiver | exporter, and even potentially the HTTP client could adopt. I don't have an issue for it in principle, since we don't have this shared encoding package and we keep the kafka code generic enough that we can add other formats. |
|
I'm about to pickup expanding out the EventHub receiver to allow Diagnostic (audit etc.) logs and metrics to be taken by the eventhub receiver. This was likely to be quite hacky in the eventhub receiver. I was originally going to expand the kafka receiver to use the right authentication, and add the formats into that, but my feeling is that the eventhubs might be the better place? My feeling is that with the eventhub receiver, we can detect what format the the event is and choose the right data type and transformation as it's more specific to event hubs? I need this for a client, however, I'm more than happy for it to be work directly against the contrib repo. I'm no go-programmer, so it will likely need more of a thorough review. |
|
Nothing is hacky. Everything is a work in progress. @cparkins just landed metric support in azureeventhubreceiver, please take a look and feel free to open issues and PRs to make things better. |
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
Currently working to rebase, test and publish a PR for this. |
|
@cparkins can you please summarize why you chose the kafka receiver instead of extending Azure Event Bus receiver? It's not clear from the issue. Without any details, it seems like Azure Event Bus is a better place. |
|
@dmitryax This feature already existed for Azure Event Hubs. This will allow the traffic to be routed to Kafka as well, if desired. For testing you can use Event Hubs to mimic Kafka and test that the traffic can be received by a Kafka instance. In the future I may use this feature to read data from Kafka instead of Event Hubs. For now it just makes the Kafka Receiver able to receive this data in case anyone is routing traffic to Kafka in Azure Resource Logs format. |
Do you mean https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/azureeventhubreceiver? If that's the case, I don't understand why do we need this functionality in the kafka receiver... Can you please elaborate more on your use case? |
|
@dmitryax My use case was streaming Azure Resource Logs to Kafka before the Event Hubs Receiver existed. I'm currently not using this feature but I think I may in the future. This will not limit the technology choices that I would make in the future for ingesting data from Azure. I can see use cases where I would Stream data in this format from Azure into Kafka. |
|
It's probably worth noting that the Azure Event Hubs is Kafka compatible, therefore not maintaining 2 separate receivers would probably be better in the long run. |
|
@martinjt That's how I was testing. Although lower SKU Event Hub instances are not Kafka compatible. |
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
…e formats (#25096) **Description:** <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> Adding a Translator to convert from Azure Resource Log format (for now) to OTel. **Link to tracking Issue:** <Issue number if applicable> #18210 **Testing:** <Describe what testing was performed and which tests were added.> Performed manual validation testing between Event Hubs and OTel and added unit tests to cover new use case to allow multiple resources to be included in a batch. Existing tests moved to new package. **Documentation:** <Describe the documentation added.> None was necessary as this format is already supported by the Event Hub Receiver.
|
Can we remove the stale label as I have an updated PR that is ready to be reviewed using the new Azure Translator Package pushed earlier this week. |
…e formats (open-telemetry#25096) **Description:** <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> Adding a Translator to convert from Azure Resource Log format (for now) to OTel. **Link to tracking Issue:** <Issue number if applicable> open-telemetry#18210 **Testing:** <Describe what testing was performed and which tests were added.> Performed manual validation testing between Event Hubs and OTel and added unit tests to cover new use case to allow multiple resources to be included in a batch. Existing tests moved to new package. **Documentation:** <Describe the documentation added.> None was necessary as this format is already supported by the Event Hub Receiver.
…e formats (open-telemetry#25096) **Description:** <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> Adding a Translator to convert from Azure Resource Log format (for now) to OTel. **Link to tracking Issue:** <Issue number if applicable> open-telemetry#18210 **Testing:** <Describe what testing was performed and which tests were added.> Performed manual validation testing between Event Hubs and OTel and added unit tests to cover new use case to allow multiple resources to be included in a batch. Existing tests moved to new package. **Documentation:** <Describe the documentation added.> None was necessary as this format is already supported by the Event Hub Receiver.
…e formats (open-telemetry#25096) **Description:** <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> Adding a Translator to convert from Azure Resource Log format (for now) to OTel. **Link to tracking Issue:** <Issue number if applicable> open-telemetry#18210 **Testing:** <Describe what testing was performed and which tests were added.> Performed manual validation testing between Event Hubs and OTel and added unit tests to cover new use case to allow multiple resources to be included in a batch. Existing tests moved to new package. **Documentation:** <Describe the documentation added.> None was necessary as this format is already supported by the Event Hub Receiver.
**Description:** <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> Add support for the Azure Resource Log format so that Diagnostic logs from Azure can be consumed by a Kafka Receiver. **Link to tracking Issue:** <Issue number if applicable> #18210 **Testing:** <Describe what testing was performed and which tests were added.> Local testing to ensure that data is being converted correctly when Streaming from Azure Diagnostic Settings to Event Hubs and using the Kafka Receiver. **Documentation:** <Describe the documentation added.> Added a bullet for the added azure format.
**Description:** <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> Add support for the Azure Resource Log format so that Diagnostic logs from Azure can be consumed by a Kafka Receiver. **Link to tracking Issue:** <Issue number if applicable> open-telemetry#18210 **Testing:** <Describe what testing was performed and which tests were added.> Local testing to ensure that data is being converted correctly when Streaming from Azure Diagnostic Settings to Event Hubs and using the Kafka Receiver. **Documentation:** <Describe the documentation added.> Added a bullet for the added azure format.
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
This issue has been closed as inactive because it has been stale for 120 days with no activity. |
Component(s)
receiver/kafka
Is your feature request related to a problem? Please describe.
When trying to ingest Infrastructure logs and metrics from Azure Resources through Diagnostic Settings using the Kafka Receiver the format used by Azure is not in an accepted format.
Describe the solution you'd like
Allow the Kafka Receiver to ingest data from Azure Resource Logs and convert it into OpenTelemetry format.
Describe alternatives you've considered
Using the Azure Event Hub Receiver or Azure Blob Storage Receiver.
Additional context
No response
The text was updated successfully, but these errors were encountered: