New component: Named Pipe Receiver #27234
Comments
sinkingpoint
added
needs triage
|
Hey, @sinkingpoint Thank you for the new component proposal. If you have not already please make sure you review the new component guidelines. If you have not found a volunteer sponsor yet then I encourage you to come to our weekly collector sig meetings. You can add an item to the agenda to discuss this new component proposal. |
|
We discussed during the APAC SIG - idea looks good, no objections. @djaglowski it seems this would build best on stanza, do you see any trouble with this approach? Would you be interested to sponsor perhaps? |
|
I think it makes a lot of sense as a stanza input operator, which is then wrapped into a receiver. I'm willing to sponsor this. @sinkingpoint, are you good with first writing the stanza input operator only? From there it should be fairly straightforward to create the receiver. |
|
Thanks @djaglowski , appreciate it. Happy to write the stanza operator - I'll work on it next week |
|
Looking around the code, would it make sense to follow the same pattern as the file receiver and further abstract the logic into a |
|
I definitely don't think we should add anything to I think it probably makes sense to first try a full implementation within Off the top of my head, I think the biggest area of duplication is going to be in splitting, trimming, flushing, etc. However, all of this functionality is already separated into distinct packages which can be composed as needed. I think there may be a little bit of copying the way these are configured and the order of composition from |
crobert-1
added
Accepted Component
**Description:** This is a namedpipe input operator, which will read from a named pipe and send the data to the pipeline. It pretty closely mimics the file input operator, but with a few differences. In particular, named pipes have an interesting property that they receive EOFs when a writer closes the pipe, but that _doesn't_ mean that the pipe is closed. To solve this issue, we crib from existing `tail -f` implementations and use an inotify watcher to detect whenever the pipe receives new data, and then read it using the standard `bufio.Scanner` reader. **Link to tracking Issue:** #27234 **Testing:** We add a couple of tests for the new operator. The first tests simply the creation of the named pipe - checking that it's created as a pipe, with the right permissions. The second goes further by inserting logs over several different `Open`s into the pipe, testing that the logs are read, and that the operator is able to handle the named pipe behavior of skipping over EOFs. **Documentation:** None, at the moment /cc @djaglowski --------- Signed-off-by: sinkingpoint <colin@quirl.co.nz>
**Description:** This is a namedpipe input operator, which will read from a named pipe and send the data to the pipeline. It pretty closely mimics the file input operator, but with a few differences. In particular, named pipes have an interesting property that they receive EOFs when a writer closes the pipe, but that _doesn't_ mean that the pipe is closed. To solve this issue, we crib from existing `tail -f` implementations and use an inotify watcher to detect whenever the pipe receives new data, and then read it using the standard `bufio.Scanner` reader. **Link to tracking Issue:** open-telemetry#27234 **Testing:** We add a couple of tests for the new operator. The first tests simply the creation of the named pipe - checking that it's created as a pipe, with the right permissions. The second goes further by inserting logs over several different `Open`s into the pipe, testing that the logs are read, and that the operator is able to handle the named pipe behavior of skipping over EOFs. **Documentation:** None, at the moment /cc @djaglowski --------- Signed-off-by: sinkingpoint <colin@quirl.co.nz>
**Description:** This adds a new receiver that reads from a named pipe, using the previously created namedpipeinput stanza operator (#28841). Because that operator does the majority of the work, this is mostly boilerplate + a few tests to confirm that it works e2e. Link to tracking Issue: #27234 **Testing:** Added a few tests, loading the config, and actually creating a receiver to verify that it can read logs from the pipe. **Documentation:** None, yet. /cc @djaglowski - where should I add docs for this if any?
**Description:** This adds a new receiver that reads from a named pipe, using the previously created namedpipeinput stanza operator (open-telemetry#28841). Because that operator does the majority of the work, this is mostly boilerplate + a few tests to confirm that it works e2e. Link to tracking Issue: open-telemetry#27234 **Testing:** Added a few tests, loading the config, and actually creating a receiver to verify that it can read logs from the pipe. **Documentation:** None, yet. /cc @djaglowski - where should I add docs for this if any?
**Description:** This adds a new receiver that reads from a named pipe, using the previously created namedpipeinput stanza operator (open-telemetry#28841). Because that operator does the majority of the work, this is mostly boilerplate + a few tests to confirm that it works e2e. Link to tracking Issue: open-telemetry#27234 **Testing:** Added a few tests, loading the config, and actually creating a receiver to verify that it can read logs from the pipe. **Documentation:** None, yet. /cc @djaglowski - where should I add docs for this if any?
**Description:** This adds a new receiver that reads from a named pipe, using the previously created namedpipeinput stanza operator (open-telemetry#28841). Because that operator does the majority of the work, this is mostly boilerplate + a few tests to confirm that it works e2e. Link to tracking Issue: open-telemetry#27234 **Testing:** Added a few tests, loading the config, and actually creating a receiver to verify that it can read logs from the pipe. **Documentation:** None, yet. /cc @djaglowski - where should I add docs for this if any?
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping |
|
Closed by #29320 |
The purpose and use-cases of the new component
It's sometimes useful to be able to sends telemetry to a collector running on a node over a named pipe, rather than having to deal with the overhead of network protocols, or the inherent disk buffering of writing to a file. I'd like to propose a receiver that has the ability to create a named pipe, and then read logs off of it, similar to https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/tcplogreceiver or https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filereceiver
Example configuration for the component
I suspect the complete config would end up looking something like the filereceiver one, with multiline config etc.
Telemetry data types supported
Logs
Is this a vendor-specific component?
Code Owner(s)
@sinkingpoint
Sponsor (optional)
@djaglowski
Additional context
No response
The text was updated successfully, but these errors were encountered: