Writing to a named pipe

[JohnOmernik]

IS there a way I can have Joy write to a named pipe and not have it buffer? Basically what I am seeing is that is writing, but it buffers records (this is in non-compressed form) ideally I'd like to have it write direct no buffering... is there a way to do this?

[fliphil]

I'm not sure what it is that you're asking. Can you be more specific?

[JohnOmernik]

I am noticing when I am running joy with the output set to a named pipe created with mkfifo (with no rotation etc set, and compressed output = 0) Records appear to be buffering somewhere. If I exit joy, a bunch write, or if a pile of records get written, there seems to be a flush, but it seems to be buffering. (this is monitoring the pipe with tail -f)

It's the difference between when I run a python script with output, and I don't want buffered output, I just python -u myscript.py and it just writes direct.

[JohnOmernik]

Also, I during the tail -f I will get "incomplete" records as it purges the buffer, but part of a record is in one dump of the buffer and the rest is in the next buffer. By doing unbuffered writes, it ensure only full records get written

[JohnOmernik]

Basically, I see output buffering is disabled in DEBUG mode, can we get a flag to disable output buffering but not have DEBUG mode turned on. (I can read C code, I am afraid to write it though...

/* in debug mode, turn off output buffering */

#if P2F_DEBUG
setvbuf(stderr, NULL, _IONBF, 0);
setbuf(stdout, NULL);

[JohnOmernik]

Line 432 in https://github.com/cisco/joy/blob/master/src/joy.c

[JohnOmernik]

This is where my C hacking sucks... so that's just for stdout, but I need to find the place that the outputfile for the JSON records is opened, and provide either an option to disabled buffering or just hack this into the file where the output is opened... that way it will write unbuffered... I didn't see that in joy.c, can you point me in the right direction to see where the output file is opened? Thanks!

setbuf(f, NULL);

[fliphil]

You shouldn't be afraid to write C code, in fact I recommend that you learn how to. It appears to me that you have an good idea of how to change the code to suit your specific needs.

Fixing bugs is one thing, but feature requests is entirely up to you to implement or at least partially contribute code. My team and I simply do not have the bandwidth to drop what we are doing and make changes such as this. Joy is offered as open-source to encourage people such as yourself to advance the project for the whole user base, not to be used as a service.

I look forward to seeing a pull-request from you in the future!

Philip

[fliphil]

You may want to take a look at src/p2f.c because that is where the flow_records are being written.

[JohnOmernik]

I am not not afraid to write C, I am afraid to push C to any sort of public repo... I looked at src/p2f.c and don't see where the actual output file is being opened. I am looking for that and I will just hard code (for my use) the setbuf(f, NULL) right after it's opened so I know I don't have buffering.

[bhudson33]

John, I think if you put the setbuf commands in 2 places in joy.c, it should do what you want. Look for “output = zopen(filename,”w”);” in joy.c. I believe that is where the json output file is being opened. Cheers, Bill. The ’S’ in ‘IoT’ stands for Security. On Sep 21, 2017, at 10:56 AM, John Omernik <notifications@github.com<mailto:notifications@github.com>> wrote: I am not not afraid to write C, I am afraid to push C to any sort of public repo... I looked at src/p2f.c and don't see where the actual output file is being opened. I am looking for that and I will just hard code (for my use) the setbuf(f, NULL) right after it's opened so I know I don't have buffering. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#106 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AQTCnNCBFAY6hMg2CVZ5kOdOyXmeoziVks5sknk6gaJpZM4PeeA1>.

[JohnOmernik]

Does it use zopen if I set compression = 0 in the input.h? This is where I am confused by the compile time compression...

[fliphil]

Yes it will still use zopen, but the macro definition changes. With compression = 0 it will define to fopen, and if compression = 1, it will be gzopen.

[JohnOmernik]

Ah cool

Trying now

[JohnOmernik]

So I tried putting the setbuf(output, NULL); in both places zopen was called in joy.c, and I am still getting buffered output. I would like to live tail this log, and the partial records is troublesome. I will continue to troubleshoot on my side, but if anyone else any ideas where I could disable buffering I would be very appreciative I am just guess and checking at this point.

[JohnOmernik]

I think it's working now... doing testing. (the original suggestion worked, I just um, tested with the wrong binary... *sigh