New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
stdin support? #3
Comments
This is a good idea, I think I can add support for reading from a file But by removing the filesystem, we can potentially improve performance by Thanks! I am not sure how soon I could implement this though.
|
true! Yeah in practice I'd say the reader won't be a bottleneck for most cases, also there shouldn't be a lot of memory required for buffering while reading (?) anyways no problem on the timeline, thanks for the quick reply! |
I'm thinking about this some more. How does varnish emit its logs that you'd want mtail to read from stdin? Would you want to open a named pipe or unix socket, or open a file descriptor? If you can give me more information about how you think it would work that would be very helpful. |
so in this particular case varnishlog (https://www.varnish-cache.org/docs/3.0/reference/varnishlog.html) would read log entries from shared memory ring buffer and output those on stdout, e.g.
Also there would be cross-line correlation to do for some things, so it might be more complicated than I initially thought. Anyways, do you still see value in accepting stdin? thanks! |
The mtail language supports temporary state, iirc, so you could write a Right now I am thinking of generalising inputs as taking an arbitrary open
|
BTW thanks the link to the varnish docs.
|
I've added a flag -logfds that you can pass numerical file descriptors to, e.g.: mtail -progs examples -logfds 0 which will load the programs in the examples directory, and open stdin for tailing logs. It's a bit rough at the moment, so more feedback appreciated -- open new bugs please! |
👍 I'll give it a try, thanks @jaqx0r ! |
Can you show me how you run mtail such that it does work?
As this was a year ago, I don't remember why I decided it didn't work. So,
if you can show me how to use it, then I will put the feature back in.
…On Fri, 12 Jul 2019 at 00:56, John Bond ***@***.***> wrote:
@jaqx0r <https://github.com/jaqx0r> thanks for your work and this tool. I
notice that this feature was removed about a year ago[1] which I have just
noticed after upgrading. Do you have more information on why this 'can't
ever work'. As it appears to be working for us.
[1]8cd80a9
<8cd80a9>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3?email_source=notifications&email_token=AAXFX66CUIHFTZY4B3CO5KLP65CX7A5CNFSM4BDSYXV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZW65YQ#issuecomment-510521058>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAXFX6YBCXTTZXTLZF2FL33P65CX7ANCNFSM4BDSYXVQ>
.
|
Thanks for getting back and also it seems my colleague filippog (who originally opened this issue) also contacted you. I think its best to continue the discussion here and save you answering both of us. To answer your question we run the program with the following /usr/bin/varnishncsa -n frontend -F "${FMT}" | mtail -progs "${PROGS}" -logfds 0 https://github.com/wikimedia/puppet/blob/production/modules/varnish/files/varnishmtail#L23 |
Aha I see now how it works. Thanks for that and the link to the puppet
module.
The fds 0 1 and 2 are the only ones mtail can possibly use because they're
created by the OS; I had removed the functionality because I had believed
that a parent process would be executing mtail by itself, and passing a
file descriptor to the child process, and then telling mtail what it was
with this flag.
So... it does in fact work that way, because stdin is always fd 0.
I'll add it back, but in the meantime does trying to open -log /dev/stdin
work ?
…On Mon., 15 Jul. 2019, 19:57 John Bond, ***@***.***> wrote:
Thanks for getting back and also it seems my colleague filippog (who
originally opened this issue) also contacted you. I think its best to
continue the discussion here and save you answering both of us. To answer
your question we run the program with the following
/usr/bin/varnishncsa -n frontend -F "${FMT}" | mtail -progs "${PROGS}" -logfds 0
https://github.com/wikimedia/puppet/blob/production/modules/varnish/files/varnishmtail#L23
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3?email_source=notifications&email_token=AAXFX62PEIQ2DMLJIJYQFU3P7RCYHA5CNFSM4BDSYXV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ5HGTI#issuecomment-511341389>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAXFX645EMHGZHHYE6WKWK3P7RCYHANCNFSM4BDSYXVQ>
.
|
I have had a quick look at it seems [to me] that -log /dev/stdin does work however i want filippog to confirm as I'm not that familiar with mtail/Prometheus etc. It it does work as it seems then we will update our usage to use -logs, I have already prepared the PR[1] Thanks [1]https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/523739 |
I can confirm that |
@jaqx0r with filippog's confirmation i will merge my change tomorrow and will use Thanks |
Great. I'm not certain that /dev/stdin is on all platforms so I want to add
'-' as a special filename that means stdin, like many gnu tools do.
…On Thu., 18 Jul. 2019, 04:22 John Bond, ***@***.***> wrote:
@jaqx0r <https://github.com/jaqx0r> with filippog's confirmation i will
merge my change tomorrow and will use -log /dev/stdin as such there is no
need to add -logfds back if its just us asking for it
Thanks
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3?email_source=notifications&email_token=AAXFX64H3CMKTB2TETUOE5TP75PNJA5CNFSM4BDSYXV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2GE4GA#issuecomment-512511512>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAXFX63EGDLA5B6BWLTW7XLP75PNJANCNFSM4BDSYXVQ>
.
|
The logfds switch has been removed[1] and -logs /dev/stdin seems to provide the same functionality see this thread: google/mtail#3 (comment) Bug: T225604 Change-Id: I92554432d05698c5adbea1392eb300161573f55d
When using
Am I doing somehing wrong? My starting command looks like this:
EDIT: These are the logs during startup:
|
In #479 we're adding functionality to parse the log path pattern, once that has landed I'll add the ability to pass |
@slomo Please can you try again with a current |
hi,
thanks for mtail, looks really useful! Is reading from stdin something supported or considered? the use case I have in mind would be programs that don't log to file but in a ring memory buffer (e.g. varnish)
thanks!
The text was updated successfully, but these errors were encountered: