FPM logger refactoring #2458

Merged
merged 1 commit into from Jul 7, 2018

Conversation

@bukka
Contributor

bukka commented Apr 7, 2017

This is just an initial work on the log limit for FPM (just a zlog part). It introduces a new global conf log_limit that defaults to previous value 1024. Currently this is meant mainly for error_log function and removing truncating of child pipes from stdio when catch_workers_output is enabled (the ...). However that part is still wrapped and I think we should have an another config for that as it's not log related exactly. I actually created an empty API in zlog to address that and it's called zlog_stream which will allow push based approach. The thing is that zlog just adds a prefix and then pass it to the zlog_fd (unless it is set for syslog which still requires buffering). If zlog_fd is a normal stream, then we should be able to just pass it without extra copying (it means doing multiple writes). The situation gets slightly more complicated with children (workers) which is for example used when you do error_log. They do also external logging using fcgi_write in sapi_cgi_log_fastcgi but it could be also optimised to not do extra copying.

So from the above it is clear that there is a huge space for improvements and optimization and the patch will be probably considerably changed before I remove the WIP from title and make it ready for review (btw. it's still probably buggy so don't use it anywhere yet :) ).

I'm opening this PR to mainly see what others think about log_limit setting and also about another option called worker_output_limit that I'm preparing for the piped variant (the one used when catch_workers_output set). That would basically extend the size of the wrapping. I was also thinking about another option that would allow to remove prefix [pool %s] child %d said into %s: \"%s\, ugly double quoting and maybe even the warning prefix but not sure atm. I will be glad to hear your thoughts about that! :)

Also this is an initial answer to #1076 . Atm. it's based on master 7.1 but the final branch will depend how much refactoring I will do, how much confident I will be with the changes and of course on the RM. If it gets too big I will probably rebase it to master...

@krakjoe krakjoe added the Enhancement label Apr 8, 2017

@bukka bukka changed the title from WIP: FPM log limit to WIP: FPM logger refactoring Apr 18, 2017

@bukka bukka changed the base branch from PHP-7.1 to master Apr 18, 2017

@arielkung

This comment has been minimized.

Show comment
Hide comment
@arielkung

arielkung Jun 9, 2017

Hi @bukka @krakjoe when do you think this will be released? Some minor version of 7.1 or we will have to wait for 7.2?
Thanks!

Hi @bukka @krakjoe when do you think this will be released? Some minor version of 7.1 or we will have to wait for 7.2?
Thanks!

@nahpeps

This comment has been minimized.

Show comment
Hide comment
@nahpeps

nahpeps Jun 9, 2017

We are heavily facing this issue and would be more than happy if this could make it into a PHP release quite fast! :-)

nahpeps commented Jun 9, 2017

We are heavily facing this issue and would be more than happy if this could make it into a PHP release quite fast! :-)

@arielkung

This comment has been minimized.

Show comment
Hide comment
@arielkung

arielkung Jun 9, 2017

Totally the same here

Totally the same here

@bukka

This comment has been minimized.

Show comment
Hide comment
@bukka

bukka Jun 11, 2017

Contributor

I'm afraid it will have to wait officially till 7.3 as there is still quite a lot of work to do and my time quite limited considering that I also need to couple of things in json and openssl ext before the 7.2 deadline that is very close.

However I'm thinking about creating a separate SAPI that I would maintain separately and it would work with 7.0+. As I said it still needs quite a lot of work as there quite a few issues and needs some extended testing. I will prioritise it after the first beta for 7.2.

Contributor

bukka commented Jun 11, 2017

I'm afraid it will have to wait officially till 7.3 as there is still quite a lot of work to do and my time quite limited considering that I also need to couple of things in json and openssl ext before the 7.2 deadline that is very close.

However I'm thinking about creating a separate SAPI that I would maintain separately and it would work with 7.0+. As I said it still needs quite a lot of work as there quite a few issues and needs some extended testing. I will prioritise it after the first beta for 7.2.

@blodan

This comment has been minimized.

Show comment
Hide comment
@blodan

blodan Jun 22, 2017

This addition would be awseome!

I'm currently patching the hardcoded limit value everytime a new release is built :)

blodan commented Jun 22, 2017

This addition would be awseome!

I'm currently patching the hardcoded limit value everytime a new release is built :)

@bukka

This comment has been minimized.

Show comment
Hide comment
@bukka

bukka Nov 12, 2017

Contributor

Just to give an update. I'm slowly working on it as part of FPMi next branch which is atm just current FPM + logging changes. Once I have it more or less stable and tested, I will update this PR.

Contributor

bukka commented Nov 12, 2017

Just to give an update. I'm slowly working on it as part of FPMi next branch which is atm just current FPM + logging changes. Once I have it more or less stable and tested, I will update this PR.

@tomfotherby

This comment has been minimized.

Show comment
Hide comment
@tomfotherby

tomfotherby Nov 13, 2017

I was also thinking about another option that would allow to remove prefix [pool %s] child %d said into %s: "%s, ugly double quoting and maybe even the warning prefix but not sure atm. I will be glad to hear your thoughts about that!

This would be a huge advantage for the community running php in docker where there is a convention to log everything to SDTOUT so that log aggregation can have a common solution for all containers, not on a app-by-app basis. It's a big problem at the moment that app logs have the above mentioned prefix and warning. It would be a huge WIN to be able to remove it (or customise it).

I was also thinking about another option that would allow to remove prefix [pool %s] child %d said into %s: "%s, ugly double quoting and maybe even the warning prefix but not sure atm. I will be glad to hear your thoughts about that!

This would be a huge advantage for the community running php in docker where there is a convention to log everything to SDTOUT so that log aggregation can have a common solution for all containers, not on a app-by-app basis. It's a big problem at the moment that app logs have the above mentioned prefix and warning. It would be a huge WIN to be able to remove it (or customise it).

@ingluisjimenez

This comment has been minimized.

Show comment
Hide comment
@ingluisjimenez

ingluisjimenez Feb 5, 2018

@bukka this is definitely hero's work, appreciate your take on this. Is there any way we can help you on this?

@bukka this is definitely hero's work, appreciate your take on this. Is there any way we can help you on this?

@bukka

This comment has been minimized.

Show comment
Hide comment
@bukka

bukka Feb 22, 2018

Contributor

@ingluisjimenez thanks for offering the help. I will send an update once I'm happy with the patch and then it would be really great if you and possibly others could test it!

In the meantime the progress (currently I'm a bit slow but hope to have more time soon) can be seen here: bukka/fpmi@0.1.0...next

Contributor

bukka commented Feb 22, 2018

@ingluisjimenez thanks for offering the help. I will send an update once I'm happy with the patch and then it would be really great if you and possibly others could test it!

In the meantime the progress (currently I'm a bit slow but hope to have more time soon) can be seen here: bukka/fpmi@0.1.0...next

@dzuelke

This comment has been minimized.

Show comment
Hide comment
@dzuelke

dzuelke Apr 7, 2018

Contributor

Thanks for your recent work on this, @bukka. Can I help test anything, discuss anything, verify anything, give feedback on current state?

Contributor

dzuelke commented Apr 7, 2018

Thanks for your recent work on this, @bukka. Can I help test anything, discuss anything, verify anything, give feedback on current state?

@bukka

This comment has been minimized.

Show comment
Hide comment
@bukka

bukka Apr 8, 2018

Contributor

@dzuelke Thanks! I think I have got most of the functionality in there (currently just in fpmi as linked above) but it needs some optimization for buffered logging - mainly reusing buffer. Also need to do a little bit more testing on my side but hopefully won't take too much time. Once it's done it would be great if you and others could test it. I will send a bit more detailed description of what has been done.

Contributor

bukka commented Apr 8, 2018

@dzuelke Thanks! I think I have got most of the functionality in there (currently just in fpmi as linked above) but it needs some optimization for buffered logging - mainly reusing buffer. Also need to do a little bit more testing on my side but hopefully won't take too much time. Once it's done it would be great if you and others could test it. I will send a bit more detailed description of what has been done.

@dzuelke

This comment has been minimized.

Show comment
Hide comment
@dzuelke

dzuelke Apr 8, 2018

Contributor

No problem @bukka, I can set aside work time to assist with testing etc. Just ping me when!

Contributor

dzuelke commented Apr 8, 2018

No problem @bukka, I can set aside work time to assist with testing etc. Just ping me when!

@bukka bukka changed the title from WIP: FPM logger refactoring to FPM logger refactoring Jun 15, 2018

@bukka

This comment has been minimized.

Show comment
Hide comment
@bukka

bukka Jun 15, 2018

Contributor

@dzuelke , @ingluisjimenez and others: I think it's ready for testing. There are 3 new options:

  • log_limit: (global) max number of characters in a single line
  • log_buffering: (global) whether to buffer line data and write them in a single write - it's more experimental and the default is on which should do for most cases.
  • decorate_worker_output: (pool) whether to use the prefix - basically as requested in comment #2458 (comment) from @tomfotherby

A little more info about the options can be seen in the updated conf files.

Contributor

bukka commented Jun 15, 2018

@dzuelke , @ingluisjimenez and others: I think it's ready for testing. There are 3 new options:

  • log_limit: (global) max number of characters in a single line
  • log_buffering: (global) whether to buffer line data and write them in a single write - it's more experimental and the default is on which should do for most cases.
  • decorate_worker_output: (pool) whether to use the prefix - basically as requested in comment #2458 (comment) from @tomfotherby

A little more info about the options can be seen in the updated conf files.

sapi/fpm/php-fpm.conf.in
+; logging to a file descriptor. It means the new line character is not present
+; when logging to syslog.
+; Default Value: 1024
+;log_level = 4096

This comment has been minimized.

@jimbobhickville

jimbobhickville Jun 21, 2018

Shouldn't this say log_limit instead of log_level?

@jimbobhickville

jimbobhickville Jun 21, 2018

Shouldn't this say log_limit instead of log_level?

This comment has been minimized.

@bukka

bukka Jun 22, 2018

Contributor

Ah good catch! Will fix it! Thanks!

@bukka

bukka Jun 22, 2018

Contributor

Ah good catch! Will fix it! Thanks!

This comment has been minimized.

@bukka

bukka Jun 22, 2018

Contributor

Should be fixed now!

@bukka

bukka Jun 22, 2018

Contributor

Should be fixed now!

@dzuelke

This comment has been minimized.

Show comment
Hide comment
@dzuelke

dzuelke Jun 22, 2018

Contributor

Is there any reason log_limit couldn't also be 0?

Contributor

dzuelke commented Jun 22, 2018

Is there any reason log_limit couldn't also be 0?

@dzuelke

This comment has been minimized.

Show comment
Hide comment
@dzuelke

dzuelke Jun 22, 2018

Contributor

Could you also describe the idea behind log_buffering? With that on, does a single line get buffered, or multiple?

Contributor

dzuelke commented Jun 22, 2018

Could you also describe the idea behind log_buffering? With that on, does a single line get buffered, or multiple?

@bukka

This comment has been minimized.

Show comment
Hide comment
@bukka

bukka Jun 22, 2018

Contributor

@dzuelke

Is there any reason log_limit couldn't also be 0?

I have been thinking about it but if buffering is used (which is default and primary mode), buffer is reused so it is kept in the memory and it could result in a really big buffer. It's kind of like zones in nginx (with an exception that it's not a shared memory [just master allocated memory] and it's the maximum that it can grow to) - you don't create an unlimited zone. If you want to allow really long lines (note that this is a limit for a single line), then you can set it high but there should be always some reasonable limit IMHO.

Could you also describe the idea behind log_buffering? With that on, does a single line get buffered, or multiple?

If it's on (default), then it's a buffering for a single line. Basically the line is then written in a single write call.

Contributor

bukka commented Jun 22, 2018

@dzuelke

Is there any reason log_limit couldn't also be 0?

I have been thinking about it but if buffering is used (which is default and primary mode), buffer is reused so it is kept in the memory and it could result in a really big buffer. It's kind of like zones in nginx (with an exception that it's not a shared memory [just master allocated memory] and it's the maximum that it can grow to) - you don't create an unlimited zone. If you want to allow really long lines (note that this is a limit for a single line), then you can set it high but there should be always some reasonable limit IMHO.

Could you also describe the idea behind log_buffering? With that on, does a single line get buffered, or multiple?

If it's on (default), then it's a buffering for a single line. Basically the line is then written in a single write call.

@dzuelke

This comment has been minimized.

Show comment
Hide comment
@dzuelke

dzuelke Jun 22, 2018

Contributor

@bukka

If it's on (default), then it's a buffering for a single line. Basically the line is then written in a single write call.

I wonder if disabling that is useful in any way. Presumably a program could e.g. file_put_contents("php://stderr", "ohai, no newline");, but what use is that? and what happens if another log call arrives before the next portion of data, then the two lines get mixed up?

IMO writes should purely line based, and atomic (in one call). For logging to stdout/stderr that are not CLI apps (think curl download progress status), there is pretty much zero use in having unterminated lines, no?

Contributor

dzuelke commented Jun 22, 2018

@bukka

If it's on (default), then it's a buffering for a single line. Basically the line is then written in a single write call.

I wonder if disabling that is useful in any way. Presumably a program could e.g. file_put_contents("php://stderr", "ohai, no newline");, but what use is that? and what happens if another log call arrives before the next portion of data, then the two lines get mixed up?

IMO writes should purely line based, and atomic (in one call). For logging to stdout/stderr that are not CLI apps (think curl download progress status), there is pretty much zero use in having unterminated lines, no?

@bukka

This comment has been minimized.

Show comment
Hide comment
@bukka

bukka Jun 22, 2018

Contributor

@dzuelke I'm actually talking about a line in a single event (single call of fpm_stdio_child_said). Writing to stderr multiple times can result in a multiple events especially if there are operations between. To easily see that, consider following code:

file_put_contents('php://stderr', "a");
usleep(0.1);
file_put_contents('php://stderr', "b");
usleep(0.1);
file_put_contents('php://stderr', "c");

That will result in 3 events which means 3 log entries.

This patch doesn't address that as it would require some worker based caching.

Basically the whole point of buffering is to not write decoration and long messages in multiple calls. I think disabled buffering should be safe for most use cases though but it's still a bit experimental.

Contributor

bukka commented Jun 22, 2018

@dzuelke I'm actually talking about a line in a single event (single call of fpm_stdio_child_said). Writing to stderr multiple times can result in a multiple events especially if there are operations between. To easily see that, consider following code:

file_put_contents('php://stderr', "a");
usleep(0.1);
file_put_contents('php://stderr', "b");
usleep(0.1);
file_put_contents('php://stderr', "c");

That will result in 3 events which means 3 log entries.

This patch doesn't address that as it would require some worker based caching.

Basically the whole point of buffering is to not write decoration and long messages in multiple calls. I think disabled buffering should be safe for most use cases though but it's still a bit experimental.

@dzuelke

This comment has been minimized.

Show comment
Hide comment
@dzuelke

dzuelke Jun 25, 2018

Contributor

The alternative would be to not print stdout/stderr contents in the master process until the worker has either sent a newline, or the worker is done.

Then behavior (with decoration off) would be identical to what happens when you run the above script on the CLI, where you get three characters next to each other, not separated by newlines.

Contributor

dzuelke commented Jun 25, 2018

The alternative would be to not print stdout/stderr contents in the master process until the worker has either sent a newline, or the worker is done.

Then behavior (with decoration off) would be identical to what happens when you run the above script on the CLI, where you get three characters next to each other, not separated by newlines.

@bukka

This comment has been minimized.

Show comment
Hide comment
@bukka

bukka Jun 28, 2018

Contributor

@dzuelke yep that would be ideal. The only problem is that it would require to cache zlog_stream in each worker and flushing that on request finish which requires some implementation and that means time that I don't currently have. Basically it's out of scope for this PR. You are more then welcome to create a follow up PR if you fancy to look into it. ;)

I just add what's the actual scope of this PR:

  • having configurable limit
  • prevent trimming of the wrapped messages
  • fix the wrapped size to be the same as the limit (it wasn't the case before)
  • add options to disable decoration which means just message prefix and suffix
  • experimental option for direct write (disabling of buffering)

All other things should stay as they are now.

Anyway if there are no objections or blockers, I plan to merge it soon - possibly this or next weekend.

Contributor

bukka commented Jun 28, 2018

@dzuelke yep that would be ideal. The only problem is that it would require to cache zlog_stream in each worker and flushing that on request finish which requires some implementation and that means time that I don't currently have. Basically it's out of scope for this PR. You are more then welcome to create a follow up PR if you fancy to look into it. ;)

I just add what's the actual scope of this PR:

  • having configurable limit
  • prevent trimming of the wrapped messages
  • fix the wrapped size to be the same as the limit (it wasn't the case before)
  • add options to disable decoration which means just message prefix and suffix
  • experimental option for direct write (disabling of buffering)

All other things should stay as they are now.

Anyway if there are no objections or blockers, I plan to merge it soon - possibly this or next weekend.

@bukka

This comment has been minimized.

Show comment
Hide comment
@bukka

bukka Jun 28, 2018

Contributor

@dzuelke btw. it might be actually good idea to create a bug report for that issue so it's not forgotten...

Contributor

bukka commented Jun 28, 2018

@dzuelke btw. it might be actually good idea to create a bug report for that issue so it's not forgotten...

@dzuelke

This comment has been minimized.

Show comment
Hide comment
@dzuelke

dzuelke Jul 4, 2018

Contributor

Alright, found a few issues, @bukka.

First, I get ", pipe is closed" suffixes all the time (see outputs below).

Second, the behavior for how output is grouped seems erratic. Refreshing the following program a few times:

file_put_contents('php://stderr', "a");
usleep(0.1);
file_put_contents('php://stderr', "b");
usleep(0.1);
file_put_contents('php://stderr', "c");

I get (blank lines mine for readability), with log_buffering set to on:

[04-Jul-2018 21:02:14] WARNING: [pool www] child 19516 said into stderr: "ab", pipe is closed
[04-Jul-2018 21:02:14] WARNING: [pool www] child 19516 said into stderr: "c", pipe is closed

[04-Jul-2018 21:02:54] WARNING: [pool www] child 19517 said into stderr: "ab", pipe is closed
[04-Jul-2018 21:02:54] WARNING: [pool www] child 19517 said into stderr: "c", pipe is closed

[04-Jul-2018 21:02:56] WARNING: [pool www] child 19518 said into stderr: "a", pipe is closed
[04-Jul-2018 21:02:56] WARNING: [pool www] child 19518 said into stderr: "bc", pipe is closed

[04-Jul-2018 21:02:57] WARNING: [pool www] child 19519 said into stderr: "a", pipe is closed
[04-Jul-2018 21:02:57] WARNING: [pool www] child 19519 said into stderr: "b", pipe is closed
[04-Jul-2018 21:02:57] WARNING: [pool www] child 19519 said into stderr: "c", pipe is closed

[04-Jul-2018 21:03:00] WARNING: [pool www] child 19520 said into stderr: "a", pipe is closed
[04-Jul-2018 21:03:00] WARNING: [pool www] child 19520 said into stderr: "bc", pipe is closed

[04-Jul-2018 21:03:02] WARNING: [pool www] child 19521 said into stderr: "abc", pipe is closed

With log_buffering set to off, it's similar:

[04-Jul-2018 21:05:57] WARNING: [pool www] child 19740 said into stderr: "abc", pipe is closed

[04-Jul-2018 21:06:00] WARNING: [pool www] child 19741 said into stderr: "abc", pipe is closed

[04-Jul-2018 21:06:01] WARNING: [pool www] child 19742 said into stderr: "a", pipe is closed
[04-Jul-2018 21:06:01] WARNING: [pool www] child 19742 said into stderr: "bc", pipe is closed

[04-Jul-2018 21:06:04] WARNING: [pool www] child 19743 said into stderr: "ab", pipe is closed
[04-Jul-2018 21:06:04] WARNING: [pool www] child 19743 said into stderr: "c", pipe is closed

[04-Jul-2018 21:06:06] WARNING: [pool www] child 19744 said into stderr: "abc", pipe is closed

[04-Jul-2018 21:06:07] WARNING: [pool www] child 19745 said into stderr: "a", pipe is closed
[04-Jul-2018 21:06:07] WARNING: [pool www] child 19745 said into stderr: "bc", pipe is closed

[04-Jul-2018 21:06:08] WARNING: [pool www] child 19747 said into stderr: "a", pipe is closed
[04-Jul-2018 21:06:08] WARNING: [pool www] child 19747 said into stderr: "bc", pipe is closed

I can confirm that the decorate_workers_output (typo in one of your last messages FYI) directive works as intended 🎉 and the limiting and wrapping with decoration on is correct even with log_level = debug 👍

Splitting across lines seems buggy for very large messages, for example:

file_put_contents('php://stderr', str_repeat("If it were done when 'tis done, then 'twere well // It were done quickly. If the assassination // Could trammel up the consequence, and catch, // With his surcease, success; that but this blow // Might be the be-all and the end-all--here, // But here, upon this bank and shoal of time,-- // We'd jump the life to come. But in these cases // We still have judgement here; that we but teach // Bloody instructions, which being taught, return // To plague the inventor: this even-handed justice // Commends the ingredients of our poison'd chalice // To our own lips. He's here in double trust: // First, as I am his kinsman and his subject, // Strong both against the deed: then, as his host, // Who should against his murderer shut the door, // Not bear the knife myself. Besides, this Duncan // Hath borne his faculties so meek, hath been // So clear in his great office, that his virtues // Will plead like angels, trumpet-tongued, against // The deep damnation of his taking-off: // And pity, like a naked new-born babe, // Striding the blast, or heaven's cherubin, hors'd // Upon the sightless couriers of the air, // Shall blow the horrid deed in every eye, // That tears shall drown the wind.--I have no spur // To prick the sides of my intent, but only // Vaulting ambition, which o'erleaps itself, // And falls on the other. // Is this a dagger which I see before me, // The handle toward my hand? Come, let me clutch thee:-- // I have thee not, and yet I see thee still. // Art thou not, fatal vision, sensible // To feeling as to sight? or art thou but // A dagger of the mind, a false creation, // Proceeding from the heat-oppressed brain? // I see thee yet, in form as palpable // As this which now I draw. // Thou marshall'st me the way that I was going; // And such an instrument I was to use. // Mine eyes are made the fools o' the other senses, // Or else worth all the rest: I see thee still; // And on thy blade and dudgeon gouts of blood, // Which was not so before.--There's no such thing: // It is the bloody business which informs // Thus to mine eyes.--Now o'er the one half-world // Nature seems dead, and wicked dreams abuse // The curtain'd sleep; now witchcraft celebrates // Pale Hecate's offerings; and wither'd murder, // Alarum'd by his sentinel, the wolf, // Whose howl's his watch, thus with his stealthy pace, // With Tarquin's ravishing strides, towards his design // Moves like a ghost.--Thou sure and firm-set earth, // Hear not my steps, which way they walk, for fear // Thy very stones prate of my whereabout, // And take the present horror from the time, // Which now suits with it.--Whiles I threat, he lives; // Words to the heat of deeds too cold breath gives. // ", 1000));

With log_limit = 3000000, this outputs correctly to a single line.

However, with log_limit = 2000000, I get eight lines (each with decoration, and it's not the shell splitting it, because they all end correctly in …", pipe is closed):

  1. 1261660 bytes
  2. 311388 bytes
  3. 213084 bytes
  4. 211917 bytes
  5. 206059 bytes
  6. 211276 bytes
  7. 278620 bytes

With decoration off:

  1. 2000000 bytes
  2. 445036 bytes
  3. 259963 bytes

At a more reasonable limit of, say, 16384, it seems fine.

However:

ext/filter/filter.c(483) :  Freeing 0x0000000102c02000 (5 bytes), script=-
Last leak repeated 4 times
=== Total 5 memory leaks detected ===
Contributor

dzuelke commented Jul 4, 2018

Alright, found a few issues, @bukka.

First, I get ", pipe is closed" suffixes all the time (see outputs below).

Second, the behavior for how output is grouped seems erratic. Refreshing the following program a few times:

file_put_contents('php://stderr', "a");
usleep(0.1);
file_put_contents('php://stderr', "b");
usleep(0.1);
file_put_contents('php://stderr', "c");

I get (blank lines mine for readability), with log_buffering set to on:

[04-Jul-2018 21:02:14] WARNING: [pool www] child 19516 said into stderr: "ab", pipe is closed
[04-Jul-2018 21:02:14] WARNING: [pool www] child 19516 said into stderr: "c", pipe is closed

[04-Jul-2018 21:02:54] WARNING: [pool www] child 19517 said into stderr: "ab", pipe is closed
[04-Jul-2018 21:02:54] WARNING: [pool www] child 19517 said into stderr: "c", pipe is closed

[04-Jul-2018 21:02:56] WARNING: [pool www] child 19518 said into stderr: "a", pipe is closed
[04-Jul-2018 21:02:56] WARNING: [pool www] child 19518 said into stderr: "bc", pipe is closed

[04-Jul-2018 21:02:57] WARNING: [pool www] child 19519 said into stderr: "a", pipe is closed
[04-Jul-2018 21:02:57] WARNING: [pool www] child 19519 said into stderr: "b", pipe is closed
[04-Jul-2018 21:02:57] WARNING: [pool www] child 19519 said into stderr: "c", pipe is closed

[04-Jul-2018 21:03:00] WARNING: [pool www] child 19520 said into stderr: "a", pipe is closed
[04-Jul-2018 21:03:00] WARNING: [pool www] child 19520 said into stderr: "bc", pipe is closed

[04-Jul-2018 21:03:02] WARNING: [pool www] child 19521 said into stderr: "abc", pipe is closed

With log_buffering set to off, it's similar:

[04-Jul-2018 21:05:57] WARNING: [pool www] child 19740 said into stderr: "abc", pipe is closed

[04-Jul-2018 21:06:00] WARNING: [pool www] child 19741 said into stderr: "abc", pipe is closed

[04-Jul-2018 21:06:01] WARNING: [pool www] child 19742 said into stderr: "a", pipe is closed
[04-Jul-2018 21:06:01] WARNING: [pool www] child 19742 said into stderr: "bc", pipe is closed

[04-Jul-2018 21:06:04] WARNING: [pool www] child 19743 said into stderr: "ab", pipe is closed
[04-Jul-2018 21:06:04] WARNING: [pool www] child 19743 said into stderr: "c", pipe is closed

[04-Jul-2018 21:06:06] WARNING: [pool www] child 19744 said into stderr: "abc", pipe is closed

[04-Jul-2018 21:06:07] WARNING: [pool www] child 19745 said into stderr: "a", pipe is closed
[04-Jul-2018 21:06:07] WARNING: [pool www] child 19745 said into stderr: "bc", pipe is closed

[04-Jul-2018 21:06:08] WARNING: [pool www] child 19747 said into stderr: "a", pipe is closed
[04-Jul-2018 21:06:08] WARNING: [pool www] child 19747 said into stderr: "bc", pipe is closed

I can confirm that the decorate_workers_output (typo in one of your last messages FYI) directive works as intended 🎉 and the limiting and wrapping with decoration on is correct even with log_level = debug 👍

Splitting across lines seems buggy for very large messages, for example:

file_put_contents('php://stderr', str_repeat("If it were done when 'tis done, then 'twere well // It were done quickly. If the assassination // Could trammel up the consequence, and catch, // With his surcease, success; that but this blow // Might be the be-all and the end-all--here, // But here, upon this bank and shoal of time,-- // We'd jump the life to come. But in these cases // We still have judgement here; that we but teach // Bloody instructions, which being taught, return // To plague the inventor: this even-handed justice // Commends the ingredients of our poison'd chalice // To our own lips. He's here in double trust: // First, as I am his kinsman and his subject, // Strong both against the deed: then, as his host, // Who should against his murderer shut the door, // Not bear the knife myself. Besides, this Duncan // Hath borne his faculties so meek, hath been // So clear in his great office, that his virtues // Will plead like angels, trumpet-tongued, against // The deep damnation of his taking-off: // And pity, like a naked new-born babe, // Striding the blast, or heaven's cherubin, hors'd // Upon the sightless couriers of the air, // Shall blow the horrid deed in every eye, // That tears shall drown the wind.--I have no spur // To prick the sides of my intent, but only // Vaulting ambition, which o'erleaps itself, // And falls on the other. // Is this a dagger which I see before me, // The handle toward my hand? Come, let me clutch thee:-- // I have thee not, and yet I see thee still. // Art thou not, fatal vision, sensible // To feeling as to sight? or art thou but // A dagger of the mind, a false creation, // Proceeding from the heat-oppressed brain? // I see thee yet, in form as palpable // As this which now I draw. // Thou marshall'st me the way that I was going; // And such an instrument I was to use. // Mine eyes are made the fools o' the other senses, // Or else worth all the rest: I see thee still; // And on thy blade and dudgeon gouts of blood, // Which was not so before.--There's no such thing: // It is the bloody business which informs // Thus to mine eyes.--Now o'er the one half-world // Nature seems dead, and wicked dreams abuse // The curtain'd sleep; now witchcraft celebrates // Pale Hecate's offerings; and wither'd murder, // Alarum'd by his sentinel, the wolf, // Whose howl's his watch, thus with his stealthy pace, // With Tarquin's ravishing strides, towards his design // Moves like a ghost.--Thou sure and firm-set earth, // Hear not my steps, which way they walk, for fear // Thy very stones prate of my whereabout, // And take the present horror from the time, // Which now suits with it.--Whiles I threat, he lives; // Words to the heat of deeds too cold breath gives. // ", 1000));

With log_limit = 3000000, this outputs correctly to a single line.

However, with log_limit = 2000000, I get eight lines (each with decoration, and it's not the shell splitting it, because they all end correctly in …", pipe is closed):

  1. 1261660 bytes
  2. 311388 bytes
  3. 213084 bytes
  4. 211917 bytes
  5. 206059 bytes
  6. 211276 bytes
  7. 278620 bytes

With decoration off:

  1. 2000000 bytes
  2. 445036 bytes
  3. 259963 bytes

At a more reasonable limit of, say, 16384, it seems fine.

However:

ext/filter/filter.c(483) :  Freeing 0x0000000102c02000 (5 bytes), script=-
Last leak repeated 4 times
=== Total 5 memory leaks detected ===
@bukka

This comment has been minimized.

Show comment
Hide comment
@bukka

bukka Jul 4, 2018

Contributor

Thanks for testing. I wouldn't too worry about that new line issue for now as it is kind of broken already but that large limit seemes definitelly like a bug that I need to fix. What about the shorter limit up to 10000? Have you seen any issue with that? Just checking if that issue is a blocker or I can merge it and fix it in the beta. The thing is that I need to merge it soon to get it to 7.3...

Contributor

bukka commented Jul 4, 2018

Thanks for testing. I wouldn't too worry about that new line issue for now as it is kind of broken already but that large limit seemes definitelly like a bug that I need to fix. What about the shorter limit up to 10000? Have you seen any issue with that? Just checking if that issue is a blocker or I can merge it and fix it in the beta. The thing is that I need to merge it soon to get it to 7.3...

@dzuelke

This comment has been minimized.

Show comment
Hide comment
@dzuelke

dzuelke Jul 4, 2018

Contributor

No issues with e.g. a 16k limit, like I said.

That might be a reasonable default too ;) It's Docker's max log line length for instance.

Do you need help with finding the memleak, @bukka?

Contributor

dzuelke commented Jul 4, 2018

No issues with e.g. a 16k limit, like I said.

That might be a reasonable default too ;) It's Docker's max log line length for instance.

Do you need help with finding the memleak, @bukka?

@dzuelke

This comment has been minimized.

Show comment
Hide comment
@dzuelke

dzuelke Jul 4, 2018

Contributor

Also would be A++ if we could land #3182 in 7.3 so that code can log the FPM pool if desired.

Contributor

dzuelke commented Jul 4, 2018

Also would be A++ if we could land #3182 in 7.3 so that code can log the FPM pool if desired.

@php-pulls php-pulls merged commit 3e5afbf into php:master Jul 7, 2018

0 of 2 checks passed

continuous-integration/appveyor/pr Waiting for AppVeyor build to complete
Details
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
@nikic

This comment has been minimized.

Show comment
Hide comment
@nikic

nikic Jul 7, 2018

Member

@bukka Just to check, does the merged version fix the "pipe is closed" issue from above? Or is that supposed to work that way?

Member

nikic commented Jul 7, 2018

@bukka Just to check, does the merged version fix the "pipe is closed" issue from above? Or is that supposed to work that way?

@bukka

This comment has been minimized.

Show comment
Hide comment
@bukka

bukka Jul 7, 2018

Contributor

@dzuelke Ok I merged it as it's already in much better state than the current logging but will take a look on that issue with a high log limit during the beta.

About the default change, I'm not really sure as it can technically break some old syslog loggers - not sure however how likely it is. I will keep it for now...

That leak looks strange as it is in filter extension so not really sure if it's even related as these are changes mainly for master process... If you can take a look, that would be great!

Contributor

bukka commented Jul 7, 2018

@dzuelke Ok I merged it as it's already in much better state than the current logging but will take a look on that issue with a high log limit during the beta.

About the default change, I'm not really sure as it can technically break some old syslog loggers - not sure however how likely it is. I will keep it for now...

That leak looks strange as it is in filter extension so not really sure if it's even related as these are changes mainly for master process... If you can take a look, that would be great!

@bukka

This comment has been minimized.

Show comment
Hide comment
@bukka

bukka Jul 7, 2018

Contributor

@nikic What "pipe is closed" issue? :) If you mean the line split, that was already present and never worked properly. The only issue that I know about is the one with very large log limit that I plan to take a look during the beta. I don't think it will require complex changes - the hardest part will be to debug it...

Contributor

bukka commented Jul 7, 2018

@nikic What "pipe is closed" issue? :) If you mean the line split, that was already present and never worked properly. The only issue that I know about is the one with very large log limit that I plan to take a look during the beta. I don't think it will require complex changes - the hardest part will be to debug it...

@dzuelke

This comment has been minimized.

Show comment
Hide comment
@dzuelke

dzuelke Jul 9, 2018

Contributor

Not the line split, @bukka, but note how every single file_put_contents() call from my examples above produced a ", pipe is closed" suffix. That doesn't happen with 7.2.

Contributor

dzuelke commented Jul 9, 2018

Not the line split, @bukka, but note how every single file_put_contents() call from my examples above produced a ", pipe is closed" suffix. That doesn't happen with 7.2.

@bukka

This comment has been minimized.

Show comment
Hide comment
@bukka

bukka Jul 15, 2018

Contributor

@dzuelke @nikic Ah I see! I think I got the logic wrong. That , pipe is closed should be there only if the read fails with and the res < 0 so that's a bug. Will need to fix it.

Contributor

bukka commented Jul 15, 2018

@dzuelke @nikic Ah I see! I think I got the logic wrong. That , pipe is closed should be there only if the read fails with and the res < 0 so that's a bug. Will need to fix it.

@dzuelke

This comment has been minimized.

Show comment
Hide comment
@dzuelke

dzuelke Jul 20, 2018

Contributor

I took a brief look at the code to see if I could fix it in one minute or two.

Is this suffix even necessary? What actionable information does ", pipe is closed" get anyone? Can't this just be dropped entirely, @bukka and @nikic?

Contributor

dzuelke commented Jul 20, 2018

I took a brief look at the code to see if I could fix it in one minute or two.

Is this suffix even necessary? What actionable information does ", pipe is closed" get anyone? Can't this just be dropped entirely, @bukka and @nikic?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment