Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Monolog] Added ElasticsearchLogstashHandler #32360

Open
wants to merge 1 commit into
base: 4.4
from

Conversation

@lyrixx
Copy link
Member

commented Jul 4, 2019

Q A
Branch? 4.4
Bug fix? no
New feature? yes
BC breaks? no
Deprecations? no
Tests pass? yes
Fixed tickets
License MIT
Doc PR

This PR was initially submitted on Monolog.
It has been refused , but Jordi suggested to add it to the symfony bridge. So here we go :)


ATM, there are few options to push log to Elastic Stack in order to play with them:

  • install logstash and use a Gelf handler. It works but you have to install logstash and configure it. Not an easy task. More over, it need an extra PHP package
  • use the ES handler: It does not play well with context and extra: Kibana is not able to filter on nested object. And this handler is tightly coupled to the ElasticaFormatter formater. More over, it need an extra PHP package
  • use something to parse file logs. This is really a bad idea since it involves a parsing... More over a daemon is needed to do that (file beat / logstash / you name it)

This is why I'm introducing a new Handler.

  • There is not need to install anything (expect ES, of course)
  • It play very well with Kibana, as it uses the Logstash format
  • It requires symfony/http-client, but in a modern PHP application (SF 4.3) this dependency is already present
  • It slow down a bit the application since it trigger an HTTP request for each logs. But symfony/http-client is non-blocking. If you want to use it in production, I recommend to wrap this handler in a buffer handler or a cross-finger handle to have only one HTTP call.

Some performance consideration en a prod env with a buffer handler + this one

As you can see, as requests are made synchronously, there is no penalty on AppKernel::Handler() 馃槏! But the PHP worker has more work to do, and it's busy much more time (about X2)

I explained everything in the PHP Doc Block


This is what you can expect out of the box
image

@nicolas-grekas

This comment has been minimized.

Copy link
Member

commented Jul 4, 2019

Note: This handler is not aimed to be used in productio

I'd suggest telling this in the docblock + explain how to do it in prod

@lyrixx

This comment has been minimized.

Copy link
Member Author

commented Jul 4, 2019

I'd suggest telling this in the docblock + explain how to do it in prod

Actually, I could work a bit on it to make it available on production:
It it's used in combinaison with a buffer handler, or a finger cross handler, all logs could be sent by batch, and so only one HTTP call.

WDYT?

@lyrixx lyrixx force-pushed the lyrixx:monolog-elastic branch from 017cacb to efd4e54 Jul 4, 2019

@fabpot
Copy link
Member

left a comment

Your suggestion for a prod mode makes sense to me.

@lyrixx lyrixx force-pushed the lyrixx:monolog-elastic branch 2 times, most recently from 0a45301 to c30cf2d Jul 8, 2019

@lyrixx

This comment has been minimized.

Copy link
Member Author

commented Jul 8, 2019

  • I updated the PHPdoc.
  • I added blackfire profile to the PR description
  • I enhanced the main dockbloc
  • I added an entry to the readme
  • I changed the default host to 127.0.0.1:9200
  • I added a timeout to 1 sec
@lyrixx

This comment has been minimized.

Copy link
Member Author

commented Jul 8, 2019

@nicolas-grekas How can I disable all exceptions in the http-client? Or at least catch them all? (But I think this is not do-able since I use the __destruct feature)

@lyrixx

This comment has been minimized.

Copy link
Member Author

commented Jul 8, 2019

Hmm, I managed to do something. Could you look at the last commit? Thanks

@lyrixx lyrixx force-pushed the lyrixx:monolog-elastic branch from 8db8b4b to 9c26eb9 Jul 8, 2019

@lyrixx lyrixx force-pushed the lyrixx:monolog-elastic branch from 9c26eb9 to 67c1c45 Jul 16, 2019

@lyrixx

This comment has been minimized.

Copy link
Member Author

commented Jul 16, 2019

I rebased this PR. It's now ready and green 馃挌

@noniagriconomie
Copy link
Contributor

left a comment

minor doc review :)

edit : do you plan to add real doc, or a blog post to explain the config for dev/vs prod ?
i think it is a must have, to not degrade perf on prod :)

@lyrixx

This comment has been minimized.

Copy link
Member Author

commented Jul 16, 2019

edit : do you plan to add real doc, or a blog post to explain the config for dev/vs prod ?
i think it is a must have, to not degrade perf on prod :)

Not really. There is already many blog post on internet about symfony + ELK

@lyrixx lyrixx force-pushed the lyrixx:monolog-elastic branch from 67c1c45 to 957d8c2 Jul 16, 2019

@lyrixx

This comment has been minimized.

Copy link
Member Author

commented Jul 16, 2019

I addressed your comments

@lyrixx lyrixx force-pushed the lyrixx:monolog-elastic branch 2 times, most recently from acf80a7 to 982f437 Jul 17, 2019

@lyrixx lyrixx force-pushed the lyrixx:monolog-elastic branch 4 times, most recently from e0751bd to 1ab999b Jul 17, 2019

@nicolas-grekas
Copy link
Member

left a comment

Nice :)

@lyrixx lyrixx force-pushed the lyrixx:monolog-elastic branch from 1ab999b to 0631133 Jul 17, 2019

@noniagriconomie

This comment has been minimized.

Copy link
Contributor

commented Jul 17, 2019

@lyrixx

Not really. There is already many blog post on internet about symfony + ELK

my comment was more for configuring this new handler in the proper way inside a symfony dev|prod stack :)

@lyrixx

This comment has been minimized.

Copy link
Member Author

commented Jul 17, 2019

my comment was more for configuring this new handler in the proper way inside a symfony dev|prod stack :)

This will be done in the monolog-bundle :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can鈥檛 perform that action at this time.