/profile endpoint using https://github.com/bdarnell/plop profiler

[kmike]

Just in case :)

[pablohoffman]

The only concern I have about this is making sure it doesn't get abused, since it could be easily used to take down a production server. Maybe we can protect it somehow?. Like putting it behind some static auth, configured in settings. Or even passing a token to the url (also configured in settings or command line arg).

[kmike]

I don't think there is much overhead here. Plop is a sampling profiler - it basically saves info about current frame (increments a counter for it) 100 times a second (this is customizable, default interval is 0.01s). This is nothing like cProfile overhead which increments a counter for every frame changed. It is true that /profile takes a persistent connection to the client (for 30s by default), but so do render.* endpoints if remote host is show to respond. As measured by CPU monitor, single /debug call uses more CPU time that leaving /profile ON for 30s (in fact, on my machine I can't notice /profile call in 0.01s resolution, while /debug call increases total CPU time for 0.04s). It is way easier to abuse splash by submitting render requests :)

[pablohoffman]

Should we stop the collector on an addBoth callback?. Otherwise, if the request fails (connection closed or similar) it may never be called and leave the collector running.

[dangra]

you may found request.notifyFinish method useful, it is used with the same goal in https://github.com/scrapy/scrapy/blob/master/scrapy/tests/mockserver.py#L48

[pablohoffman]

Thanks for the clarification @kmike, I'm no longer concerned now :). I added another comment.

[kmike]

Thanks for comments! Code was indeed messy.

I have another concern about this PR: it exposes paths and line numbers of code that is executed, and this could be seen as a security risk, because based on this information offender can figure out directory structure + Python, twisted, PyQT and splash versions.

[andresp99999]

I believe this PR would be quite useful. About the security concerns maybe we can make it disabled by default (with a setting) and then a Splash admin can enable it if it needs to do some profiling.

[pablohoffman]

Agreed, I don't have any objections to merge it.

As for the security concern, the simplest would be to add a debug token (configured in settings) that you'd need to pass to the /profile url (in an argument) in order to work.

Can we add that and merge?

[kmike]

I've added token support. TODO:

• tests;
• changes to debian package

I don't want to add default token to debian package because most people would leave this insecure default as-is. It looks like a better idea to get the token from environment variable. Is this ok?

[pablohoffman]

Yeah, environment variable or make the settings overrideable with a /etc/splash/local_settings.py.