SQL query audit + add database indexes #127

Open
tobscure opened this Issue Jun 20, 2015 · 7 comments

Projects

None yet

6 participants

@tobscure
Member

Lots of low-hanging fruit to improve performance and efficiency. Basically we just need to set up a query log, do every action possible on the forum, and then inspect each query individually and optimize where it's not too hard.

Also add SQL indexes (we don't have any currently apart from primary keys).

There are some more interesting cases too, like:

  • Flarum\Forum\Actions\IndexAction line 58: we send an API request to get the current user, but we already have the user data; we just want it formatted in an API response. We could technically call a serializer directly, but then we don't quite get all of the response features (like sideloaded data), and also Forum would depend on Api. I don't think we can optimize this case.
  • Flarum\Core\Repositories\EloquentPostRepository (see big comment at top of file)
@tobscure tobscure added the backend label Jun 20, 2015
@tobscure tobscure added this to the 1.0 Beta 1 milestone Jun 25, 2015
@tobscure tobscure changed the title from Audit SQL queries to SQL query audit + add database indexes Aug 27, 2015
@tobscure
Member

No SQL query optimization has been done on Flarum yet, so there is probably quite a bit of room for performance gains here.

What needs to be done:

  • Create a "Debug" extension which attaches the page load time and a list of queries that were run to each API response (under the "meta" key).
  • Generate a large dataset (hundreds of thousands of discussions/posts) to test with.
  • Analyse the queries that are run on requests to each API endpoint and work out ways to reduce/optimize them – especially making sure there are no n+1 situations.
  • Add database indexes where appropriate.
@tobscure tobscure removed this from the 1.0 Beta 1 milestone Aug 27, 2015
@tobscure tobscure referenced this issue Aug 28, 2015
Open

v0.1.0 roadmap #74

19 of 53 tasks complete
@tobscure tobscure added Performance Cleanup and removed backend labels Aug 28, 2015
@michaelmior

I'm going to suggest https://github.com/maximebf/php-debugbar for this purpose as it's easy to use and can quickly collect a lot of helpful data.

@tobscure
Member
tobscure commented Sep 1, 2015

Thanks, will check that out.

@justjavac justjavac referenced this issue in justjavac/Flarum Sep 7, 2015
Open

Flarum v0.1.0 开发路线图 #3

10 of 53 tasks complete
@tobscure tobscure added the Backend label Sep 16, 2015
@younes0
younes0 commented Sep 16, 2015

if it can help: Laravel's package of maximebf/debugbar: https://github.com/barryvdh/laravel-debugbar

@franzliedke franzliedke modified the milestone: 0.1.0 Apr 7, 2016
@believer-ufa

Hello! You can use Mysql profiling builted functions, for example.

@BartVB
BartVB commented Jun 14, 2016 edited

I would like to suggest another course of action. Analysing queries is fun stuff, but this mostly results in theoretical optimisation strategies. If you really want to improve performance it makes more sense to test with a database with thousands of users and millions of posts. This will make it much (much) easier to spot bottlenecks and you'll spot actual problems instead of indices which are missing only in theory.

Creating this database can be done with a fixtures script or by converting an existing large forum. Advantage of fixtures is that this is trivial to share with other developers and it's easier to adapt to the specs of your development environment. The main advantage of converting an existing forum is that this results in a very realistic database.

@franzliedke
Member

@BartVB Good thinking - we can integrate this when we finally get to building the migration tool.

Still, there are some places where indices are clearly needed for basic filtering tasks, if I'm not mistaken...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment