Move system symbol lookup into a postgres database.

[mitsuhiko]

This is still work in progress but the idea is to move all system symbol lookup into the database. Right now it uses the main database but i want to split this up and maybe later move it into a webservice.

• verify database layout
• add support for fuzzy matching on binaries when uuids are not found
• bump symsynd requirement

Potential improvements for later are to look up multiple symbols at once but that would work best with a stored procedure.

[codecov-io]

Current coverage is 82.31%

Merging #3022 into master will decrease coverage by -0.33% as of cc45369

@@            master   #3022   diff @@
======================================
  Files          931     932     +1
  Stmts        36397   36643   +246
  Branches         0       0       
  Methods          0       0       
======================================
+ Hit          30079   30162    +83
  Partial          0       0       
- Missed        6318    6481   +163

Review entire Coverage Diff as of cc45369

Powered by Codecov. Updated on successful CI builds.

[mattrobenolt]

Isn't this O(n) for number of frames? with a RT to database? Possibly two queries per frame.

[mitsuhiko]

I will try to play with splitting the sentry_dsymsymbol into two tables because lots of symbols are reused but at different addresses. That might cut down storage quite a bit.

[mitsuhiko]

Splitting up the symbols into just the strings and then referencing them from elsewhere does not appear to be a great storage saver. The various indexes get huge in total. I will try a full import over night to see if with more data it improves though.

[mitsuhiko]

@getsentry/api I will aim for getting this in for getsentry by EOW. So probably friday the last. If it's bad, we can easily back it out. The old code runs in addition for now. I will set up a router in a new pull request on getsentry that moves it to a separate DB if we need to.

I want to push this out asap because i need to get some other support in (tvos and code) and i don't want to have to do that with the system symbol code path.

[mitsuhiko]

@getsentry/api This is ready for merge as far as I'm concerend. It does not yet remove the other branch for system symbols so we can slowly upload the preprocessed ones.

I also want to do those things after this is merged:

• if needed route them to a different database for getsentry
• upload the processed symbols to getsentry

[mattrobenolt]

TIL LookupError

[mitsuhiko]

I'm just going to kill the preparing now since that only worked on postgres anyways and just give up with trying to optimize this. Too much of a hassle with also making it idempotent and not really worth the effort. We don't import often enough.

@getsentry/api going to merge and deploy this tomorrow.

[mitsuhiko]

This is ready to merge from where I'm standing.

@getsentry/api does anyone want to do a second look over the migrations before I merge this?