Fetch invocation history from clickhouse, merge in unfinished builds from mysql #6447
Conversation
jdhollen
changed the title
|
Fully intending to write tests, but hoping to have you all take a look and decide if this is worth it or not. This only improves performance for queries where the user is either:
Short of doing something that lets us skip MySQL entirely, you need some version of this hack. If we don't add an index for overall status, then queries for other sort orders will stay slow--they'll require a full scan of all 30 days or whatever the user is looking at (i don't think it's reasonable to add an index for every sort order). If we add an index and it's still not enough for some orgs, we could make performance better by marking long-disconnected and long-pending builds as "completed" with a janitor task (and flush to clickhouse)--then we could rely the status index to make the MySQL queries for "actually-pending" builds fast, since there shouldn't be that many per org at any point in time. |
| blendedInvocationSearchEnabled = flag.Bool("app.blended_invocation_search_enabled", false, "If true, InvocationSearchService will query clickhouse *and* the regular db.") | ||
| olapInvocationSearchEnabled = flag.Bool("app.olap_invocation_search_enabled", true, "If true, InvocationSearchService will query clickhouse for some queries.") |
There was a problem hiding this comment.
nit: I couldn't figure out the difference between these 2 flags by reading the help strings, maybe the second flag could use more detail (or just have the 1 "blended" flag?)
There was a problem hiding this comment.
Did what I can here to make this clearer (if wordier)
| blendedInvocationSearchEnabled = flag.Bool("app.blended_invocation_search_enabled", false, "If true, InvocationSearchService will query clickhouse *and* the regular db.") | ||
| olapInvocationSearchEnabled = flag.Bool("app.olap_invocation_search_enabled", true, "If true, InvocationSearchService will query clickhouse for some queries.") |
There was a problem hiding this comment.
Did what I can here to make this clearer (if wordier)
jdhollen
changed the title
Related issues: N/A