indexed-search: more prometheus metrics for search

[keegancsmith]

Add metrics for indexed search:

• Gauges for running
• Histogram for duration of search
• Counter for failed
• A counter for each stat https://github.com/google/zoekt/blob/abd3a189031e9f36a0c12961f208b2a1a1f3d412/api.go#L92-L134

Part of #9001

[keegancsmith]

I upstreamed this at https://gerrit-review.googlesource.com/c/zoekt/+/259177
Already landed in our fork at sourcegraph/zoekt@eda08d3

Example raw metrics:

# HELP zoekt_list_running The number of concurrent list requests running
# TYPE zoekt_list_running gauge
zoekt_list_running 0
# HELP zoekt_list_shard_running The number of concurrent list requests in a shard running
# TYPE zoekt_list_shard_running gauge
zoekt_list_shard_running 0
# HELP zoekt_search_content_loaded_bytes_total Total amount of I/O for reading contents
# TYPE zoekt_search_content_loaded_bytes_total counter
zoekt_search_content_loaded_bytes_total 2.1132263e+07
# HELP zoekt_search_crashes_total Total number of search shards that had a crash
# TYPE zoekt_search_crashes_total counter
zoekt_search_crashes_total 0
# HELP zoekt_search_duration_seconds The duration a search request took in seconds
# TYPE zoekt_search_duration_seconds histogram
zoekt_search_duration_seconds_bucket{le="0.005"} 9
zoekt_search_duration_seconds_bucket{le="0.01"} 10
zoekt_search_duration_seconds_bucket{le="0.025"} 11
zoekt_search_duration_seconds_bucket{le="0.05"} 14
zoekt_search_duration_seconds_bucket{le="0.1"} 14
zoekt_search_duration_seconds_bucket{le="0.25"} 14
zoekt_search_duration_seconds_bucket{le="0.5"} 14
zoekt_search_duration_seconds_bucket{le="1"} 14
zoekt_search_duration_seconds_bucket{le="2.5"} 14
zoekt_search_duration_seconds_bucket{le="5"} 14
zoekt_search_duration_seconds_bucket{le="10"} 14
zoekt_search_duration_seconds_bucket{le="+Inf"} 14
zoekt_search_duration_seconds_sum 0.150681458
zoekt_search_duration_seconds_count 14
# HELP zoekt_search_failed_total The total number of search requests that failed
# TYPE zoekt_search_failed_total counter
zoekt_search_failed_total 0
# HELP zoekt_search_file_count_total Total number of files containing a match
# TYPE zoekt_search_file_count_total counter
zoekt_search_file_count_total 294
# HELP zoekt_search_files_considered_total Total files that we evaluated. Equivalent to files for which all atom matches (including negations) evaluated to true
# TYPE zoekt_search_files_considered_total counter
zoekt_search_files_considered_total 311
# HELP zoekt_search_files_loaded_total Total files for which we loaded file content to verify substring matches
# TYPE zoekt_search_files_loaded_total counter
zoekt_search_files_loaded_total 294
# HELP zoekt_search_files_skipped_total Total candidate files whose contents weren't examined because we gathered enough matches
# TYPE zoekt_search_files_skipped_total counter
zoekt_search_files_skipped_total 39665
# HELP zoekt_search_index_loaded_bytes_total Total amount of I/O for reading from index
# TYPE zoekt_search_index_loaded_bytes_total counter
zoekt_search_index_loaded_bytes_total 119681
# HELP zoekt_search_match_count_total Total number of non-overlapping matches
# TYPE zoekt_search_match_count_total counter
zoekt_search_match_count_total 41188
# HELP zoekt_search_ngram_matches_total Total number of candidate matches as a result of searching ngrams
# TYPE zoekt_search_ngram_matches_total counter
zoekt_search_ngram_matches_total 2413
# HELP zoekt_search_running The number of concurrent search requests running
# TYPE zoekt_search_running gauge
zoekt_search_running 0
# HELP zoekt_search_shard_files_considered_total Total number of files in shards that we considered
# TYPE zoekt_search_shard_files_considered_total counter
zoekt_search_shard_files_considered_total 75572
# HELP zoekt_search_shard_running The number of concurrent search requests in a shard running
# TYPE zoekt_search_shard_running gauge
zoekt_search_shard_running 0
# HELP zoekt_search_shards_skipped_total Total shards that we did not process because a query was canceled
# TYPE zoekt_search_shards_skipped_total counter
zoekt_search_shards_skipped_total 0
# HELP zoekt_shards_load_failed_total The total number of shard loads that failed
# TYPE zoekt_shards_load_failed_total counter
zoekt_shards_load_failed_total 0
# HELP zoekt_shards_loaded The number of shards currently loaded
# TYPE zoekt_shards_loaded gauge
zoekt_shards_loaded 4
# HELP zoekt_shards_loaded_total The total number of shards loaded
# TYPE zoekt_shards_loaded_total counter
zoekt_shards_loaded_total 4

[keegancsmith]

Additionally gonna add some debug logs in. This should also help admins understand what zoekt is doing. sourcegraph/zoekt#40