Monitoring Indexing

[ethankhall]

With the release of 3.10.x and the changes to the environment variables. We weren't indexing for about 13 days.

I looked at zoekt-* and they don't seem to have a way to scrape data into prometheus. Is there another way to verify that indexes have been updated?

I think really like to know how old an index is per repo so that I can monitor on ones we know get a lot of commits.

[beyang]

@uwedeportivo @slimsag this seems like a good candidate for a Grafana dashboard. Other customers have requested the following in the past:

• Repository update latency (when was the last time repo contents were synced from GitHub, across all repos)
• Index latency (when the last time repos were indexed, across all repos)

@sourcegraph/core-services is there an easy way to fetch this data across all repos?

[unknwon]

For Repository update latency, the updated_at column in public.repo table looks like for this purpose. Not sure about Index latency.

[keegancsmith]

What do you mean by "across all repos". Do you want this information per repo? What would it look like for an aggregate number?

Both last update and index time information is available at a repositories settings page. The index time will be the last time we changed the index. However, we more than likely would of checked if the index needed updating since then and decided there was no work to do. So this number can be old, depending on how often HEAD changes for the repo.

For prometheus here is a search to all the potentially relevant instrumentation https://sourcegraph.com/search?q=repo:%5Egithub%5C.com/sourcegraph/%28sourcegraph%7Czoekt%29%24+f:%28gitserver%7Crepo-updater%7Cindex%29+%28promauto%7CMustRegister%29&patternType=regexp#5

For indexing we have a histogram on index durations, with the label fail or success. This is likely quite useful for what you want: https://sourcegraph.com/github.com/sourcegraph/zoekt/-/blob/cmd/zoekt-sourcegraph-indexserver/main.go#L51-55

For gitserver updates you probably want https://sourcegraph.com/github.com/sourcegraph/sourcegraph@4a0e5e0620268cf625cee480fd131d34fc317048/-/blob/cmd/repo-updater/repos/metrics.go#L57-86 This may not capture some updates we do (for example whenw e update due to EnsureRevision). However, this should get you most of the visibility needed to alert.

[keegancsmith]

For Repository update latency, the updated_at column in public.repo table looks like for this purpose. Not sure about Index latency.

FYI updated_at is about metadata, not git contents. So when the API returns new data (eg the repo has changed its name).

[tsenart]

@beyang: We have this Prometheus metrics for repo metadata (not git data) last updates:
https://github.com/sourcegraph/sourcegraph/blob/ceb08fcc86fd3e7e94405aed04c42e4956acc2bb/cmd/repo-updater/repos/metrics.go#L16

[ethankhall]

@keegancsmith I was looking at it, and I don't think zoekt-server has a Prometheus port open by default. I tried 6060 and it wasn't open. Looking at https://github.com/sourcegraph/deploy-sourcegraph-docker/blob/master/deploy-zoekt-indexserver.sh it doesn't have the port listed as something that is open.

[emidoots]

Yeah, we will definitely add Prometheus metrics and alerting for this out-of-the-box. We did an alerting overhaul in 3.11 that will make this easier overall as we standardize health alerting/monitoring on all Sourcegraph instances. I can own this issue & will follow up once I'm back from holiday :)

…

On Thu, Dec 19, 2019, 10:39 PM Ethan Hall ***@***.***> wrote: @keegancsmith <https://github.com/keegancsmith> I was looking at it, and I don't think zoekt-server has a Prometheus port open by default. I tried 6060 and it wasn't open. Looking at https://github.com/sourcegraph/deploy-sourcegraph-docker/blob/master/deploy-zoekt-indexserver.sh it doesn't have the port listed as something that is open. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#55?email_source=notifications&email_token=AAYGWOC6XTMR62UOIRTHD4DQZRLCFA5CNFSM4J4PEMUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHL6EAQ#issuecomment-567796226>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYGWODA2CYZX2QM2G6LVFTQZRLCFANCNFSM4J4PEMUA> .

[ethankhall]

Awesome thanks! Have a great holiday!

[keegancsmith]

@keegancsmith I was looking at it, and I don't think zoekt-server has a Prometheus port open by default. I tried 6060 and it wasn't open. Looking at https://github.com/sourcegraph/deploy-sourcegraph-docker/blob/master/deploy-zoekt-indexserver.sh it doesn't have the port listed as something that is open.

Hi @ethankhall the zoekt-indexserver listens on port 6072. @slimsag can you update deploy-sourcegraph-docker to expose that port. Note: This port is purely for debugging purposes, and shouldn't be exposed more widely.