repo-updater: Fix update queue length metric

[ryanslade]

Previously we used calls to Push and Pop which don't appear to be
correct as we observed negative length values.

Instead, we'll monitor the length of the heap itself.

[codecov]

Codecov Report

Merging #17308 (84db64b) into main (08f0293) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main   #17308   +/-   ##
=======================================
  Coverage   52.14%   52.14%           
=======================================
  Files        1712     1712           
  Lines       85441    85441           
  Branches     7596     7596           
=======================================
+ Hits        44554    44555    +1     
- Misses      36972    36973    +1     
+ Partials     3915     3913    -2     

Flag	Coverage Δ		*Carryforward flag
go	51.22% <100.00%> (+<0.01%)	⬆️
integration	30.59% <ø> (ø)		Carriedforward from 08f0293
storybook	30.23% <ø> (ø)		Carriedforward from 08f0293
typescript	54.39% <ø> (ø)		Carriedforward from 08f0293
unit	34.87% <ø> (ø)		Carriedforward from 08f0293

*This pull request uses carry forward flags. Click here to find out more.

Impacted Files	Coverage Δ
internal/repos/scheduler.go	65.34% <100.00%> (ø)
...nal/campaigns/resolvers/changeset_apply_preview.go	58.97% <0.00%> (-0.86%)	⬇️
.../internal/codeintel/resolvers/graphql/locations.go	85.56% <0.00%> (+2.06%)	⬆️

[arussellsaw]

LGTM, but observing negative values seems like we're either panicking & recoivering or not calling Push & Pop correctly, do you have an idea of why we saw negative values?

[ryanslade]

I'm not sure. Push and Pop are part of the heap interface and we don't call them ourselves so I have to assume they're being called correctly.

The value seems to consistently drop over time but at some point it jumps down to zero and then continues "dropping" below zero:

We do also set the value to zero when we stop reading from the update queue but this only called once in our code when the entire scheduler is shutting down.

I'll continue to dig into this as it's also strange how large the update queue is