Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

codeintel: add dashboard for precide-code-intel-worker total in-progress upload size #39690

Merged
merged 12 commits into from
Aug 11, 2022

Conversation

Strum355
Copy link
Member

Introduces a new panel in the precise-code-intel-worker dashboard for tracking the total in-progress uploads per-instance. This may help us in tracking correlation between in-progress uploads' size and resource usage for better instance sizing.

Follows on from @rafax: I think the guidance was "precise-code-intel-worker needs more than 3x the size of largest LSIF upload", but we don't have a metric that shows the size of attempted uploads - adding a metric for this would be useful.

Test plan

Tested locally, see picture:
image

TODO: see if theres any unit tests we can add

@Strum355 Strum355 added managed-instances Managed instances: https://about.sourcegraph.com/handbook/engineering/distribution/managed team/language-platform-and-navigation labels Jul 29, 2022
@Strum355 Strum355 self-assigned this Jul 29, 2022
@cla-bot cla-bot bot added the cla-signed label Jul 29, 2022
@sourcegraph-bot
Copy link
Contributor

sourcegraph-bot commented Jul 29, 2022

Codenotify: Notifying subscribers in CODENOTIFY files for diff 9d12958...54d3ef4.

Notify File(s)
@beyang enterprise/cmd/precise-code-intel-worker/internal/worker/handler.go
enterprise/cmd/precise-code-intel-worker/internal/worker/worker.go
@bobheadxi monitoring/definitions/shared/codeintel.go
monitoring/definitions/shared/workerutil.go
@efritz enterprise/cmd/frontend/internal/codeintel/httpapi/upload_handler.go
enterprise/cmd/frontend/internal/codeintel/httpapi/upload_handler_multipart.go
enterprise/cmd/frontend/internal/codeintel/httpapi/upload_handler_single.go
enterprise/cmd/frontend/internal/codeintel/httpapi/upload_handler_state.go
enterprise/cmd/frontend/internal/codeintel/httpapi/upload_handler_test.go
enterprise/cmd/precise-code-intel-worker/internal/worker/handler.go
enterprise/cmd/precise-code-intel-worker/internal/worker/worker.go
internal/codeintel/stores/dbstore/uploads.go
internal/codeintel/uploads/internal/store/scan.go
internal/codeintel/uploads/internal/store/store_uploads.go
internal/codeintel/uploads/shared/types.go
lib/codeintel/upload/request.go
lib/codeintel/upload/upload.go
@slimsag monitoring/definitions/shared/codeintel.go
monitoring/definitions/shared/workerutil.go
@sourcegraph/delivery doc/admin/observability/dashboards.md
monitoring/definitions/shared/codeintel.go
monitoring/definitions/shared/workerutil.go

@Strum355 Strum355 force-pushed the nsc/pciw-upload-size-dashboard branch 2 times, most recently from f4183f5 to 7903aaa Compare August 10, 2022 17:41
@Strum355 Strum355 requested a review from efritz August 10, 2022 17:41
Comment on lines +97 to +98
gzipSize := h.getUploadSize(upload.UploadSize)
atomic.AddInt64(&h.budgetRemaining, -gzipSize)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use the uncompressed size for the budget instead in the future?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ultimately yea I think so

@Strum355 Strum355 force-pushed the nsc/pciw-upload-size-dashboard branch from 5190597 to abf19fa Compare August 11, 2022 11:01
@Strum355 Strum355 merged commit 30275c2 into main Aug 11, 2022
@Strum355 Strum355 deleted the nsc/pciw-upload-size-dashboard branch August 11, 2022 20:43
@varungandhi-src varungandhi-src added team/graph Graph Team (previously Code Intel/Language Tools/Language Platform) and removed team/language-platform labels Feb 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-signed managed-instances Managed instances: https://about.sourcegraph.com/handbook/engineering/distribution/managed team/graph Graph Team (previously Code Intel/Language Tools/Language Platform)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants