Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add resource utilization warnings #4559

Conversation

vmapetr
Copy link
Contributor

@vmapetr vmapetr commented Dec 11, 2023

This PR adds ADO warnings in case if disc or memory resource utilization exceeds 95%. It also add telemetry for high CPU usage.

Changes:

  • Multiple methods from ResourceMetricsManager were reworked.
  • Separate memory utilization, disk space utilization and CPU utilization monitors were added into ResourceMetricsManager.
  • New monitors from ResourceMetricsManager enabled in TaskRunner.
  • Memory monitor now checks overall system memory utilization instead of memory used by agent process.
  • Agent throws ADO warnings in case of high disc space or memory utilization.
  • Agent sends telemetry warnings in case of high CPU utilization.
  • Strings related to the ADO context moved into localizable resources.
  • Related FF knob EnableResourceUtilizationWarnings is added.

Testing:
Validated on Windows 11 / Ubuntu 22.04 / MacOS 14.0 / CentOS 7 (Docker)
Buddy-tested by @kirill-ivlev

How it was tested:
Agent was built with memory and disk space thresholds set to 99% percent.
image

Related WI: 2127964

@vmapetr vmapetr marked this pull request as ready for review December 11, 2023 20:54
@vmapetr vmapetr requested review from a team as code owners December 11, 2023 20:54
src/Agent.Worker/ResourceMetricsManager.cs Outdated Show resolved Hide resolved
src/Agent.Worker/ResourceMetricsManager.cs Outdated Show resolved Hide resolved
@vmapetr vmapetr changed the title Add resource utilization warnings [WIP]Add resource utilization warnings Dec 14, 2023
@vmapetr vmapetr force-pushed the v-mpetrov/us-2127964-Provide-discoverable-resource-utilization-information branch from e7d85ce to 1b7f6db Compare December 19, 2023 00:44
Copy link

There was an error handling pipeline event b6bb09f4-db76-4f54-b6f1-20e3b9c50093.

@vmapetr vmapetr force-pushed the v-mpetrov/us-2127964-Provide-discoverable-resource-utilization-information branch from a0c5bae to f018a53 Compare December 22, 2023 02:55
@vmapetr vmapetr changed the title [WIP]Add resource utilization warnings Add resource utilization warnings Dec 25, 2023
@vmapetr vmapetr force-pushed the v-mpetrov/us-2127964-Provide-discoverable-resource-utilization-information branch from 5e75f61 to bdee0a6 Compare January 8, 2024 13:45
@max-zaytsev
Copy link
Contributor

/azp run

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@vmapetr vmapetr merged commit 17056c6 into master Jan 23, 2024
10 checks passed
vmapetr added a commit that referenced this pull request Feb 20, 2024
* Add resource utilization warnings and telemetery

* Remove excessive error handling in CPU monitor

* Fix casting for macos memory monitor

* Hide internal monitor methods

* Remove custom exception

* Minor fix in comment

* Fix typos

---------

Co-authored-by: Ivan Duplenskikh <115665590+ivanduplenskikh@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants