Feature Request: Use Aperf as a monitoring tool for EC2 hosts

[molenin-moodys]

TL;DR

I want to replace some metrics currently produced by the CloudWatch Agent with aperf records stored in S3. I’m not a performance-optimization expert, so I may be missing some common best practices.

Aperf seems to be designed mainly for manual performance analysis, so it lacks some features I expected. I also couldn’t find certain data that’s important for my intended use cases.

Proposed usage

Today, we rely on the CloudWatch Agent for host-level performance metrics. I’m exploring whether aperf could be used instead to collect more detailed, low-level performance data and store it in S3 for later analysis. This would allow us to have fine-grained visibility into host behavior, without being limited to the 1-minute or 1-hour granularity of CloudWatch metrics.

My goal is to collect performance records for the entire lifetime of each EC2 host running in AWS ECS.

Because I don’t control when a host is terminated, my current idea is:

• Run a cron job every minute on each host.
• Each run:
  • Records performance data for 60 seconds using aperf.
  • Uploads the resulting record to S3.
• This should minimize gaps in host-level performance history and make sure we don’t lose data when instances are terminated unexpectedly.

Questions / concerns

1. Is this usage pattern aligned with how aperf is intended to be used?
   If not, could you please share any concerns you may have?
2. Are there any known limitations when using aperf in this way (for example, overhead, storage, or data quality)?
   Based on my estimates, the record size should remain reasonable.

Features I would like to have as part of aperf

1. Currently, records do not include absolute timestamps. I would like to have the ability to merge multiple records into a single larger record, even if they overlap.
2. I would appreciate native support for using S3 as the output location for records. This would remove the need to upload them manually using awscli.
3. If aperf record could run in the background and periodically upload partial records to S3, I would not need to rely on cron jobs.

Data that I cannot find in the report and would like to have in the future

1. Network usage speed - at the moment I can only find packet counters, but I would like to see actual throughput values.
2. Disk usage speed - we are using EC2 instances with NVMe disks, and disk performance is an important factor for us.
3. Speed of communication with S3 - we upload and download many gigabytes of data to and from S3, so it would be very helpful to have a separate graph for this traffic.
4. Free and used disk space for each mount point.

I’d really appreciate your feedback on this request and any guidance on possible implementations. I’m happy to contribute and help implement these features through PRs.

[lancelui-amzn]

Hello, thanks for the request!

To answer your questions:

1. Currently, APerf is does not have the features to support this usage pattern easily, demonstrated by the setup needed for your use. However, we have received several similar feature requests for continuous profiling and are currently evaluating different approaches to do so. We will keep this issue updated on our progress.
2. One potential limitation is in the setup time for PMU counters. Currently, on larger instances with many cpus there may be a large delay (>5s) between running aperf record and the first sample being collected, which would lead to gaps in data. This is caused by needing to open a file descriptor for every counter on every cpu. We plan on optimizing this to reduce the total setup time, but rerunning record should be avoided in a continuous profiling use case. Another immediate limitation would be in visualization. Although you may view each record separately in one report, APerf does not support combining sequential runs into one run. This makes visualizing and analyzing data in the report difficult for investigating time periods that span multiple 60 second records. We do have plans for a feature that allows users to convert record data into csv/json/etc for easier processing outside of the report (which would also require absolute timestamps).

If you are interested in contributing to APerf, you can refer to https://github.com/aws/aperf/blob/main/DEVELOPMENT.md for a guide on adding new data.

• Network/Disk throughput: This can be calculated from existing data by dividing byte counters by sampling intervals.
• It will be challenging to isolate S3 traffic from general network traffic without application-level instrumentation.
• Free/used space per mount point would be valuable and may be straightforward to implement.

[CongkaiTan]

For network throughput APerf has the network stats metrics IpExt:InOctets and IpExt:OutOctets, which show the total number of bytes received and sent per interval at the IP layer.

For disk throughput APerf has the disk stats metrics sectors_read and sectors_written , which show the number of 512-byte sectors read and written per interval.

I think one quick improvement we can do is to perform some extra processing to turn them into throughput metrics that are more commonly known, and probably they can always be ordered at the top of the page.