Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Device Defender demo to send custom metrics #1547

Merged
merged 27 commits into from
Feb 24, 2021

Conversation

aggarw13
Copy link
Contributor

@aggarw13 aggarw13 commented Feb 18, 2021

This PR updates the device defender demo to demonstrate sending of custom metrics from a Linux platform to the AWS IoT Device Defender service. For more information on device-side custom metrics, refer to the AWS docs: https://docs.aws.amazon.com/iot/latest/developerguide/dd-detect-custom-metrics.html.

This PR makes the following changes:

  1. Add support for collecting custom metrics for CPU usage time data (obtained from /proc/uptime) and System Memory data (obtained from /proc/meminfo)
  2. Update utility for generating JSON report to support addition of custom metrics of string-list and number-list types for the 2 new metrics.
  3. Update demo file to obtain the CPU usage time and memory data metrics from the support and add them to JSON report before sending to the Devide Defender service.
  4. Update submodule for Device Defender library to access JSON report string literals relating to custom metrics.
  5. Hygiene improvement of JSON report builder to use key names from Defender library instead of have them hard-coded in internal implementation so that either of short or long key names are used depending on user configuration of library.

Following is an example JSON report created from running the demo (Notice the custom metric objects of cpu-usage and memory-info at the end of the report):

{"header": {"report_id": 1613686957,"version": "1.0"},"metrics": {"listening_tcp_ports": {"ports": [{"port": 53},{"port": 53},{"port": 53},{"port": 22},{"port": 1883},{"port": 35853}],"total": 6},"listening_udp_ports": {"ports": [],"total": 0},"network_stats": {"bytes_in": 1139694747,"bytes_out": 2397302807,"packets_in": 67562100,"packets_out": 55610075},"tcp_connections": {"established_connections": {"connections": [{"local_port": 50094,"remote_addr": "127.0.0.1:35853"},{"local_port": 35853,"remote_addr": "127.0.0.1:50096"},{"local_port": 35853,"remote_addr": "127.0.0.1:50094"},{"local_port": 50096,"remote_addr": "127.0.0.1:35853"},{"local_port": 49044,"remote_addr": "<redacted>"},{"local_port": 22,"remote_addr": "<redacted>"},{"local_port": 41016,"remote_addr": "<redacted>"},{"local_port": 44402,"remote_addr": "<redacted>"}],"total": 8}}},"custom_metrics":{"cpu-usage": [{"number_list": [1348437120,1208020352]}],"memory-info": [{"string_list": ["8150792kB","7011132kB"]}]}}

@aggarw13 aggarw13 force-pushed the defender-demo/add-custom-metrics branch from a5b834d to 382cd8b Compare February 19, 2021 21:53
@aggarw13 aggarw13 force-pushed the defender-demo/add-custom-metrics branch from 61121e0 to c132c55 Compare February 19, 2021 22:02
demos/defender/defender_demo_json/defender_demo.c Outdated Show resolved Hide resolved
demos/defender/defender_demo_json/defender_demo.c Outdated Show resolved Hide resolved
demos/defender/defender_demo_json/defender_demo.c Outdated Show resolved Hide resolved
demos/defender/defender_demo_json/defender_demo.c Outdated Show resolved Hide resolved
demos/defender/defender_demo_json/demo_config.h Outdated Show resolved Hide resolved
demos/defender/defender_demo_json/report_builder.c Outdated Show resolved Hide resolved
"}"

#define JSON_REPORT_CUSTOM_METRIC_START \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can the following 4 macros be combined and thereby written in one snprintf call as opposed to 5 snprintf calls?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, done. I was trying to separate out the formats of number-list and string-list through macros to make the different formats easier for the reader to see.

@@ -479,6 +670,41 @@ ReportBuilderStatus_t GenerateJsonReport( char * pBuffer,
}
}

/* Write custom metrics. */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Combining the 5 macros into one could reduce all this code into:

#define JSON_REPORT_FORMAT_PART5    \
    "\"custom_metrics\":{"          \
    "\"cpu-usage\":["               \
    "{"                             \
    "\"number_list\":["             \
    "%lu,"                          \
    "%lu"                           \
    "]"                             \
    "}"                             \
    "],"                            \
    "\"memory-info\":["             \
    "{"                             \
    "\"number_list\":["             \
    "%u,"                           \
    "%u"                            \
    "]}"                            \
    "]"                             \
    "}"                             \
    /* Write custom metrics. */
    if( status == ReportBuilderSuccess )
    {
        charactersWritten = snprintf( pCurrentWritePos,
                                      remainingBufferLength,
                                      JSON_REPORT_FORMAT_PART5,
                                      pMetrics->pCustomMetrics->cpuUsageStats.upTime,
                                      pMetrics->pCustomMetrics->cpuUsageStats.idleTime,
                                      pMetrics->pCustomMetrics->memoryStats.totalMemory,
                                      pMetrics->pCustomMetrics->memoryStats.availableMemory );

         if( !SNPRINTF_SUCCESS( charactersWritten, remainingBufferLength ) )
        {
            LogError( ( "Failed to write part 5." ) );
            status = ReportBuilderBufferTooSmall;
        }
        else
        {
            remainingBufferLength -= charactersWritten;
            pCurrentWritePos += charactersWritten;
        }
    }

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

demos/defender/defender_demo_json/report_builder.h Outdated Show resolved Hide resolved
demos/defender/defender_demo_json/report_builder.h Outdated Show resolved Hide resolved
aggarw13 and others added 2 commits February 23, 2021 10:20
Co-authored-by: Gaurav-Aggarwal-AWS <33462878+aggarg@users.noreply.github.com>
@aggarw13 aggarw13 force-pushed the defender-demo/add-custom-metrics branch from 7a08223 to 469c03e Compare February 23, 2021 23:01
CHANGELOG.md Outdated Show resolved Hide resolved
demos/defender/defender_demo_json/defender_demo.c Outdated Show resolved Hide resolved
demos/defender/defender_demo_json/defender_demo.c Outdated Show resolved Hide resolved
demos/defender/defender_demo_json/defender_demo.c Outdated Show resolved Hide resolved
demos/defender/defender_demo_json/demo_config.h Outdated Show resolved Hide resolved
{
while( ( fgets( &( lineBuffer[ 0 ] ), MAX_LINE_LENGTH, fileHandle ) != NULL ) )
{
lineNumber++;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only other function that has lineNumber is GetEstablishedConnections which uses it to skip the first line.

aggarw13 and others added 4 commits February 23, 2021 18:15
Co-authored-by: Gaurav-Aggarwal-AWS <33462878+aggarg@users.noreply.github.com>
…3/aws-iot-device-sdk-embedded-C into defender-demo/add-custom-metrics
aggarg
aggarg previously approved these changes Feb 24, 2021
@aggarw13 aggarw13 merged commit d410219 into aws:main Feb 24, 2021
@aggarw13 aggarw13 deleted the defender-demo/add-custom-metrics branch February 24, 2021 06:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants