Greengrass DeviceDefender: weird behaviour with Bytes In and Bytes Out

[olivierpichon]

Describe the bug
Can't seem to be able to configure AWS Device Defender Security Profile with the data Bytes In / Bytes Out that Greengrass device defender sends

To Reproduce
Greengrass component setup using the following

    "aws.greengrass.DeviceDefender": {
      "componentVersion": "3.0.0",
      "configurationUpdate": {
        "merge": "{\"SampleIntervalSeconds\":300,\"UseInstaller\":true}"
      }
    }

Security Profile with following criteria (here as IaC, can be created manually in AWS Console):

        {
          name: "BytesIn",
          criteria: {
            comparisonOperator: "less-than-equals",
            consecutiveDatapointsToAlarm: 1,
            consecutiveDatapointsToClear: 3,
            durationSeconds: 600,
            value: {
              count: "100000",
            },
          },
          metric: "aws:all-bytes-in",
          suppressAlerts: true,
        }

Expected behavior
Between two data points of 5 min interval, we never have more than 100000 Bytes sent. It should not alarm.

Actual behavior
Instead it alarms, most probably because instead of making the difference in bytes between two data points, what's being pushed is a cumulative BytesIn values which never cease to increase over time?

Should this component be changed to send delta between two datapoints to be compatible with a Security Profile? Or otherwise how can we use the BytesIn BytesOut data as pushed by this component in a Device Defender Security Profile?

Thanks a lot for your time!

[MikeDombo]

Hello Olivier,
Can you use the AWS IoT Console (or other MQTT client) to subscribe to the topic: $aws/things/{thing_name}/defender/metrics/json, and replace {thing_name} with the thing name which is having this problem?

I'd like to know what metric values are actually being published from the device, and if the metrics are actually making it into AWS IoT.

aws:all-bytes-in metric is the total amount of incoming data to that device from any source. How do you know that the value is always less than 100000? What are you using to measure that metric?

[olivierpichon]

Hi Mike,

Thanks a lot for your answer, really kind of you to take the time!

Can you use the AWS IoT Console (or other MQTT client) to subscribe to the topic: $aws/things/{thing_name}/defender/metrics/json, and replace {thing_name} with the thing name which is having this problem?

Unfortunately it does not look like I can do this. When it comes to Defender related topics, it does not look like we can receive any of them: https://docs.aws.amazon.com/iot/latest/developerguide/reserved-topics.html#reserved-topics-device-defender

We can only subscribe to $aws/things/thingName/defender/metrics/payload-format/accepted (notice the trailing /accepted) but are not allowed to receive anything. I am only honestly not quite sure what is the purpose of allowing subscribe if we do not allow receive. But this is another topic (no pun intended :)). Here are some references when one might want to do subscribe without receive which does not seem to be our case here: https://stackoverflow.com/questions/56902509/aws-iot-policy-subscribe-vs-receive-actions . Maybe do you have another understanding around this?

When it comes to making sure metrics are actually making it into AWS IoT, I do have this certainty. I am able to navigate to Defend -> Detect -> Security Profile -> Defender Metrics -> Bytes In.

The resulting graph looks like this:

As you can see Bytes In is steadily growing. From one data point to another (at 5 min Interval), there is never more than a 50,000 Bytes difference. Our IoT device is supposed to push and pull data at a steady rate (the graph confirms it is indeed the case).

We would like to setup a defender rule in our security profile around these metrics (BytesIn and Out). Essentially saying that between two data points (5 min interval) there should never be more than 100,000 Bytes difference (bumping a bit the previous 50,000 value). How can we do this with our security profile rule?

Our understanding was that the rule we talked about earlier would do just this:

But it alarms unfortunately. And when it does, it shows the total number of BytesIn:

So it does look like the rules takes into consideration the total number of BytesIn as opposed to the difference between two data points.

I understand this above description is quite long but a sum up is quite easy: the Greengrass Device Defender pushes BytesIn and BytesOut metrics which are "ever-growing" metrics. Metrics that are sent from Greemgrass Device Defender are supposed to be used to setup rules when creating a Security Profile. How can we set up a rule based on these BytesIn / BytesOut metrics? In other words how can we setup a threashold around something that never stops growing? It feels like the rule should allow to do the difference between two data points. The fact that the rule states a duration tends to indicate it is the intent. But it does not look like it is working as we expect: eg the alarm in our case should not be triggered.

Thanks again for your help Mike!

[olivierpichon]

Update: after a bit more investigations, it looks like the number shown in the alarm above (34104770) is the sum of two Bytes In metric that are sent within 5 min interval.

So if the rule itself for "aws:all-bytes-in" (in the security profile) is doing a sum of what's being sent during the interval Duration, then that means that the rule anticipates the metrics to send a delta of Bytes In rather than a cumulative value we would think.

Otherwise, if we are mistaken, then how can we use the Bytes In / Bytes Out metrics as it is currently being sent by this component into a Device Defender Security Profile?

Thanks!

[MikeDombo]

Hello again,
The Greengrass device defender component doesn't really do much of anything as you will see if you have a look at the code in this repo. This component relies on the device defender SDK available at https://github.com/aws-samples/aws-iot-device-defender-agent-sdk-python. So I would suggest that you please reach out to them in their support channels. They have much better knowledge about how device defender works and so they are much better equipped to help you out.

Cheers,
Michael

[olivierpichon]

Hi again Michael,

Thanks for that. As indicated I contacted as well the device-defender-agent-sdk team.

But even if this repo is doing very little, the bug could still sit in this one. I am not a Python expert, but from what I can see, this seems to be the case: the bug seems to be in this very repo. The Python SDK is exactly doing what I would anticipate here: https://github.com/aws-samples/aws-iot-device-defender-agent-sdk-python/blob/a3d27c51311e12625bd0b774e8b730ead3bc515d/AWSIoTDeviceDefenderAgentSDK/metrics.py#L136-L151

ie: they are storing the last interface stats and are making a diff between the current stat and the last one. If there is no previous stat, they just send the stat (without doing the delta).

However it does not look like this module can ever make use of it if I understand well. Because we launch a thread that we repeat at the given interval here.

Everytime we relaunch a thread, we are re-instantiating a new metrics_collector here. When we do this, we are discarding the possibility to keep the past stats.

Would you be able to take a look please?

Thank you!

[MikeDombo]

Thank you, you are quite right. After looking at our code and the code in the device defender SDK it seems that any delta based metrics will not be emitted correctly.

We will work to address this quickly. Until then, you could try using the 2.0.x version series of the device defender component.

[olivierpichon]

Thanks for your help Michael!

[MikeDombo]

Thank you again for reporting this issue. We have updated the code and released the fix as version 3.0.1, it is publicly available in all regions now.