Agency 2 Pilot: Defender 2.2 potential false "PASS" results

[schrolla]

Agency 2 noted that their DLP is managed outside Defender through another system, however they noted that have policies enabled for credit cards, TIN, SSN and PII for monitoring purposes. When they ran the assessment script the results for the sensitive information came up as "PASS" but are not configured to be blocked when the policy states “the DLP policy SHOULD be set to block sharing sensitive information”.

[schrolla]

After reviewing and testing the scenarios, I believe what happened here is that the agency had a rule in place that did use the block action. However, they had the policy set to test mode. In test mode, blocking actions are not performed even if the rule indicates that it is blocking (BlockAccess is true). So, technically speaking, the assessment is correct since the rule itself is set to block as indicated in the implementation guidance. However, the overall effect is that the policy is NOT blocking sharing of sensitive information even if the rule is set to do so as the policy itself would only generate a notification (at best).

Recommend we add code to the check that validates whether or not the policy is "On" (Mode is Enable) vs. test (TestingWithNotifications) or off and flag the check as failed if not on even if a blocking action is present in the rule. Ideally with feedback that the issue isn't in the rule, but in the policy configuration although that might be trickier to pull off.

[schrolla]

One of the big issues complicating this from a technical level is that the relevant policy bullet is stated as:

"The action for the DLP policy SHOULD be set to block sharing sensitive information with everyone when DLP conditions are met."

However, blocking actions are associated at the rule level, not the policy level, and a policy may contain multiple policies. At the policy level, the only control related to blocking is whether the policy is Turn it on right away, Keep it off, or set in Test It Out First mode.

A follow-on in the future would be to update the baseline policy to have two separate items... the existing item to make sure that rules include blocking actions and a separate SHOULD item that policies with blocking actions should have their mode set to On (as opposed to Off or Test) to ensure rule actions are taken.