Dolphin release candidate testing - Defender Product Assessment Sanity Testing

[nanda-katikaneni]

💡 Summary

In preparation of releasing the Dolphin or v0.3.0 of ScubaGear code, conduct sanity testing of the Defender product. Objective and Scope of the task are provided below.

Objectives:

1. There are no regression issues in the Defender product assessments with Dolphin Release
2. Additional sanity testing to ensure that: each policy assessment result is shown in the report, assessment works against all available tenants (G5/E5, G3/E3)
3. Defender product assessment works both in interactive and non-interactive (service principal) modes.

Scope:

1. Detailed functional testing of each policy statement result is out of scope
2. Consistent results between interactive/non-interactive modes and no operational issues in running the test against all tenant types are within the scope.

Motivation and context

This would be useful to ensure that Dolphin release is stable

Implementation notes

Before the test, ensure that test user has minimum user role on a given tenant to assess Defender (look into README). Then, execute the Defender product assessment on all available tenants – first in interactive mode and then in non-interactive mode. After the test verify the following

1. Verify that all tests run without errors and results reports are generated. Each policy has a result (no empty results)
2. Ensure that there are no regressions from Coral release – for the tested tenant compare the result report from current assessment against saved Coral release results - ensure that any different result is consistent with code change (provide a detailed explanation on any observed diff in results)

Acceptance criteria

1. Defender product assessment works in both interactive and non-interactive mode against G5, E5, G3 and E3 tenants.
2. There are no crashes and/or empty results
3. Results are consistent with Coral release assessment results - any diff is consistent with code changes.

[ssatyapal123]

• GCC G5 Tenant Interactive Login | 1 Error for Policy 2.9 (Known Issue: Known issue: Fix EXO deprecated alert policies in MS.EXO.16.1 #29)
• GCC G5 Tenant Non-Interactive Login | 1 Error for Policy 2.9 (Known issue: Known issue: Fix EXO deprecated alert policies in MS.EXO.16.1 #29)
• Commercial E5 Tenant Interactive Login | 24 Errors
• Commercial E3 Tenant Interactive Login | 20 Errors

G5 results are as expected based on the policy settings, with one error due to known issue. MSFT deprecated the two alert policies the Rego is checking for. As we called out the alert policies in the baseline, this requires a baseline update and a rego update. This will addressed during Emerald.

[ssatyapal123]

To confirm policy changes are properly working, ran a quick test on two policies on Commercial E5 tenant:

• Changed Policy 2.2 (added data types ITIN and SSN to the Default policy and followed the implementation steps) however this policy is still showing a failure. Believe this is not a problem with Dolphin but with the tenant configuration.

• Changed Policy 2.3 (enabled the common attachments filter in the threat policy for email & collaboration) the Tenant is now showing a pass.

Before:

After:

[nanda-katikaneni]

After further discussions with Shanti, seems like the Policy 2.2 result for E5 is same in v0.2.1 as well. So there is no regression with Dolphin. On 'why it is failing when it is expected to pass' - it will be investigated further and if needed a separate issue will be opened for a future release.