Ideas for redundant and overlapping tests

[SamErde]

Are there any scenarios where it might be beneficial to supersede tests that are already written in the CIS, CISA, or ORCA test suites?

That might be a hard decision tree to apply consistently to tests, but we can also consider the possibility of combined or improved documentation around the tests?

Here are a few ideas that I would love to hear feedback about from @merill, the @maester365/core-module team, and the community:

• When two or more test suites check the same setting:
  • Create a single helper script and point both tests at the central script instead of writing the same check for both suites.
  • Implement a cache-like capability so the online lookup only happens once, and subsequent tests check the value that has been already pulled.
  • Create a single documentation set that is referenced or linked from the suite-specific tests.
  • Decision: count pass/fail results once or per suite.

Example

Test-MtCisaAuditLog (CISA.MS.EXO.17.1) and Test-MtCisAuditLogSearch (CIS.M365.3.1.1) both check the same configuration item (unified audit log ingestion). See also, PR #1703.

[Mynster9361]

For me the disadvantage would be if they are changing tests as they have done over time for CIS.M365 where a test was depreciated and then a new test was introduced in the place as we saw when updating from v4/5 to v6

I know CIS released a V7 recently with a bunch of updates and i can imagine it would be a pain in the a** to make sure that your change to update a test for 1 framework does not suddenly ruin the logic for the other framework. (hope that makes sense)

The advantage in general would be a lot less code maintenance but tbf you would still need to keep metadata up to date anyways i also think that some of the tests that has already been implemented into Maester as a Maester test might actually belong in a given framework instead so we don't have duplicate test 1 without a framework and 1 within a framework.

The overall issue i see with combining them is when you get to the point where 2 frameworks no longer align you will end up with the same result again having 2 tests 1 for each framework like (CISA,CIS) so i think the best solution is to keep framework tests separate from each other even though they have overlap as it in general makes it easier to update a singular or multiple tests when only looking at 1 framework.

[SamErde]

That makes great sense, @Mynster9361! Thanks for sharing your thoughts.

[merill]

+1 with @Mynster9361 on CISA tests themselves. As those tests can evolve independantly we should try to keep them isolated. I think it's okay to re-use cmdlets but I would avoid simply taking the same test result and passing it.

However I would avoid adding Master tests (mt...) for tests that are already in CIS or CISA.

[Mynster9361]

I agree @merill My suggestion was more to remove the mt test and create the Cis/other test instead like i did for MT.1040 in this PR
#1674

So it is not a binding to a MT but turned into a specific framework test