[For Improvement] Add flag/mark if an ASVS requirement is Pentestable or not

[csfreak92]

I have observed that there are some verification requirements in ASVS L1 that are not pentestable. In our practice, we mark them as security design review related verification requirements and then verify these through documentation, source code, etc with a whitebox approach. In my understanding, ASVS L1 requirements are supposed to be fully pentestable in a blackbox perspective.

I propose for us to add a flag/mark of ASVS requirements whether they are pentestable/testable, and design review for non-testable items if we cannot move those requirements to ASVS L2.

For illustration, ASVS 2.1.10 - Verify that there are no periodic credential rotation or password history requirements.
This requirement has two parts, check if there are periodic credential rotation (forcing users to change passwords every 30, 60, 90 days, etc) and the other one checking for password history requirements (not allowing users to re-use the last 5-10 passwords, etc). The first part is almost untestable in a blackbox perspective, while the second part is totally testable without any help from the organization to tweak things in the backend, etc.

Another examples are ASVS 7.1.1 - Verify that the application does not log credentials or payment details. Session tokens should only be stored in logs in an irreversible, hashed form. and ASVS 7.1.2 - Verify that the application does not log other sensitive data as defined under local privacy laws or relevant security policy. which are both L1 requirements but it needs a whitebox/graybox approach to see those application logs.

[elarlang]

I try to make this issue alive.

So - proposal is to mark/flag all blackbox pentestable requirements. But is it always that clear? With blackbox you may not reach to that part of application (like some admin interfaces) and you can not perform any test there. Maybe it's more test-scope question.

But I agree with example like 7.1.1 it's confusing - with marked L1 ASVS says you need to test it even with blackbox, but you can not (without having vulnerabilities which gives you access to logs). If to "fix" this issue with moving 7.1.1 to level 2, then ASVS could send message: "storing plain-text credentials to logs with level 1 application is ok" which is clearly incorrect as well.

To make everyone's life easier I leave link to levels description here:
https://github.com/OWASP/ASVS/blob/master/4.0/en/0x03-Using-ASVS.md

[jmanico]

I think the issue of wether a issue is testable and by what method is too confusion and is a bit out of scope of ASVS's intention. I prefer to see our levels to be more like NIST SP800-63 levels based on risk and not on "testability". So I'm not a fan of marking items as testable or not, respectfully.

[ThunderSon]

The part about marking things as testable is done by ASVS. It's even mentioned in the link provided by Elar on levels:

ASVS Level 1 is for low assurance levels, and is completely penetration testable

So the question comes, if someone doesn't have a low level of security but isn't penetration testable, is that actually considered L2, or L1 but not testable?

I love the part that ASVS tried to make all L1 testable, but it gives back issues for people doing a design review to actually have a minimum level of assurance.
So then the question would be: If ASVS should be based on risk and testability, why is section 2.4 given a paragraph saying that this of huge importance but not testable, thus is L2?

The above question isn't made to trigger aggressiveness, there's just a misunderstanding in the terms being used and the focus.

[jmanico]

The point I’m making - for future consideration - is that just because it’s easily testable doesn’t say everything about the overall risk, it’s just the likelihood factor at best. I’d rather •totally change• ASVS to have level 1-2-3 be risk based as opposed to testability as the sole factor. I’m planting seeds for consideration. Who cares if something is testable by a scanner? The industry is horrible at security testing as a whole and I do not want to map to dumb scanners for a critical standard. Risk based security is how other standards like SP800-63 (which is MUCH more mature than ASVS) handle their various categories. If we did move in this direction it would be a •massive change• and would require that the paragraphs you mentioned be fully changed. It would not be easy. I plan to fight for this change to risk based security in ASVS 5. It's not an appropriate change for future 4x releases. Respectfully, Jim

[danielcuthbert]

This is a great issue. We know the levels are going to change with the push towards 5.0, and this is something we are taking into account. I'm closing the issue for now and we will revisit this with 5.0 dev work.