-
-
Notifications
You must be signed in to change notification settings - Fork 651
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Risk-based instead of testability-based levels #1495
Comments
I don't like this current "must be blackbox pentestable" too much, but at least is easy and clear rule. Whatever is the way we define levels, we really need to define them. It should be clear, why some requirements is level 1, another is level 2 and some is level 3. There will be gray areas for sure, but we need some guidelines. |
So the logic behind making it pentestable is, well just that. We wanted to make it so that automated tools could adopt L1 and have scanning profiles etc. the problem with making it just risk-based is that often risk-based is fluffy and handwavy and this standard is not 100% a GRC-based one. There has to be a balance here and this was at least our attempt at doing this. So how would you see this happening with the above in mind? |
ISO/IEC 27034-7:2018 specifies both an "Expected Level of Trust" and "Actual Level of Trust" which aligns to their various risk management standards. |
So @Sjord you are saying move away from testability? |
For me, the levels relate to the security requirements of your application. If you are building a nuclear missile launch system, you would comply to L3. If you are building a tic-tac-toe website, you would comply to L1. However, even a webapp with such low security requirements should do logging, have expiring sessions, and have other security measures that are difficult or impossible to pentest. The attribute whether something is pentestable seems to be independent of security requirement levels. |
Ok, I have recently had conversations with two separate organizations using ASVS for verification. In both cases, it would have been easier for them to start with L1 to be assessed by pen testing and then find a way of verifying L2+ as a stretch goal or a 2nd phase. In both cases, they were very clear that just being able to assess to L1, even as a stepping stone was not sufficient and that even their initial assessment methodology had to also L2, despite the added difficulty. As such, I think I can agree that L1 being pen testable does not really help if no one thinks it is enough. The 5.0 roadmap/plan reinforces this and I think it is clear that this will be the direction. Any further comments from anyone? |
What where their additional test cases considered under L2 @tghosth? |
Can we just rephrase or translate it: "collect low-hanging fruits fast and early" to get at least some feedback. For me it seems that in this situation there is no actual feedback how we (should) group requirements to levels. |
Some L2 requirements are pen testable and the others they are having to think about what questions to ask @cmlh
Basically, we are going to have to assess a risk/prioritization level at some point but probably best to leave that for a later date :) |
I would be tempted to add the ASVS Draft label and leave this issue for now. It is something we will only really be able to do at the end. |
We need to finetune the text at the end, but we need to set correct levels based on some arguments for every requirement and it is not that rare when we have issue opened because current description and levels are not matching. |
I wouldn't think a tic-tac-toe app would require authentication either (at least in a couch co-op setting) and this is what I would consider Level 1 - bare-bones functionality. As soon as sensitive data is introduced (e.g. username/password authentication), the app becomes Level 2. At the very least, I'd advocate for introducing a Level 0 so that Level 1 can be required more often. Example: |
With risk - I think we need to start from question - why and for whom we define them / Who and how uses them? My experience here is level 2 and level 3 from pentesting perspective and it matches the descriptions like @appills described them, so I don't know in practice (again from pen-testing perspective) do anyone order pen-test with level 0 or level 1? Basically "scan me"? I can understand from development point of view those are needed, but not that clear as well. Authentication != senstive data - an application may have username-password authentication, but if it do not collect any PII, then I think it's the same non-sensitive application like application with non-auth. So, if someone knows someone who uses ASVS risk levels somehow measured and validated way, we here are really interested to hear about the experience, please send them here to share their experience. |
I have worked with a couple of clients that were working up to a security certification. There, we first asserted that all (custom developed) software should be at least level 1 (based on advs 4), and then based on a risk assessment increase levels. We did crystal box testing focused on level 1. Since most systems are connected in some way, we needed to establish a base line overall first. So I can see some value in level 0 and 1, just to get started with a landscape. Especially if a level 0 contains stuff that there can’t be any reason not to implement it. |
Thank you for sharing @alamers If we look back to previous versions, maybe the simplicity like in v3.0.1 works best. The main problem with risk definition for v4.0.3 is "completely penetration testable" end especially this part from the opened issue:
Also, I think we need to keep max 3 levels, otherwise we go crazy with setting and discussing correct level for each requirement. There can be addition "level 0 - do whatever you want" which do not require any definition for the requirements. Previous definitions:
ASVS v3.0.1:
ASVS v2.1:
|
Ok. I thought about this a little this morning. I agree with ignoring the pen testable aspect as I discussed here: #1495 (comment) On the other hand, I am not sure I like the "risk based" concept we currently use. We say that L3 requirements are for the most highly sensitive applications but really what we usually mean is that L3 requirements are too complicated or widely scoped for most apps to implement. I also don't think that the levels should describe the applications which will be certified but rather should be focused more specifically on the types of requirements within those levels. As such, I thought about the following:
We would then leave it to app consumers or the market in general to decide which level they think a particular app should comply with. What do people think? |
Josh,
I like your proposal, it’s well thought out.
However, I’d like to consider the direction MASVS is going which is only two levels. One for basics and another for solid security.
And we can’t expect everyone to do every requirement as ASVS is quickly turning into a catalog of all requirements to consider.
My 2 cents.
|
@jmanico unfortunately I think that since we have so many more requirements, that is why we need more levels. |
@tghosth - I don't get what is difference between levels 0 and 1? Do you imagine/plan to set levels from 0 to 3 to every requirement?
If we are not clear with that, we need to say that it is indication, not the rule. |
Level 0 is designed as a gateway to the rest. It includes the obvious/easy/high value requirements but we cannot say it provides sufficient security to be a real level. Level 1 is designed as the minimum level where we can say there is some actual level of security here.
Yes
I agree, this is why we want to clarify the documentation :) |
If the [DNS] "implementation" the application depends on is excluded then we'll need to remove the definition of "pentestable" as its dependent infrastructure can no longer be verified as per #1788 et al. |
In an internal working group meeting, the idea of a Level 0 was not popular. I personally am strongly against the 2 levels approach as I don't think it allows enough granularity and leads to a massive workload to implement simultaneously. I have moved this to a discussion here: |
In Scope for 5.0 and beyond · Issue #1127 · OWASP/ASVS, @jmanico says:
So, let's drop the requirement that conformance to L1 requirements can be determined by a hands-on test.
ASVS/0x03-Using-ASVS.md at master · OWASP/ASVS · GitHub says:
ASVS/0x04-Assessment_and_Certification.md at master · OWASP/ASVS · GitHub says:
The text was updated successfully, but these errors were encountered: