Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Risk-based instead of testability-based levels #1495

Closed
Sjord opened this issue Jan 3, 2023 · 23 comments
Closed

Risk-based instead of testability-based levels #1495

Sjord opened this issue Jan 3, 2023 · 23 comments
Assignees
Labels
1) Discussion ongoing Issue is opened and assigned but no clear proposal yet requirement level Issue related to requirement levels _5.0 - prep This needs to be addressed to prepare 5.0

Comments

@Sjord
Copy link
Contributor

Sjord commented Jan 3, 2023

In Scope for 5.0 and beyond · Issue #1127 · OWASP/ASVS, @jmanico says:

I think the levels should be RISK-BASED and absolutely not TESTABILITY BASED

So, let's drop the requirement that conformance to L1 requirements can be determined by a hands-on test.

ASVS/0x03-Using-ASVS.md at master · OWASP/ASVS · GitHub says:

ASVS Level 1 is for low assurance levels, and is completely penetration testable.

ASVS/0x04-Assessment_and_Certification.md at master · OWASP/ASVS · GitHub says:

In version 4.0, we decided to make L1 completely penetration testable without access to source code, documentation, or developers.

@jmanico
Copy link
Member

jmanico commented Jan 3, 2023

I agree with this 100%. CC @tghosth and @elarlang

@elarlang
Copy link
Collaborator

elarlang commented Jan 4, 2023

I don't like this current "must be blackbox pentestable" too much, but at least is easy and clear rule.

Whatever is the way we define levels, we really need to define them. It should be clear, why some requirements is level 1, another is level 2 and some is level 3. There will be gray areas for sure, but we need some guidelines.

@elarlang elarlang added 1) Discussion ongoing Issue is opened and assigned but no clear proposal yet Leaders decision Big decisions, like re-structuring or concept changes labels Jan 4, 2023
@danielcuthbert
Copy link
Collaborator

So the logic behind making it pentestable is, well just that. We wanted to make it so that automated tools could adopt L1 and have scanning profiles etc.

the problem with making it just risk-based is that often risk-based is fluffy and handwavy and this standard is not 100% a GRC-based one. There has to be a balance here and this was at least our attempt at doing this. So how would you see this happening with the above in mind?

@cmlh
Copy link
Contributor

cmlh commented Jan 7, 2023

ISO/IEC 27034-7:2018 specifies both an "Expected Level of Trust" and "Actual Level of Trust" which aligns to their various risk management standards.

@tghosth
Copy link
Collaborator

tghosth commented Jan 8, 2023

So @Sjord you are saying move away from testability?

@Sjord
Copy link
Contributor Author

Sjord commented Jan 15, 2023

For me, the levels relate to the security requirements of your application. If you are building a nuclear missile launch system, you would comply to L3. If you are building a tic-tac-toe website, you would comply to L1. However, even a webapp with such low security requirements should do logging, have expiring sessions, and have other security measures that are difficult or impossible to pentest.

The attribute whether something is pentestable seems to be independent of security requirement levels.

@tghosth
Copy link
Collaborator

tghosth commented Mar 21, 2023

Ok, I have recently had conversations with two separate organizations using ASVS for verification. In both cases, it would have been easier for them to start with L1 to be assessed by pen testing and then find a way of verifying L2+ as a stretch goal or a 2nd phase.

In both cases, they were very clear that just being able to assess to L1, even as a stepping stone was not sufficient and that even their initial assessment methodology had to also L2, despite the added difficulty.

As such, I think I can agree that L1 being pen testable does not really help if no one thinks it is enough.

The 5.0 roadmap/plan reinforces this and I think it is clear that this will be the direction.

Any further comments from anyone?

@tghosth tghosth added 2) Awaiting response Awaiting a response from the original poster and removed 1) Discussion ongoing Issue is opened and assigned but no clear proposal yet Leaders decision Big decisions, like re-structuring or concept changes labels Mar 21, 2023
@cmlh
Copy link
Contributor

cmlh commented Mar 21, 2023

In both cases, they were very clear that just being able to assess to L1, even as a
stepping stone was not sufficient and that even their initial assessment
methodology had to also L2, despite the added difficulty.

What where their additional test cases considered under L2 @tghosth?

@elarlang
Copy link
Collaborator

In both cases, they were very clear that just being able to assess to L1, even as a stepping stone was not sufficient and that even their initial assessment methodology had to also L2, despite the added difficulty.

Can we just rephrase or translate it: "collect low-hanging fruits fast and early" to get at least some feedback. For me it seems that in this situation there is no actual feedback how we (should) group requirements to levels.

@tghosth
Copy link
Collaborator

tghosth commented Mar 22, 2023

What where their additional test cases considered under L2

Some L2 requirements are pen testable and the others they are having to think about what questions to ask @cmlh

Can we just rephrase or translate it: "collect low-hanging fruits fast and early" to get at least some feedback. For me it seems that in this situation there is no actual feedback how we (should) group requirements to levels.

Basically, we are going to have to assess a risk/prioritization level at some point but probably best to leave that for a later date :)

@tghosth
Copy link
Collaborator

tghosth commented Mar 22, 2023

I would be tempted to add the ASVS Draft label and leave this issue for now. It is something we will only really be able to do at the end.

@tghosth tghosth added the _5.0 - draft This should be discussed once a 5.0 draft has been prepared. label Jul 10, 2023
@cmlh cmlh mentioned this issue Jul 17, 2023
@elarlang
Copy link
Collaborator

elarlang commented Oct 30, 2023

I would be tempted to add the ASVS Draft label and leave this issue for now. It is something we will only really be able to do at the end.

We need to finetune the text at the end, but we need to set correct levels based on some arguments for every requirement and it is not that rare when we have issue opened because current description and levels are not matching.

@elarlang elarlang added the requirement level Issue related to requirement levels label Oct 30, 2023
@appills
Copy link

appills commented Oct 30, 2023

For me, the levels relate to the security requirements of your application. If you are building a nuclear missile launch system, you would comply to L3. If you are building a tic-tac-toe website, you would comply to L1.

I wouldn't think a tic-tac-toe app would require authentication either (at least in a couch co-op setting) and this is what I would consider Level 1 - bare-bones functionality. As soon as sensitive data is introduced (e.g. username/password authentication), the app becomes Level 2.

At the very least, I'd advocate for introducing a Level 0 so that Level 1 can be required more often.

Example:
Level 0: handles no data sensitive to itself, implements bare-bones security by accident (e.g. logging)
Level 1: handles data sensitive to itself (it has implemented some form of authentication)
Level 2: most apps
Level 3: apps under regulations

@elarlang
Copy link
Collaborator

elarlang commented Nov 5, 2023

With risk - I think we need to start from question - why and for whom we define them / Who and how uses them?

My experience here is level 2 and level 3 from pentesting perspective and it matches the descriptions like @appills described them, so I don't know in practice (again from pen-testing perspective) do anyone order pen-test with level 0 or level 1? Basically "scan me"?

I can understand from development point of view those are needed, but not that clear as well. Authentication != senstive data - an application may have username-password authentication, but if it do not collect any PII, then I think it's the same non-sensitive application like application with non-auth.

So, if someone knows someone who uses ASVS risk levels somehow measured and validated way, we here are really interested to hear about the experience, please send them here to share their experience.

@alamers
Copy link

alamers commented Nov 5, 2023

I have worked with a couple of clients that were working up to a security certification. There, we first asserted that all (custom developed) software should be at least level 1 (based on advs 4), and then based on a risk assessment increase levels. We did crystal box testing focused on level 1. Since most systems are connected in some way, we needed to establish a base line overall first.

So I can see some value in level 0 and 1, just to get started with a landscape. Especially if a level 0 contains stuff that there can’t be any reason not to implement it.

@elarlang
Copy link
Collaborator

elarlang commented Nov 5, 2023

Thank you for sharing @alamers

If we look back to previous versions, maybe the simplicity like in v3.0.1 works best. The main problem with risk definition for v4.0.3 is "completely penetration testable" end especially this part from the opened issue:

In version 4.0, we decided to make L1 completely penetration testable without access to source code, documentation, or developers.

Also, I think we need to keep max 3 levels, otherwise we go crazy with setting and discussing correct level for each requirement. There can be addition "level 0 - do whatever you want" which do not require any definition for the requirements.

Previous definitions:
ASVS v4.0.3:
The Application Security Verification Standard defines three security verification levels, with each level increasing in depth.

  • ASVS Level 1 is for low assurance levels, and is completely penetration testable
  • ASVS Level 2 is for applications that contain sensitive data, which requires protection and is the recommended level for most apps
  • ASVS Level 3 is for the most critical applications - applications that perform high value transactions, contain sensitive medical data, or any application that requires the highest level of trust.

ASVS v3.0.1:

  • ASVS Level 1 is meant for all software.
  • ASVS Level 2 is for applications that contain sensitive data, which requires protection.
  • ASVS Level 3 is for the most critical applications - applications that perform high value transactions, contain sensitive medical data, or any application that requires the highest level of trust.

ASVS v2.1:

  • Level 0 (or Cursory) is an optional certification, indicating that the application has passed some type of verification.
  • Level 1 Opportunistic - An application achieves Level 1 (or Opportunistic) certification if it adequately defends against application security vulnerabilities that are easy to discover.
  • Level 2 Standard - An application achieves Level 2 (or Standard) verification if it also adequately defends against prevalent application security vulnerabilities whose existence poses moderate-to-serious risk.
  • Level 3 Advanced - An application achieves Level 3 (or Advanced) certification if it also adequately defends against all advanced application security vulnerabilities, and also demonstrates principles of good security design.

@tghosth tghosth added _5.0 - prep This needs to be addressed to prepare 5.0 and removed _5.0 - draft This should be discussed once a 5.0 draft has been prepared. labels Nov 8, 2023
@tghosth
Copy link
Collaborator

tghosth commented Nov 8, 2023

Ok. I thought about this a little this morning.

I agree with ignoring the pen testable aspect as I discussed here: #1495 (comment)

On the other hand, I am not sure I like the "risk based" concept we currently use. We say that L3 requirements are for the most highly sensitive applications but really what we usually mean is that L3 requirements are too complicated or widely scoped for most apps to implement.

I also don't think that the levels should describe the applications which will be certified but rather should be focused more specifically on the types of requirements within those levels.

As such, I thought about the following:

  • Level 0 - Early value - These are basic requirements that any application should have and should bring some level of security without significant effort.
  • Level 1 - Minimum Security - These are the minimum security requirements that an application should have in place to prevent common attacks.
  • Level 2 - Standard security - This adds additional requirements to provide additional layers of defence and protect against a wider range of standard attacks.
  • Level 3 - Advanced security - This adds requirements to provide defence against a wider scope of attacks or defences which are more complex to implement.

We would then leave it to app consumers or the market in general to decide which level they think a particular app should comply with.

What do people think?

@jmanico
Copy link
Member

jmanico commented Nov 8, 2023 via email

@tghosth
Copy link
Collaborator

tghosth commented Nov 8, 2023

Josh, I like your proposal, it’s well thought out. However, I’d like to consider the direction MASVS is going which is only two levels. One for basics and another for solid security. And we can’t expect everyone to do every requirement as ASVS is quickly turning into a catalog of all requirements to consider. My 2 cents.

@jmanico unfortunately I think that since we have so many more requirements, that is why we need more levels.

@elarlang
Copy link
Collaborator

elarlang commented Nov 8, 2023

@tghosth - I don't get what is difference between levels 0 and 1? Do you imagine/plan to set levels from 0 to 3 to every requirement?

On the other hand, I am not sure I like the "risk based" concept we currently use. We say that L3 requirements are for the most highly sensitive applications but really what we usually mean is that L3 requirements are too complicated or widely scoped for most apps to implement.

If we are not clear with that, we need to say that it is indication, not the rule.

@tghosth
Copy link
Collaborator

tghosth commented Nov 9, 2023

@tghosth - I don't get what is difference between levels 0 and 1?

Level 0 is designed as a gateway to the rest. It includes the obvious/easy/high value requirements but we cannot say it provides sufficient security to be a real level.

Level 1 is designed as the minimum level where we can say there is some actual level of security here.

Do you imagine/plan to set levels from 0 to 3 to every requirement?

Yes

If we are not clear with that, we need to say that it is indication, not the rule.

I agree, this is why we want to clarify the documentation :)

@elarlang elarlang added 1) Discussion ongoing Issue is opened and assigned but no clear proposal yet and removed 2) Awaiting response Awaiting a response from the original poster labels Nov 11, 2023
@cmlh
Copy link
Contributor

cmlh commented Nov 20, 2023

If the [DNS] "implementation" the application depends on is excluded then we'll need to remove the definition of "pentestable" as its dependent infrastructure can no longer be verified as per #1788 et al.

@tghosth
Copy link
Collaborator

tghosth commented Jan 24, 2024

In an internal working group meeting, the idea of a Level 0 was not popular.

I personally am strongly against the 2 levels approach as I don't think it allows enough granularity and leads to a massive workload to implement simultaneously.

I have moved this to a discussion here:
#1839

@tghosth tghosth closed this as completed Jan 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1) Discussion ongoing Issue is opened and assigned but no clear proposal yet requirement level Issue related to requirement levels _5.0 - prep This needs to be addressed to prepare 5.0
Projects
None yet
Development

No branches or pull requests

8 participants