Security of AI applications

[maryiat]

Research goal:
Explore how OWASP guidance and AI security practices compare and complement regulations/standards (NIST, EU AI Act, GDPR, ISO?).

Suggested structure:

1. Introduction
   • Why AI security is a priority.
   • Why OWASP and standards/regulations both matter.
2. Foundations
   • Background on OWASP.
   • Overview of AI-specific risks.
   • Overview of regulations & standards (NIST AI RMF, GDPR, EU AI Act).
3. Comparison of Standards and Regulations
   • How does OWASP Top 10 compare to NIST AI RMF and EU AI Act in addressing AI-related application security risks?
   • To what extent can OWASP serve as a bridge between technical security practices and compliance with NIST and EU AI Act requirements?
   • What are the gaps between OWASP guidelines and regulatory frameworks regarding AI-driven applications?
4. Industry & Case Studies
   • What evidence exists in literature on using OWASP as a complementary tool for regulatory audits or certifications in AI systems?
   • How do companies align OWASP practices with ISO certification or EU AI Act compliance in real projects?
5. Conclusion
   • Recap of findings

[maryiat]

My questions are:

• is this even relevant?
• could you recommend any resources to check while preparing this research?
• any examples of how you use any of these regulations and/or frameworks at your project
• any examples of when things went wrong when not following basic security principles while integrating AI

[oren-reshef]

To my modest understanding security treats AI as any other software, there is no explicit reference to AI in most security standards, and when you conduct Threat Modeling on an AI system you will ask similar questions to a non-AI system (with different answers of course).
The EU AI Act is one of the only relevant regulatory requirements, and it's mostly about things other than Security.

[sharovatov]

from what I see, OWASP is "practice driven" while regulations are "compliance driven". So imo there's a gap here, but I'd like to participate in the research and dig deeper.

Additionally, latest OWASP 2025 is still to be released, so maybe AI-related security risks need to be evaluated now.

As @oren-reshef fairly notes, EU AI Act only states that AI apps must be secure, which is far from helpful. I wonder, how they will assess security of a certain app if there's literally no stated requirements (or I'm not aware if they are stated somewhere).

[sharovatov]

curious cases:

• First Known LLM-Powered Malware
• AI-enabled identity fraud

I generally enjoy reading https://incidentdatabase.ai

[sharovatov]

another curious read https://apiiro.com/blog/4x-velocity-10x-vulnerabilities-ai-coding-assistants-are-shipping-more-risks/

[maryiat]

Resources after a chat with Sara Martinez Giner:
https://www.cvedetails.com/cisa-known-exploited-vulnerabilities/kev-1.html
https://www.youtube.com/watch?v=0zqYNC8n9Po
https://cwe.mitre.org/index.html
https://snyk.io/articles/secure-sdlc/
https://owasp.org/www-project-integration-standards/writeups/owasp_in_sdlc/
https://www.microsoft.com/en-us/securityengineering/sdl/practices
https://github.com/dependabot
https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/

[maryiat]

https://deepeval.com/
https://deepeval.com/tutorials/tutorial-introduction
https://www.promptfoo.dev/docs/category/guides/
https://github.com/anthropics/courses/tree/master/prompt_evaluations/01_intro_to_evals
https://vimeo.com/1023317525

[maryiat]

https://www.linkedin.com/pulse/building-quality-llms-through-testing-observability-craig-risi-0je7f/?trackingId=eiMToRNBSM3TnYwFfhH%2FzQ%3D%3D

[maryiat]

https://attack.mitre.org/

[maryiat]

https://www.zerodayinitiative.com/advisories/published/ samt https://attackerkb.com/

[maryiat]

https://medium.com/digitalfrontiers/sast-dast-and-iast-explained-9324572a5d2b - SAST, DAST, IAST explained