feat(asm): add exploit prevention LFI support #8568

christophe-papazian · 2024-03-01T12:59:19Z

First PR to introduce the new exploit prevention feature to ASM.
Add LFI support to Exploit Prevention, and patch mechanism to support more entry points.
This PR is internal only. No new feature is activated, new tests are not enable either for now.

move patch mechanism from IAST to appsec common file to support multiple ASM features with a common patch instrumentation
add support for stack trace report in waf run using metastruct
add new endpoints and skipped unit test for that feature.
add a new exploit prevention setting in asm config set to always False (the feature can't be enabled for now)

This feature needs an unreleased libddwaf version, and was only tested locally. Tests and feature will be activated later.

APPSEC-51853

Checklist

Change(s) are motivated and described in the PR description
Testing strategy is described if automated tests are not included in the PR
Risks are described (performance impact, potential for breakage, maintainability)
Change is maintainable (easy to change, telemetry, documentation)
Library release note guidelines are followed or label changelog/no-changelog is set
Documentation is included (in-code, generated user docs, public corp docs)
Backport labels are set (if applicable)
If this PR changes the public interface, I've notified @DataDog/apm-tees.
If change touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.

Reviewer Checklist

Title is accurate
All changes are related to the pull request's stated goal
Description motivates each change
Avoids breaking API changes
Testing strategy adequately addresses listed risks
Change is maintainable (easy to change, telemetry, documentation)
Release note makes sense to a user of the library
Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
Backport labels are set in a manner that is consistent with the release branch maintenance policy

APPSEC-51853

…xploit_prevention_stack_traces_support

datadog-dd-trace-py-rkomorn · 2024-03-01T13:58:19Z

Datadog Report

Branch report: christophe-papazian/exploit_prevention_stack_traces_support
Commit report: b7a98ab
Test service: dd-trace-py

✅ 0 Failed, 148124 Passed, 20428 Skipped, 8h 21m 48.03s Total duration (1h 21m 39.1s time saved)

pr-commenter · 2024-03-01T14:43:08Z

Benchmarks

Benchmark execution time: 2024-03-25 19:28:42

Comparing candidate commit e658581 in PR branch christophe-papazian/exploit_prevention_stack_traces_support with baseline commit 6955407 in branch main.

Found 0 performance improvements and 1 performance regressions! Performance is the same for 200 metrics, 9 unstable metrics.

scenario:sethttpmeta-obfuscation-disabled

🟥 max_rss_usage [+470.170KB; +710.706KB] or [+2.180%; +3.295%]

…xploit_prevention_stack_traces_support

tests/appsec/contrib_appsec/flask_app/app.py

…xploit_prevention_stack_traces_support

…_traces_support

…af (#8734) Previously, all waf addresses were persistent, meaning that we only needed to send them once to the waf per request. With the introduction of new features, ephemeral addresses will become more and more common, and they need to be resent each time to the waf. This PR address that by keeping a list of persistent addresses and treat all other addresses as ephemeral. Required for #8568 ## Checklist - [x] Change(s) are motivated and described in the PR description - [x] Testing strategy is described if automated tests are not included in the PR - [x] Risks are described (performance impact, potential for breakage, maintainability) - [x] Change is maintainable (easy to change, telemetry, documentation) - [x] [Library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) are followed or label `changelog/no-changelog` is set - [x] Documentation is included (in-code, generated user docs, [public corp docs](https://github.com/DataDog/documentation/)) - [x] Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) - [x] If this PR changes the public interface, I've notified `@DataDog/apm-tees`. - [x] If change touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a review from `@DataDog/security-design-and-guidance`. ## Reviewer Checklist - [ ] Title is accurate - [ ] All changes are related to the pull request's stated goal - [ ] Description motivates each change - [ ] Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - [ ] Testing strategy adequately addresses listed risks - [ ] Change is maintainable (easy to change, telemetry, documentation) - [ ] Release note makes sense to a user of the library - [ ] Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - [ ] Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

…xploit_prevention_stack_traces_support

tests/appsec/contrib_appsec/fastapi_app/app.py

tests/appsec/contrib_appsec/flask_app/app.py

tests/appsec/contrib_appsec/django_app/urls.py

tests/appsec/contrib_appsec/fastapi_app/app.py

tests/appsec/contrib_appsec/django_app/urls.py

ddtrace/appsec/_asm_request_context.py

ddtrace/appsec/_common_module_patches.py

Co-authored-by: Federico Mon <federico.mon@datadoghq.com>

ddtrace/appsec/_common_module_patches.py

ddtrace/appsec/_exploit_prevention/stack_traces.py

ddtrace/appsec/_processor.py

tests/appsec/contrib_appsec/utils.py

…_traces_support

ddtrace/appsec/_exploit_prevention/stack_traces.py

…_traces_support

Following #8568, this PR add support for SSRF for exploit prevention. 1. Add support for SSRF using urllib.request in standard Python API 2. Improve handling of parameters for exploit prevention (positioned or named) 3. Add endpoints and new unit tests for SSRF 4. Add preliminary support for iast in threat hatch tests This feature is still private and disabled. Corresponding tests were run locally and on the CI before being marked skipped. APPSEC-51853 ## Checklist - [x] Change(s) are motivated and described in the PR description - [x] Testing strategy is described if automated tests are not included in the PR - [x] Risks are described (performance impact, potential for breakage, maintainability) - [x] Change is maintainable (easy to change, telemetry, documentation) - [x] [Library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) are followed or label `changelog/no-changelog` is set - [x] Documentation is included (in-code, generated user docs, [public corp docs](https://github.com/DataDog/documentation/)) - [x] Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) - [x] If this PR changes the public interface, I've notified `@DataDog/apm-tees`. - [x] If change touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a review from `@DataDog/security-design-and-guidance`. ## Reviewer Checklist - [x] Title is accurate - [x] All changes are related to the pull request's stated goal - [x] Description motivates each change - [x] Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - [x] Testing strategy adequately addresses listed risks - [x] Change is maintainable (easy to change, telemetry, documentation) - [x] Release note makes sense to a user of the library - [x] Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - [x] Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Co-authored-by: Romain Komorn <136473744+romainkomorndatadog@users.noreply.github.com> Co-authored-by: Federico Mon <federico.mon@datadoghq.com> Co-authored-by: Teague Bick <teague.bick@datadoghq.com> Co-authored-by: Emmett Butler <723615+emmettbutler@users.noreply.github.com> Co-authored-by: Yun Kim <35776586+Yun-Kim@users.noreply.github.com> Co-authored-by: erikayasuda <153395705+erikayasuda@users.noreply.github.com> Co-authored-by: Alberto Vara <alberto.vara@datadoghq.com>

christophe-papazian added 10 commits February 27, 2024 16:48

add stack trace generation for exploit prevention

3d30f0f

use secrets

e402007

make span optional

bddb58d

make span optional

6067dcf

appsec -> appsec_traces

c85be36

update triggers

a79da6f

add logs

eabf781

use waf_triggers

1912fa4

Merge remote-tracking branch 'origin/main' into christophe-papazian/e…

90fb491

…xploit_prevention_stack_traces_support

add exploit prevention env var and settings management

9ac1aad

previous work

1e42a28

christophe-papazian force-pushed the christophe-papazian/exploit_prevention_stack_traces_support branch from 8563a19 to 1e42a28 Compare March 13, 2024 16:05

christophe-papazian added 4 commits March 13, 2024 17:08

Merge remote-tracking branch 'origin/main' into christophe-papazian/e…

1b1e8dd

…xploit_prevention_stack_traces_support

Merge remote-tracking branch 'origin/main' into christophe-papazian/e…

d0f3efc

…xploit_prevention_stack_traces_support

revert api_manager changes

d72591b

first lfi attack implemented on Flask

f43bf1e

github-advanced-security bot found potential problems Mar 14, 2024

View reviewed changes

tests/appsec/contrib_appsec/flask_app/app.py Fixed Show fixed Hide fixed

tests/appsec/contrib_appsec/flask_app/app.py Dismissed Show dismissed Hide dismissed

christophe-papazian added 5 commits March 15, 2024 09:37

update waf version in snapshots

fce4851

Merge remote-tracking branch 'origin/main' into christophe-papazian/e…

e731d43

…xploit_prevention_stack_traces_support

add lfi support with stack trace report (no blocking yet)

5767ec3

Merge remote-tracking branch 'origin/main' into christophe-papazian/e…

646d5bf

…xploit_prevention_stack_traces_support

fix block

2c78cc2

christophe-papazian mentioned this pull request Mar 21, 2024

chore(asm): add persistent and ephemeral addresses handling for the waf #8734

Merged

18 tasks

Merge branch 'main' into christophe-papazian/exploit_prevention_stack…

d9f53a4

…_traces_support

christophe-papazian added 2 commits March 21, 2024 17:12

more of the ep feature

29fd721

Merge remote-tracking branch 'origin/main' into christophe-papazian/e…

0a347fa

…xploit_prevention_stack_traces_support

github-advanced-security bot found potential problems Mar 21, 2024

View reviewed changes