As ACF tech lead, I need the nightly-owasp-scan CircleCI workflow to run successfully and exclude non-TDP URLs in scan report

[ADPennington]

Context:

the nightly OWASP scans have been timing out more frequently, at the backend scan step. As a reminder, we run these security vulnerability scans and store them in the staging backend (for now) to meet ACF OCIO compliance standards (ref).

See screenshot of workflow 30-day success rate as of 2/15 below:

The error message in the job is: Too long with no output (exceeded 10m0s): context deadline exceeded. According to CircleCI documentation, 10 minutes is the default timeout setting, and some suggestions are offered therein to resolve, including increasing the timeout.

After some investigations noted in the 2/14 and 2/15 comments below, we decided that we need to increase the timeout and also prevent the non-TDP URLs from being scanned for security vulnerabilities which appears to be the root cause of the timeout.

ACs:

• nightly-owasp-scan workflow scan runs without timing out at 10min
• non-TDP URLs are not reported in the backend scan

Tasks:

• Decide strategy for excluding non-TDP URLs from being scanned
• Increase timeout to > 10min
• Determine strategy for testing fix for qasp review

Supporting Documentation:

• backend scan report with non-TDP URLs and evidence of CI step (below)
  
• ZAP issue re: mozilla URLs
• syntax for increasing timeout

[valcollignon]

Ticket needs to be refined further before being picked up. @ADPennington will continue to monitor. If success rate falls further (50% or below), she will make a note and will bring issue back up in a future backlog refinement.

[ADPennington]

it's been a tough week for owasp. this is just an update for awareness.

Realizing that we didnt decide on the baseline for the success rate monitoring. 90 days feels a bit long, I recommend 30 days (we're still above the threshold), since we provide monthly updates to OCIO. cc: @valcollignon @lfrohlich @abottoms-coder

30 day snapshot as of 2/10/22

90 day snapshot as of 2/10/22

[lfrohlich]

@ADPennington 30 days makes sense to me. Thanks for continuing to monitor.

[ADPennington]

2/14 Update:

Backend scan step continues to fail. We've now had 9 consecutive days of owasp job failure (see below) and will soon reach the 50% threshold. cc: @abottoms-coder @lfrohlich @valcollignon

I've been investigating how long it takes for the zap results to be produced since the default timeout in CI is 10 min. Some notes:

• ran the nightly scan against backend (since this is what is timing out in CI) locally via bash -x ./scripts/zap-scanner.sh backend nightly . its taking the report 11-13 minutes to be produced.
• Given the above, I tested increasing the timeout to 15min on my fork and have been monitoring the CI results over the weekend, it's been successful. code change snippet below.

 run-owasp-scan:
    ...
    steps:
      - run:
          name: Execute OWASP ZAP vulnerability scan
          #wait up to 15 minutes for zap results
          no_output_timeout: 15m
          command: ./scripts/zap-scanner.sh <<parameters.target>> <<parameters.environment>>
      ...

• Note the vast majority of the backend scan reports ive seen during testing match the latest version of the report that is stored in the staging backend, but I did notice a couple of scans yesterday that caught 3K+ low risk instances vs the 130+ low risk instances that we've been getting. I've saved a copy here.

[ADPennington]

2/15 Update:

• we're down to a 61% success rate, so recommend bringing this issue into Sprint 41.
• @abottoms-coder reviewed the report I mentioned above and noticed that this is because non-TDP URLs are being included for some reason. I did some poking around to better understand why and stumbled into this issue which speaks to the same anomalous scan results from mozilla. It suggests that we need to update the exclusion criteria. Note that the last comment in the thread notes that these alerts may need be removed via a hook.
• Relatedly, the last 2 backend scans timed out at 15 minutes on my fork, and I believe that this is because of the extra mozilla URLs since those scans have taken longer even when successful. So while we may need to increase the timeout anyway (because its taking >10m even locally), updating the exclusion criteria seems like it could help ensure irrelevant URLs aren't being scanned.

cc: @lfrohlich @valcollignon

[ADPennington]

2/28 update:

Currently testing on feat/1534-updating-to-latest-zaproxy branch:

• ./tdrs-backend/docker-compose.yml updated to latest zap version -- want to see if this will fix this issue with non-TDP URLs from being scanned
• ./.circleci/config.yml updated to increase owasp scan timeout to 60min, have cron job run on remote branch twice a day -- this should be enough to ensure the backend scan doesnt timeout

Will monitor results for next couple of days.

[ADPennington]

3/2 Update:

• changing the zap version in ./tdrs-backend/docker-compose.yml did not prevent non-TDP URLs from being scanned.
• adding ID 100001: Unexpected Content-Type was returned in zap.config and zap-hook.py to be ignored did not prevent non-TDP URLs from being scanned (as JW noted in ZAP False Positives - Unexpected Content Type & Client Error response code #1455)
• adding the non-TDP URLs to the zap.config as OUTOFSCOPE also did not prevent these non-TDP URLs from being scanned (and I'm not confident that the few lines that have this with the merge of Issue 1403: Fix 500 errors found by ZAP scans #1448 are doing anything.)
• downloaded zaproxy desktop to peruse the globalexclude URL options and it includes the regex URL that matches 2 of the non-TDP URLs being scanned (e.g. https://content-signature-2.cdn.mozilla.net). It also looks like this merge checks if any of the URLs that would be caught by the 100001 alert are on the globalexclude list and if so, will exclude. Updated ./scripts/zap.scanner.sh with 1 non-TDP URL and this worked. Next steps:
  • decide w/ dev on whether to add some TDP URLs to reduce the instance count since we know these to be false positive (e.g. https://s3-us-gov-west-1.amazonaws.com) (this may also address ZAP False Positives - Unexpected Content Type & Client Error response code #1455

cc: @abottoms-coder @lfrohlich

[ADPennington]

3/3 Update:

• added s3 urls to the exclusion list per sync with dev. running successfully--example here
• Since we're keeping all TDP URLs in the scan, suggest keeping ZAP False Positives - Unexpected Content Type & Client Error response code #1455 open and in backlog until this zaproxy issue is addressed. 100000 and 100001 alerts are in separate zaproxy scripts, which is why suppressing these in the reports is not feasible.
• opening draft PR today.

[reitermb]

Demoed by @abottoms-coder on 3/15/2022