WebCacheDeceptionScanRule and CorsScanRule taking long + WebCacheDeceptionScanRule 404 behaviour

[barbeque-squared]

Describe the bug
Since owasp/zap2docker-weekly:w2021-06-15, our CI is taking significantly longer, plus eventually fails on what is probably a false positive:

• WebCacheDeceptionScanRule takes over 1000 seconds (log says 33 messages sent)
• CorsScanRule takes over 400 seconds (log says 133 messages sent, it does pass though)
• WebCacheDeceptionScanRule causes failure: both the authorised and unauthorised response are a generic 404 page. It fails because the responses are (too) similar. Both responses properly set the 404 status code.

Full timings are available in timings.log. Note that DomXssScanRule aside, all the other rules take couple of seconds at most. I don't see anything strange in firewall logs. CPU usage is also incredibly high when these two rules are running.

(was told on IRC to put all three in one issue -- if you want separate ones instead, comment and I'll split them out and close this one)

To Reproduce

1. Get a recent owasp/zap2docker-weekly
2. zap-full-scan.py -j -a -t https://REDACTED.org/login -c config.tsv -r report.html

(config.tsv is just every available rule on WARN, plus some OUTOFSCOPE for very specific rule/URL combinations, but those combinations don't affect this)

(I can't share the actual domain publicly. Either email me or comment to PM it to a specific user on IRC if you are unable to reproduce it elsewhere)

Expected behavior
The rules should probably not take this long cq need this much CPU. For WebCacheDeceptionScanRule, it should probably skip certain status codes for the authorised request (at least 404) or perhaps only check successful requests.

Software versions

• ZAP: owasp/zap2docker-weekly:w2021-06-15
• Host OS: various Linux (CentOS, Archlinux)

Would you like to help fix this issue?
If there's a way to run the rule .java files directly, I can provide more details on what it's actually doing on the specific domain / put a profiler on it and see where the CPU usage is coming from.

[thc202]

Thanks for raising the issue.

[barbeque-squared]

Did some testing on this, part of the WebCacheDeceptionScanRule CPU usage is definitely caused by the distance.apply(a, b) call where it computes the distance.
See also this gist, the null threshold is the same as not setting a threshold at all:
https://gist.github.com/barbeque-squared/1ee2a4e80256037653926fb08fc141c3
(it deliberately compares the same strings, if your computer is too quick just increase the 30 in the main loop)

The code doesn't care about the exact value of the Levenshtein distance, at least not once it gets over 200. It appears that with a threshold it's linear in terms of input length, but without it goes superlinear. (note that the Apache Commons library will return -1 if a threshold is given and no distance can be found).

Some very rough figures from my system to give an indication:

String length	No threshold	Threshold 200
500	0.6s	0.4s
1000	2.0s	0.6s
1500	4.4s	1.0s
2000	7.8s	1.3s
2500	12s	1.7s

[J12934]

I also stumbled across this issue (https://owasp.slack.com/archives/C04SX2GAS/p1626340200135100), added some debug logs and can also confirm that this seems to be caused by the Levenshtein distance.

This can be reproduced reliably by scanning Juice Shop (Just using a ajax spider, followed by an active scan without any special config). This causes my ZAP to hour on the scan rule often spending minutes at a time as it seems to be comparing big binary files using Levenshtein.

[hahwul]

I had a similar issue, and in my system, the ZAP was hang.
First of all, I disable the WebCacheDeceptionScanRule in my policy for solve. and I write a comment now because I think it'll help someone!

zap.log

158655 [Thread-10] INFO  org.parosproxy.paros.core.scanner.HostProcess - 
start host https://REDACTED | WebCacheDeceptionScanRule strength MEDIUM threshold MEDIUM

// Freezed here.

WebCacheDeceptionScanRule

https://www.zaproxy.org/docs/alerts/40039/

[thc202]

Was CorsScanRule also addressed or no longer needs to? The title should be updated accordingly.

[barbeque-squared]

@thc202 it seems that CorsScanRule is/was also addressed, I get these kind of timings now:

WebCacheDeceptionScanRule in 4.032s with 33 message(s) sent and 1 alert(s) raised.
CorsScanRule in 2.673s with 133 message(s) sent and 3 alert(s) raised.

The WebCacheDeceptionScanRule 404 behaviour still exists, but I'm less sure whether that's a feature or a bug. If you think it's worthwile to investigate, I can try to set up some kind of minimal example. I'll leave it up to you whether you want it as a separate issue or a reopening of this one (in the latter case I'll edit the title + reopen once I figure out said minimal example)

[kingthorin]

The fixed WCD rule hasn't been released yet.

[barbeque-squared]

Oops, my bad. I missed that the WCD fix was recent.
About CorsScanRule: the ~400 seconds was significant at the time, but it's also the only data point of something that ran after WCD. It's fast now though, so it no longer needs to be addressed. Feel free to update the title if needed to reflect this.

[huornlmj]

This issue is still present in 2.11.1

[kingthorin]

Please be more specific. Scan rules are not part of ZAP core. Add-ons (which contain scan rules) have their own version numbering.

https://www.zaproxy.org/docs/desktop/ui/tlmenu/help/#support-info
https://www.zaproxy.org/docs/desktop/cmdline/#options ( -suppinfo)

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.