Incorrectly waiting for the entire timeout duration #595
Description
Repro
- Serve a simple html page:
mkdir repro-katana-timeout
echo '<html> <body> <a href="http://localhost:3000?header">Header</a> <a href="http://localhost:3000?footer">Footer</a> </body> </html>' > repro-katana-timeout/index.html
npx serve -p 3000 repro-katana-timeout- Crawl it with katana
# open new terminal
katana -u http://localhost:3000 -duc -timeout 1
katana -u http://localhost:3000 -ducExpected
Both commands should complete within similar time.
Actual
If we specify a -timeout 1 flag, the command takes 2 seconds. Otherwise it takes 11(!) seconds. The requests, responses, and output of both commands are identical. The duration difference between the two commands is ~10 seconds, which happens to be the default timeout value. So I'm guessing that katana is waiting this full timeout duration when it shouldn't be, since there is nothing to wait for?
More info
Logs
$ katana -u http://localhost:3000 -duc --timeout 1 --verbose --debug
__ __
/ /_____ _/ /____ ____ ___ _
/ '_/ _ / __/ _ / _ \/ _ /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/
projectdiscovery.io
[INF] Started standard crawling for => http://localhost:3000
[GET] http://localhost:3000
[a] [GET] http://localhost:3000?header
[a] [GET] http://localhost:3000?footer
$ katana -u http://localhost:3000 -duc --verbose --debug
__ __
/ /_____ _/ /____ ____ ___ _
/ '_/ _ / __/ _ / _ \/ _ /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/
projectdiscovery.io
[INF] Started standard crawling for => http://localhost:3000
[GET] http://localhost:3000
[a] [GET] http://localhost:3000?header
[a] [GET] http://localhost:3000?footer
Time comparison
$ hyperfine --max-runs 10 --warmup 2 'katana -u http://localhost:3000 -duc -timeout 1' 'katana -u http://localhost:3000 -duc'
Benchmark 1: katana -u http://localhost:3000 -duc -timeout 1
Time (mean ± σ): 2.120 s ± 0.016 s [User: 0.086 s, System: 0.037 s]
Range (min … max): 2.097 s … 2.142 s 10 runs
Benchmark 2: katana -u http://localhost:3000 -duc
Time (mean ± σ): 11.127 s ± 0.017 s [User: 0.102 s, System: 0.043 s]
Range (min … max): 11.106 s … 11.155 s 10 runs
Summary
katana -u http://localhost:3000 -duc -timeout 1 ran
5.25 ± 0.04 times faster than katana -u http://localhost:3000 -duc
Versions
katana: v1.0.4