support for delay between request

[0xAwali]

Based on censys docs it's has rate limited

so while I used uncover I got too many requests so it's possible to add delay between requests when using censys engine to bypass rate limited , plus I think uncover doesn't randomize multiple tokens because I used one token and got 12K result then I got too many requests so I used multiple tokens and got the same result , I think if uncover randomize the multiple tokens we can bypass the rate limited

RATE-LIMIT:
   -delay int                    delay in seconds between requests to the same host (default 1)

[ehsandeep]

@0xAwali thanks for creating this issue, about randomisation, it works based on execution, i.e new key will be picked on each execution of uncover and not per request as same key is required to rotate through multiple pages in case of results contains multiple results with higher limits.

[0xAwali]

censys depend on cursor value to get next page e.g.

curl -s -u API-KEY:SECRET -H 'Content-Type: application/json' "https://search.censys.io/api/v2/hosts/search?q=company&per_page=100&cursor=base64-value" | jq -r .result.links.next

so token not issue here , I think it will be more efficiency to randomize the multiple tokens instead of one token every time because rate limit per seconds or minutes will not solve this issue if your result e.g. 100k because censys allow 250 queries if you are normal user so e.g. I was using this script to bypass the rate limit and allowed queries

#!/usr/bin/env bash
Query="services.tls.certificates.leaf_data.subject.organization:Facebook\bInc."
TokenCount=$(shuf -n 1 token.txt)
Count=$(curl -s -u $TokenCount -H 'Content-Type: application/json' "https://search.censys.io/api/v2/hosts/search?q=$Query&per_page=100" | jq -r .result.total)
Iters=$(expr "$Count" / 100 + 1)
Cursor=""
for ((I = 1 ; I <= "$Iters" ; I++))
do
	TokenSearch=$(shuf -n 1 token.txt)
	curl -s -u $TokenSearch -H 'Content-Type: application/json' "https://search.censys.io/api/v2/hosts/search?q=$Query&per_page=100&cursor=$Cursor" | jq -r .result.hits[].ip | tee -a result.txt
	sleep 1
	Cursor=$(curl -s -u $TokenSearch -H 'Content-Type: application/json' "https://search.censys.io/api/v2/hosts/search?q=$Query&per_page=100&cursor=$Cursor" | jq -r .result.links.next)
done

[ehsandeep]

thanks for sharing more information @0xAwali around randomization improvements, and yes you are right about the rate limit, a default delay would be a better option to avoid the rate limit in general across all the engines.

[ehsandeep]

@0xAwali should be fixed on the dev version, now as default delay of 1 second is added.