Too Many Requests when request /healthz via curl

[christiansacks]

Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG

# git pull
Already up to date.
# git log -1
commit e76656a162c40c04ecc4b119d2a28ec2104c5275 (HEAD -> master, origin/master, origin/HEAD)
Merge: d4f06df 12d726f
Author: Alexandre Flament <alex@al-f.net>
Date:   Sat Dec 31 18:35:37 2022 +0100

    Merge pull request #110 from searxng/dac_override

    docker-compose.yaml: remove CAP_DAC_OVERRIDE

As per the footer on the web page: 2023.4.2+b8100e4e

How did you install SearXNG?
Used the docker install from the oficial wiki

What happened?
I'm using curl on a webserver to check the HTTP status code to make sure the site is up or not, and update a status page I made for the services I run, the curl command checks the http response code;

# curl -sI https://ng.erb.pw/|head -1
HTTP/1.1 429 Too Many Requests

How To Reproduce

# curl -sI https://ng.erb.pw/|head -1
HTTP/1.1 429 Too Many Requests

Expected behavior

# curl -sI https://ng.erb.pw/|head -1
HTTP/1.1 200 OK

Additional information;

# curl -sI https://ng.erb.pw/
HTTP/1.1 429 Too Many Requests
Content-Length: 17
Content-Security-Policy: upgrade-insecure-requests; default-src 'none'; script-src 'self'; style-src 'self' 'unsafe-inline'; form-action 'self' https://github.com/searxng/searxng/issues/new; font-src 'self'; frame-ancestors 'self'; base-uri 'self'; connect-src 'self' https://overpass-api.de; img-src 'self' data: https://*.tile.openstreetmap.org; frame-src https://www.youtube-nocookie.com https://player.vimeo.com https://www.dailymotion.com https://www.deezer.com https://www.mixcloud.com https://w.soundcloud.com https://embed.spotify.com
Content-Type: text/html; charset=utf-8
Feature-Policy: accelerometer 'none';ambient-light-sensor 'none'; autoplay 'none';camera 'none';encrypted-media 'none';focus-without-user-activation 'none'; geolocation 'none';gyroscope 'none';magnetometer 'none';microphone 'none';midi 'none';payment 'none';picture-in-picture 'none'; speaker 'none';sync-xhr 'none';usb 'none';vr 'none'
Permissions-Policy: accelerometer=(),ambient-light-sensor=(),autoplay=(),camera=(),encrypted-media=(),focus-without-user-activation=(),geolocation=(),gyroscope=(),magnetometer=(),microphone=(),midi=(),payment=(),picture-in-picture=(),speaker=(),sync-xhr=(),usb=(),vr=()
Referrer-Policy: no-referrer
Server-Timing: total;dur=10.29, render;dur=0
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Robots-Tag: noindex, noarchive, nofollow
X-Xss-Protection: 1; mode=block
Date: Mon, 03 Apr 2023 09:04:07 GMT

[return42]

The HTTP/1.1 429 Too Many Requests comes from the limiter, its a bot blocker. You can disable bot blocking but its recommended.

[unixfox]

Try to use another page for healthcheck like this one: https://ng.erb.pw/info/en/about

IMO the limiter shouldn't block on the root page

[christiansacks]

# curl -sI https://ng.erb.pw/info/en/about|head -1
HTTP/1.1 429 Too Many Requests

Hmm is there a way to just whitelist an IP from this limiter?

EDIT: Is there also a way to remove the IP from the block list so it can at least work again?

[return42]

EDIT: Is there also a way to remove the IP from the block list so it can at least work again?

Sorry, not yet .. there is a PR #2129 .. the PR hangs ATM and I'm unsure if this is the best way to realize whitelisting .. on the long run I think we need some kind of "access that passes through the limiter" .. but may be "lets pass through by a token" is a better solution that fits to more use-cases .. I haven't had time to think about it yet, because I want to fix some urgent bugs before we hastily implement new features.

Sorry that I can not offer more at the moment.

[unixfox]

There would be no need if the limiter didn't filter on the non search endpoints like GET /, GET /info and more shouldn't be restricted by the limiter.

[christiansacks]

Is there a reason this is marked as closed when the problem still persists? Did I duplicate this from another issue? I did search and couldn't see anything the same as the issue I have.

There would be no need if the limiter didn't filter on the non search endpoints like GET /, GET /info and more shouldn't be restricted by the limiter.

I agree this would probably be most useful.

I've turned off the limiter for now, but would like to re-enable it if a workaround is found.

[return42]

I'm using curl on a webserver to check the HTTP status code to make sure the site is up or not, and update a status page I made for the services I run, the curl command checks the http response code;
I've turned off the limiter for now, but would like to re-enable it if a workaround is found.

What you are looking for is /healthz .. I have renamed the title of this issue to "Too Many Requests when request /healthz via curl".

I merge #2313 to fix an issue / tested on my instance:

$ curl https://darmarit.org/searx/healthz
OK

Is there a reason this is marked as closed when the problem still persists?

Sorry, my fail .. is your issue now "closed" or should I reopen (if, for what reason)?