HEAD to /robots.txt returns 404 error

[osfrickler]

Description

When accessing the /robots.txt location with curl -I, a 404 error is returned. Instead a proper 200 response is expected. Other locations seem not to be affected.

$ curl -I https://try.gitea.io/robots.txt
HTTP/2 404 
alt-svc: h3=":443"; ma=2592000
cache-control: max-age=0, private, must-revalidate, no-transform
content-type: text/plain;charset=utf-8
date: Fri, 19 Apr 2024 12:24:58 GMT
server: Caddy
set-cookie: i_like_gitea=0b00704e9048a57e; Path=/; HttpOnly; Secure; SameSite=Lax
set-cookie: _csrf=6C6usmqRx9Q9v3UExz4aJICNcMU6MTcxMzUyOTQ5ODYwMDU0MTEzNg; Path=/; Max-Age=86400; HttpOnly; Secure; SameSite=Lax
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
content-length: 11

$ curl -s -i https://try.gitea.io/robots.txt | head
HTTP/2 200 
accept-ranges: bytes
alt-svc: h3=":443"; ma=2592000
cache-control: private, max-age=21600
content-type: text/plain; charset=utf-8
date: Fri, 19 Apr 2024 12:25:08 GMT
last-modified: Thu, 16 Sep 2021 04:52:52 GMT
server: Caddy
content-length: 108
...

Gitea Version

1.23.0+dev-172-geb24d973b0

Can you reproduce the bug on the Gitea demo site?

Yes

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

using try.gitea.io, same issue on https://opendev.org

Database

MySQL/MariaDB

[wxiaoguang]

-> Fix HEAD method for robots.txt #30603