[Bug] `HTTP` func fails with `Failed to get file size for URI`

[zghong]

Search before asking

• I had searched in the issues and found no similar issues.

Version

4.1.2, master

What's Wrong?

When querying a GET pre-signed object-storage URL via the HTTP(), the query fails with 403 Forbidden while getting the file size:

(1105, 'errCode = 2, detailMessage = Can not build http(): errCode = 2, detailMessage = Failed check http storage props, Failed to get file size for URI: http://.../xxx.txt?Expires=...&Signature=.... java.io.IOException: HTTP request failed with response code: 403, message: Forbidden')

What You Expected?

Query OK.

How to Reproduce?

SELECT * FROM HTTP(
    uri='http://storage.xx.local/20260128170024.txt?Expires=xx&AccessKey=xx&Signature=xx',
    format="csv")
limit 10;

The exact same URL works fine with a GET request:

wget http://storage.xx.local/20260128170024.txt?Expires=xx&AccessKey=xx&Signature=xx   # returns 200

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_7a0c26a89db8

Initial triage

Assessment: high-confidence HTTP TVF compatibility bug. The issue is currently unlabelled. The reported failure is consistent with Doris changing a GET-only pre-signed request into HEAD while discovering object size, before the actual scan begins.

Verified from the source

• The behavior is present in both the 4.1.2 tag and the inspected master commit (82646c38c0069eee4b1704eafef416f969bd4b93).
• HttpTableValuedFunction.generateFileStatus() calls HttpUtils.getHttpFileSize() while constructing the TVF. getHttpFileSize() unconditionally executes connection.setRequestMethod("HEAD"). Its exception text exactly matches the error in this issue, so this failure occurs in FE analysis/planning, not in query execution.
• The scan-side reader has a second occurrence of the same incompatibility: be/src/io/fs/http_file_reader.cpp::HttpFileReader::open() also sends HEAD unless the reader is already initialized. The normal range carries a file size through FileReaderOptions.file_size, but HttpFileReader::create()/open() do not consume that option to initialize the reader; the constructor only recognizes a separate file_size entry in the HTTP property map. Therefore, changing only the FE probe can still leave a later BE HEAD failure.
• Actual HTTP reads default to GET, and range detection uses a small ranged GET. This is compatible with the reporter's successful GET and isolates the mismatch to metadata discovery.
• The current HTTP TVF regression server explicitly accepts both GET and HEAD; there is no coverage for a GET-authorized endpoint that rejects HEAD.

What is inferred vs. still to verify

It is highly likely that the object store returns 403 because the signature authorizes GET but not HEAD (or because the endpoint otherwise forbids HEAD). The Doris-side method substitution is verified; the storage-side rejection reason still needs confirmation from a direct sanitized request or storage access log. This should not be treated as a generic credential failure because the same URL succeeds with GET.

Please provide the following without exposing the full URL or signature values:

1. Status and response headers from a HEAD request and from GET with Range: bytes=0-0 (especially Content-Length, Content-Range, Accept-Ranges, and any redirect). For example, run curl -v -I '<URL>' and curl -v -r 0-0 '<URL>' -o /dev/null, then redact the query parameter values before posting.
2. The object-storage vendor/signing scheme, and whether Range was included in the signed headers.
3. The exact Doris commit SHA for the reported master test and, if available, a redacted object-storage authentication/access-log reason for the rejected HEAD.

A Doris Profile is not needed for this case because execution is never reached.

Recommended next steps

1. Replace method-changing metadata discovery with a GET-compatible mechanism. A small GET range probe is the natural candidate: for 206, parse the total length from Content-Range rather than the one-byte Content-Length; also handle an empty object (416 with Content-Range: bytes */0) and a server that ignores Range and returns 200.
2. Cover both metadata paths. Either make the BE reader honor the already-planned per-file size while preserving its Range/non-Range initialization semantics, or change its independent metadata probe to the same GET-compatible mechanism. Ensure that a successful FE fix cannot be followed by a BE HEAD.
3. Add a regression endpoint that accepts GET/ranged GET but rejects HEAD, then exercise schema inference and full query execution. Include empty files, Range-supported and Range-ignored responses, redirects, and custom http.header.* propagation.
4. If the fix is accepted on master, assess a branch-4.1 backport because the same code is present in 4.1.2.

Current workaround: use an endpoint/proxy that permits both HEAD and GET, or use Doris's object-storage/file TVF with credentials instead of a GET-only pre-signed HTTP URL. http.method, http.enable.range.request, and http.enable.chunk.response do not avoid the FE planning-time HEAD probe.