Skip to content

Harden parseErrorCodeFromS3 to never throw on non-XML responses 7.3#12997

Merged
spraza merged 5 commits intoapple:release-7.3from
saintstack:parseErrorCodeFromS3_7.3
Apr 15, 2026
Merged

Harden parseErrorCodeFromS3 to never throw on non-XML responses 7.3#12997
spraza merged 5 commits intoapple:release-7.3from
saintstack:parseErrorCodeFromS3_7.3

Conversation

@saintstack
Copy link
Copy Markdown
Contributor

Return "" instead of throwing on parse failures since many HTTP error responses (502/503 HTML pages, empty bodies) are not XML. Remove unused backup_parse_s3_response_failure error code. Fix const-ref parameter and add unit tests.

This is backport of #12996 and then makes use of the backported method so this PR subsumes #12979

…e#12986)

Add parseErrorCodeFromS3 function that safely extracts S3 error codes
from XML responses, returning "" instead of throwing on parse failures
since many HTTP error responses (502/503 HTML pages, empty bodies) are
not XML. Wire it into doRequest_impl error handling to log S3ErrorCode
and detect bad requests. Add unit tests.

(cherry picked from commit 7388f54)
@saintstack saintstack force-pushed the parseErrorCodeFromS3_7.3 branch from 8e7cae3 to 8de84e6 Compare April 15, 2026 18:56
@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

  • Commit ID: 8e7cae3
  • Duration 0:04:44
  • Result: ❌ FAILED
  • Error: git checkout failed with exit status 128: fatal: unable to read tree (8e7cae34af1c758f603958e26e8b1ee647566dc8) for primary source and source version 8e7cae34af1c758f603958e26e8b1ee647566dc8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-73 on Linux RHEL 9

  • Commit ID: 8e7cae3
  • Duration 0:04:47
  • Result: ❌ FAILED
  • Error: git checkout failed with exit status 128: fatal: unable to read tree (8e7cae34af1c758f603958e26e8b1ee647566dc8) for primary source and source version 8e7cae34af1c758f603958e26e8b1ee647566dc8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-73 on Linux RHEL 9

  • Commit ID: 8e7cae3
  • Duration 0:04:45
  • Result: ❌ FAILED
  • Error: git checkout failed with exit status 128: fatal: unable to read tree (8e7cae34af1c758f603958e26e8b1ee647566dc8) for primary source and source version 8e7cae34af1c758f603958e26e8b1ee647566dc8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

Widen the S3 error logging from only HTTP 400 to all 4xx client errors
(403, 404, 409, etc). Remove unused badRequestCode variable. Rename
trace event from S3BlobStoreBadRequest to S3BlobStoreClientError.
@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-73 on Linux RHEL 9

  • Commit ID: 8de84e6
  • Duration 0:33:19
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

gxglass
gxglass previously approved these changes Apr 15, 2026
@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-73 on Linux RHEL 9

  • Commit ID: 8de84e6
  • Duration 0:44:05
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-73 on Linux RHEL 9

  • Commit ID: 593e792
  • Duration 0:33:57
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-73 on Linux RHEL 9

  • Commit ID: 593e792
  • Duration 0:46:03
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

  • Commit ID: 8de84e6
  • Duration 2:28:53
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

  • Commit ID: 593e792
  • Duration 2:14:15
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-73 on Linux RHEL 9

  • Commit ID: 62255f4
  • Duration 0:54:18
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-73 on Linux RHEL 9

  • Commit ID: 62255f4
  • Duration 0:58:30
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-73 on Linux RHEL 9

  • Commit ID: 62255f4
  • Duration 0:54:09
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@saintstack saintstack requested a review from spraza April 15, 2026 22:04
@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-73 on Linux RHEL 9

  • Commit ID: 62255f4
  • Duration 0:58:58
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

Comment thread fdbclient/S3BlobStore.actor.cpp Outdated
std::string s3Error = parseErrorCodeFromS3(r->data.content);
event.detail("S3ErrorCode", s3Error);
if (r->code >= 400 && r->code < 500) {
TraceEvent(SevWarnAlways, "S3BlobStoreClientError")
Copy link
Copy Markdown
Collaborator

@gxglass gxglass Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would write this as event and let the detail calls below append to the existing one. Otherwise we have a new one at unlimited frequency

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ugh. Yeah. This is redundant now. Let me push...


diff --git a/fdbclient/S3BlobStore.actor.cpp b/fdbclient/S3BlobStore.actor.cpp
index d0675c5c82..5f9e6a175b 100644
--- a/fdbclient/S3BlobStore.actor.cpp
+++ b/fdbclient/S3BlobStore.actor.cpp
@@ -1083,12 +1083,6 @@ ACTOR Future<Reference<HTTP::IncomingResponse>> doRequest_impl(Reference<S3BlobS
                        event.detail("HttpResponseContent", r->data.content);
                        std::string s3Error = parseErrorCodeFromS3(r->data.content);
                        event.detail("S3ErrorCode", s3Error);
-                       if (r->code >= 400 && r->code < 500) {
-                               TraceEvent(SevWarnAlways, "S3BlobStoreClientError")
-                                   .detail("HttpCode", r->code)
-                                   .detail("HttpResponseContent", r->data.content)
-                                   .detail("S3Error", s3Error);
-                       }
                }

                event.detail("Verb", verb)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-73 on Linux RHEL 9

  • Commit ID: 7e3fd6e
  • Duration 1:02:21
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@saintstack
Copy link
Copy Markdown
Contributor Author

20260415-220556-stack_7.3-ecb0fc0a8cef6852 compressed=True data_size=51151522 duration=4624404 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:59:49 sanity=False started=100000 stopped=20260415-230545 submitted=20260415-220556 timeout=5400 username=stack_7.3

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-73 on Linux RHEL 9

  • Commit ID: 7e3fd6e
  • Duration 1:07:39
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

  • Commit ID: 62255f4
  • Duration 2:21:53
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

  • Commit ID: 62255f4
  • Duration 2:15:48
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@spraza spraza merged commit 3b28859 into apple:release-7.3 Apr 15, 2026
0 of 3 checks passed
@saintstack
Copy link
Copy Markdown
Contributor Author

I ran a quick 25k test 20260415-231635-stack_7.3-da31a41ff3b3eb9e compressed=True data_size=51152744 duration=1471476 ended=25000 fail_fast=10 max_runs=25000 pass=25000 priority=100 remaining=0 runtime=0:20:36 sanity=False started=25000 stopped=20260415-233711 submitted=20260415-231635 timeout=5400 username=stack_7.3

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-73 on Linux RHEL 9

  • Commit ID: fe16017
  • Duration 0:57:39
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

  • Commit ID: 7e3fd6e
  • Duration 2:24:24
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-73 on Linux RHEL 9

  • Commit ID: fe16017
  • Duration 1:16:31
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

  • Commit ID: fe16017
  • Duration 2:44:53
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

saintstack added a commit to saintstack/foundationdb that referenced this pull request Apr 18, 2026
…pple#12997)

* Harden parseErrorCodeFromS3 to never throw on non-XML responses (apple#12986)

Add parseErrorCodeFromS3 function that safely extracts S3 error codes
from XML responses, returning "" instead of throwing on parse failures
since many HTTP error responses (502/503 HTML pages, empty bodies) are
not XML. Wire it into doRequest_impl error handling to log S3ErrorCode
and detect bad requests. Add unit tests.

(cherry picked from commit 7388f54)

* Log all S3 4xx client errors, not just 400

Widen the S3 error logging from only HTTP 400 to all 4xx client errors
(403, 404, 409, etc). Remove unused badRequestCode variable. Rename
trace event from S3BlobStoreBadRequest to S3BlobStoreClientError.

* Fix trace

* Simplify S3BlobStore request error handling (merge PR apple#13001)

* Remove separate S3BlobStoreClientError event, use existing event

---------

Co-authored-by: michael stack <stack@duboce.com>
saintstack added a commit that referenced this pull request Apr 22, 2026
* Harden parseErrorCodeFromS3 to never throw on non-XML responses 7.3 (#12997) - 7.3

* Harden parseErrorCodeFromS3 to never throw on non-XML responses (#12986) - main

Add parseErrorCodeFromS3 function that safely extracts S3 error codes
from XML responses, returning "" instead of throwing on parse failures
since many HTTP error responses (502/503 HTML pages, empty bodies) are
not XML. Wire it into doRequest_impl error handling to log S3ErrorCode
and detect bad requests. Add unit tests.

(cherry picked from commit 7388f54)

* Log all S3 4xx client errors, not just 400

Widen the S3 error logging from only HTTP 400 to all 4xx client errors
(403, 404, 409, etc). Remove unused badRequestCode variable. Rename
trace event from S3BlobStoreBadRequest to S3BlobStoreClientError.

* Fix trace

* Simplify S3BlobStore request error handling (merge PR #13001)

* Remove separate S3BlobStoreClientError event, use existing event
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants