HDDS-15000. Improve S3 audit log stack traces by adoroszlai · Pull Request #10097 · apache/ozone

adoroszlai · 2026-04-20T18:03:03Z

What changes were proposed in this pull request?

Audit log shows only the last two elements of stack traces. For several failed S3 Gateway requests it is always the same:

... | ERROR | S3GAudit | ... | op=GET_BUCKET ... | ret=FAILURE | org.apache.hadoop.ozone.s3.exception.OS3Exception
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:192)
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:179)

due to two levels of newError calls.

This PR changes code to avod chained newError calls by creating OS3Exception in various newError implementations. Related changes to avoid duplication:

Simplify OS3Exception creation:
- pass error code (S3ErrorTable) instead of separate variables
- also pass cause exception and resource
- remove unused constructors
- log from constructor instead of newError
Create few more variants of newError to reduce null arguments in callers.
Move BucketEndpoint#translateException to S3ErrorTable (renamed to translateResultCode), add more cases to switch. Change ObjectEndpoint to rely on result code translation in newError.

Related minor fix:

Deleting non-existent keys returns success response (204), not an error. However, the audit log shows ERROR instead of SUCCESS, because NOT_FOUND error is handled in the outer delete method, not handleDeleteRequest. This is now moved to let audit log reflect the correct result.

https://issues.apache.org/jira/browse/HDDS-15000

How was this patch tested?

Before:

| ERROR | S3GAudit | user=dlfknslnfslf | ip=172.18.0.6 | op=GET_BUCKET {"bucket":"[bucket-etqvtcowkf]","list-type":"[2]","max-keys":"[-1]","encoding-type":"[url]","x-amz-request-id":"1f772aef-ed39-41a6-b470-5fff0962422e","x-amz-id-2":"bn6THRxxaQso"} | ret=FAILURE | org.apache.hadoop.ozone.s3.exception.OS3Exception
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:192)
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:179)

After:

| ERROR | S3GAudit | user=dlfknslnfslf | ip=172.18.0.6 | op=GET_BUCKET {"bucket":"[bucket-mpqrnkuavm]","list-type":"[2]","max-keys":"[-1]","encoding-type":"[url]","x-amz-request-id":"28a04087-212e-4e82-bc3b-60976b690f86","x-amz-id-2":"0mVf1nwDd"} | ret=FAILURE | org.apache.hadoop.ozone.s3.exception.OS3Exception: Invalid Argument
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:261)
	at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.validateMaxKeys(BucketEndpoint.java:257)

Before:

| ERROR | S3GAudit | user=dlfknslnfslf | ip=172.18.0.6 | op=INIT_MULTIPART_UPLOAD {"bucket":"[bucket-etqvtcowkf]","path":"[ozone-test-6052266241/mpuWithLargeMetadata]","x-amz-request-id":"0e1c28c9-3451-498b-933e-2b668991216f","x-amz-id-2":"5kEMzckEG19","uploads":"[]"} | ret=FAILURE | org.apache.hadoop.ozone.s3.exception.OS3Exception
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:192)
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:179)

After:

| ERROR | S3GAudit | user=dlfknslnfslf | ip=172.18.0.6 | op=INIT_MULTIPART_UPLOAD {"bucket":"[bucket-mpqrnkuavm]","path":"[ozone-test-5385731147/mpuWithLargeMetadata]","x-amz-request-id":"41329a9a-7ab3-4fcb-8f41-06e130631a2e","x-amz-id-2":"S6IY69aIGOpGP","uploads":"[]"} | ret=FAILURE | org.apache.hadoop.ozone.s3.exception.OS3Exception: Illegal user defined metadata. Combined size exceeds the maximum allowed metadata size of 2KB
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:261)
	at org.apache.hadoop.ozone.s3.endpoint.EndpointBase.getCustomMetadataFromHeaders(EndpointBase.java:315)

Before:

| ERROR | S3GAudit | user=dlfknslnfslf | ip=172.18.0.6 | op=DELETE_KEY {"bucket":"[bucket-etqvtcowkf]","path":"[thereisnosuchfile]",...} | ret=FAILURE | KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:766)
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.deleteKey(OzoneManagerProtocolClientSideTranslatorPB.java:965)

After:

| INFO  | S3GAudit | user=dlfknslnfslf | ip=172.18.0.6 | op=DELETE_KEY {"bucket":"[bucket-mpqrnkuavm]","path":"[thereisnosuchfile]",...} | ret=SUCCESS |

https://github.com/adoroszlai/ozone/actions/runs/24665673476

priyeshkaratha

Thanks @adoroszlai for the patch. Changes overall LGTM. Please check some minor comments.

priyeshkaratha · 2026-04-21T04:25:23Z


-  public static OS3Exception newError(S3ErrorTable e, String resource) {
-    return newError(e, resource, null);
+  /** Converts result code of @{code OMException} to {@code S3ErrorTable}, which


There is a typo in the Javadoc tag. It should be {@code instead of @{code.

priyeshkaratha · 2026-04-21T04:27:33Z

+    return new OS3Exception(errorCode, null, resource);
+  }
+
+  /** Creates new {@link OS3Exception} for {@link S3ErrorTable}, {@code resource} and {@code cause}. */


The Javadoc comment for this method is inaccurate as it mentions a resource parameter which is not present in this specific overload.

ivandika3

Thanks @adoroszlai for the improvement, left some comments.

ivandika3

Thanks @adoroszlai , LGTM +1.

priyeshkaratha

LGTM +1

Russole

Thanks @adoroszlai for working on this. LGTM!

ivandika3 · 2026-04-22T02:14:07Z

Thanks @adoroszlai for the improvement and @priyeshkaratha @Russole for the reviews.

adoroszlai · 2026-04-22T04:50:50Z

Thanks @ivandika3, @priyeshkaratha, @Russole for the review.

adoroszlai added 8 commits April 20, 2026 12:46

move translateException to S3ErrorTable, add more cases

f11df1e

simplify OS3Exception constructor

2dd0d0d

let ObjectEndpoint rely on result code translation in newError

ff16240

HDDS-15000. Improve S3 audit log stack traces

bf49cf2

pass cause to newError

8e066b2

setErrorMessage without dummy exception

964468e

remove newError null arguments

79af068

move non-error NOT_FOUND case into handleDeleteRequest

85b6ff7

adoroszlai self-assigned this Apr 20, 2026

adoroszlai added the s3 S3 Gateway label Apr 20, 2026

adoroszlai requested review from Russole, ivandika3 and peterxcli April 20, 2026 19:48

priyeshkaratha reviewed Apr 21, 2026

View reviewed changes

fix javadoc typos

9d63975

ivandika3 reviewed Apr 21, 2026

View reviewed changes

ivandika3 approved these changes Apr 21, 2026

View reviewed changes

priyeshkaratha approved these changes Apr 21, 2026

View reviewed changes

Russole approved these changes Apr 21, 2026

View reviewed changes

ivandika3 merged commit 31c18df into apache:master Apr 22, 2026
45 checks passed

adoroszlai deleted the HDDS-15000 branch April 22, 2026 04:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-15000. Improve S3 audit log stack traces#10097

HDDS-15000. Improve S3 audit log stack traces#10097
ivandika3 merged 9 commits intoapache:masterfrom
adoroszlai:HDDS-15000

adoroszlai commented Apr 20, 2026

Uh oh!

priyeshkaratha left a comment

Uh oh!

priyeshkaratha Apr 21, 2026

Uh oh!

priyeshkaratha Apr 21, 2026

Uh oh!

ivandika3 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ivandika3 left a comment

Uh oh!

priyeshkaratha left a comment

Uh oh!

Russole left a comment

Uh oh!

Uh oh!

ivandika3 commented Apr 22, 2026

Uh oh!

adoroszlai commented Apr 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

adoroszlai commented Apr 20, 2026

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

priyeshkaratha left a comment

Choose a reason for hiding this comment

Uh oh!

priyeshkaratha Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

priyeshkaratha Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

ivandika3 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ivandika3 left a comment

Choose a reason for hiding this comment

Uh oh!

priyeshkaratha left a comment

Choose a reason for hiding this comment

Uh oh!

Russole left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ivandika3 commented Apr 22, 2026

Uh oh!

adoroszlai commented Apr 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants