Let bucket replication take precedence over the S3 storage class header (operator switch)

[greenwich]

Background

Before HDDS-10979 (2.1.0), the S3 Gateway gave an EC bucket precedence over the client's x-amz-storage-class header (S3Utils.java#L85-L97 in 2.0.0). The comment there gave the reason: "keys created from file system under EC bucket will always be EC'd."

Since HDDS-10979, any non-empty header wins outright (S3Utils.java#L89-L90 in 2.2.1). STANDARD now always means RATIS/THREE, even in a bucket the administrator configured as EC, so setting EC on the bucket doesn't help.

What happened to us

We run an EC-by-default cluster. After upgrading from 2.0.0 to 2.2.1, a bulk copy job used an off-the-shelf S3 copy tool that sends x-amz-storage-class: STANDARD. All of its keys were written RATIS/THREE instead of EC:

• about 3× the raw capacity for that data, instead of 1.67× with rs-3-2;
• on our configuration, the first use of Ratis DataStream on the datanodes, with a large jump in threads and file descriptors;
• no error or warning. We found it in the OM audit log (replicationType: RATIS) only because the datanode resource graphs moved.

The only fix today is changing every client, and cluster operators often don't control the clients.

The storage-policy work (HDDS-11233, HDDS-16039 / #11229) uses the same precedence: request, then bucket, then default. Once it lands, a client sending GLACIER could place data on COLD storage in a bucket the administrator set to HOT.

Proposal

An operator switch, off by default so current behaviour is unchanged, that makes the server-side settings authoritative. For example, ozone.s3g.storage-class.client-override.enabled (default true). When it is false:

• The S3 Gateway ignores x-amz-storage-class and x-amz-meta-storage-config when choosing replication, and sends none to the OM. The bucket default applies, and then the server default.
• Once storage policies exist, the same switch stops the header from choosing the policy too.
• The header is still validated: unknown classes still get InvalidStorageClass. Valid classes are accepted and ignored, not rejected, so tools that always send STANDARD keep working.
• StorageClass on GET, HEAD and LIST keeps reporting the class of the key's actual replication (HDDS-12877).

The new S3StorageClassResolver in #11229 looks like the natural place for this.

Possible follow-up at the OM level. The same override exists outside S3: OzoneConfigUtil.resolveReplicationConfigPreference uses any client-supplied replication before the bucket default (OzoneConfigUtil.java#L44-L49 in 2.2.1). A bucket-level "enforce default replication" attribute would cover ofs and native clients too.

Questions

1. Is an opt-in switch like this acceptable, or would you prefer restoring the old EC-bucket precedence directly?
2. Should it apply to all writes, or only to buckets with an explicit replication setting? (The second is more conservative.)
3. One switch for replication and storage policy, or two?

We're happy to help with the patch and the tests.

Jira

I wanted to file this as an HDDS Improvement (as I normally do) under HDDS-10980, but I can't create issues anymore: every attempt fails with "We can't create this issue for you right now, it could be due to unsupported content…". Does anyone know why this happens, or could someone file this one?

[jojochuang]

Looks like Apache JIRA is under maintenance. I can tell the JIRA frontend looks different now, but I can't create jira issues either.

[smengcl]

Jira issue tracked under https://issues.apache.org/jira/browse/INFRA-28567

[smengcl]

Jira issue is resolved now.

[greenwich]

Ticket created: https://issues.apache.org/jira/browse/HDDS-17303