`put-bucket-lifecycle-configuration` MalformedXML's if an expiration rule has both the `ExpiredObjectDeleteMarker` key and a file size filter key

[sagihirshfeld]

Describe the bug

While experimenting with setting lifecycle configurations with the following config json:

$ cat lifecycle_config.json | jq
{
  "Rules": [
    {
      "Expiration": {
        "Days": 1,
        "ExpiredObjectDeleteMarker": true
      },
      "ID": "my_rule_id",
      "Filter": {
        "And": {
          "Prefix": "my_prefix/",
          "ObjectSizeGreaterThan": 102400,
          "ObjectSizeLessThan": 204800
        }
      },
      "Status": "Enabled"
    }
  ]
}

Using put-bucket-lifecycle-configuration via the AWS CLI or it's boto3 equivalent failed with the errors posted below.

Once Expiration.ExpiredObjectDeleteMarker was removed or both the size filter keys have been removed, this is no longer happening.

Expected Behavior

For for the lifecycle configuration to be successfully set.

Current Behavior

Via the AWS CLI:

$ aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration=file://lifecycle_config.json   

An error occurred (MalformedXML) when calling the PutBucketLifecycleConfiguration operation: The XML you provided was not well-formed or did not validate against our published schema

Via boto3:

...
...
venv/lib/python3.8/site-packages/botocore/client.py:337: in _api_call
    return self._make_api_call(operation_name, kwargs)
venv/lib/python3.8/site-packages/botocore/client.py:628: in _make_api_call
    request_dict = self._convert_to_request_dict(
venv/lib/python3.8/site-packages/botocore/client.py:676: in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <botocore.validate.ParamValidationDecorator object at 0x2a30aa5e0>
parameters = {'Bucket': 'my-bucket, 'LifecycleConfiguration': {'Rules': [{'Expiration': {'Days': 1,...ObjectSizeLessThan': 2097152, 'Prefix': 'to_expire_b/', 'Tags': [...]}}, 'ID': 'rule-2d768dc8', 'Status': 'Enabled'}]}}
operation_model = OperationModel(name=PutBucketLifecycleConfiguration)

    def serialize_to_request(self, parameters, operation_model):
        input_shape = operation_model.input_shape
        if input_shape is not None:
            report = self._param_validator.validate(parameters,
                                                    operation_model.input_shape)
            if report.has_errors():
>               raise ParamValidationError(report=report.generate_report())
E               botocore.exceptions.ParamValidationError: Parameter validation failed:
E               Unknown parameter in LifecycleConfiguration.Rules[2].Filter.And: "ObjectSizeGreaterThan", must be one of: Prefix, Tags
E               Unknown parameter in LifecycleConfiguration.Rules[2].Filter.And: "ObjectSizeLessThan", must be one of: Prefix, Tags

Reproduction Steps

1. Create lifecycle_config.json with the above content
2. Create an AWS S3 bucket with permissions that allows your account to set lifecycle policies
3. Attempt to apply the lifecycle config by running:

aws s3api put-bucket-lifecycle-configuration --bucket <your-bucket> --lifecycle-configuration=file://lifecycle_config.json 

Possible Solution

Either remove the (obsolete?) ExpiredObjectDeleteMarker key, or both ObjectSizeGreaterThan and ObjectSizeLessThan.

Additional Information/Context

No response

CLI version used

aws-cli/2.13.12 Python/3.11.4 Darwin/23.0.0 source/arm64 prompt/off

Environment details (OS name and version, etc.)

macOS 14.0 (23A344)

[aBurmeseDev]

Hi @sagihirshfeld - thanks for reaching out.

It looks like service API might have recent changes on this behavior and I'm reaching out to s3api team on your behalf for the correct usage of the API.

I'll make sure to keep it posted once I hear back from the team.

[aBurmeseDev]

Can I ask for the debug logs while we wait for response from service team? You can get the logs by adding --debug to the command and please make sure any sensitive info is redacted.

[sagihirshfeld]

Hi @aBurmeseDev, thanks for looking into it! Here are the debug logs along with the config json for reference:
https://gist.github.com/sagihirshfeld/b39ee2ac0b362c72933ece717f9c19b7