PutBucketLifecycleConfiguration writing fails with MalformedXML #1722

rabarar · 2022-06-06T12:18:23Z

Describe the bug

when attempting to Put a lifecycle configuration onto a bucket, it fails with:

Got an error creating the policy:
operation error S3: PutBucketLifecycleConfiguration, https response error StatusCode: 400, RequestID: M2506HYABH1AT7BK, HostID: ttF1JGprIhRDwOkz8e22C4EQR3CoUFvWyFau4z6fckhzzOH9aIGB25HtqZH7cxPy1DOTqD5lzlc=, api error MalformedXML: The XML you provided was not well-formed or did not validate against our published schema

The input looks as follows:

  input := &s3.PutBucketLifecycleConfigurationInput{
        Bucket: bucket,
        LifecycleConfiguration: &types.BucketLifecycleConfiguration{
            Rules: []types.LifecycleRule{
                {       
                    Status: types.ExpirationStatusEnabled,
                    AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
                        DaysAfterInitiation: 1,
                    },  
                    Expiration: &types.LifecycleExpiration{
                        ExpiredObjectDeleteMarker: true,
                        Days:                      0,
                        Date:                      nil,
                    },
                    NoncurrentVersionExpiration:  nil,
                    NoncurrentVersionTransitions: []types.NoncurrentVersionTransition{},
                    ID:                           &lifecyleID,
                    Prefix:                       nil,
                    Transitions:                  []types.Transition{},
                },
            },
        }, 
    }

Expected Behavior

Expect it to write the lifecycle configuration with the instantiated input values

Current Behavior

failure with:

Got an error creating the policy:
operation error S3: PutBucketLifecycleConfiguration, https response error StatusCode: 400, RequestID: M2506HYABH1AT7BK, HostID: ttF1JGprIhRDwOkz8e22C4EQR3CoUFvWyFau4z6fckhzzOH9aIGB25HtqZH7cxPy1DOTqD5lzlc=, api error MalformedXML: The XML you provided was not well-formed or did not validate against our published schema

Reproduction Steps

Use this input:

   input := &s3.PutBucketLifecycleConfigurationInput{
        Bucket: bucket,
        LifecycleConfiguration: &types.BucketLifecycleConfiguration{
            Rules: []types.LifecycleRule{
                {   
                    Status: types.ExpirationStatusEnabled,
                    AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
                        DaysAfterInitiation: 1,
                    },  
                    Expiration: &types.LifecycleExpiration{
                        ExpiredObjectDeleteMarker: true,
                        Days:                      0,  
                        Date:                      nil,
                    },  
                    NoncurrentVersionExpiration:  nil,
                    NoncurrentVersionTransitions: []types.NoncurrentVersionTransition{},
                    ID:                           &lifecyleID,
                    Prefix:                       nil,
                    Transitions:                  []types.Transition{},
                },  
            },  
        },  
    }

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/service/s3/types"

Compiler and Version used

go version go1.18.3 darwin/arm64

Operating System and version

Mac OSX 12.4

The text was updated successfully, but these errors were encountered:

skmcgrail · 2022-06-08T16:44:11Z

Could you follow the instructions in the Developer Guide to enable client logging with the aws.LogRequestWithBody option enabled? That would help us understand the serialized XML that is being sent to the service for your particular request. Thanks!

rabarar · 2022-06-08T20:51:41Z

Here’s the serialized XML on the request body: <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><AbortIncompleteMultipartUpload><DaysAfterInitiation>1</DaysAfterInitiation></AbortIncompleteMultipartUpload><Expiration><ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker></Expiration><ID>cleanup853211</ID><Status>Enabled</Status></Rule></LifecycleConfiguration>

…

On Jun 8, 2022, at 12:44 PM, Sean McGrail ***@***.***> wrote: Could you follow the instructions in the Developer Guide <https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/logging/> to enable client logging with the aws.LogRequestWithBody option enabled? That would help us understand the serialized XML that is being sent to the service for your particular request. Thanks! — Reply to this email directly, view it on GitHub <#1722 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABB2J6HWUPSKYUINIBMK2VTVODEWNANCNFSM5X7GOUEQ>. You are receiving this because you authored the thread.

skmcgrail · 2022-06-09T15:50:34Z

Thank you for the logging information @rabarar ,

I was also able to reproduce this on my end as well in both the V2 and V1 Go SDKs, and the AWS CLI. Will follow-up this investigation with the service team.

skmcgrail · 2022-06-09T16:52:40Z

I've root caused this down to the service expecting the filter API member to be serialized to the request regardless if sent or not. If you provide an empty filter at the CLI it will work correctly, same with the V1 SDK (provide a pointer to a structure).

Works

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Rule>
        <AbortIncompleteMultipartUpload>
            <DaysAfterInitiation>1</DaysAfterInitiation>
        </AbortIncompleteMultipartUpload>
        <Expiration>
            <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
        </Expiration>
        <Filter />
        <ID>abortMulti</ID>
        <Status>Enabled</Status>
    </Rule>
</LifecycleConfiguration>

Doesn't Work

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Rule>
        <AbortIncompleteMultipartUpload>
            <DaysAfterInitiation>1</DaysAfterInitiation>
        </AbortIncompleteMultipartUpload>
        <Expiration>
            <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
        </Expiration>
        <ID>abortMulti</ID>
        <Status>Enabled</Status>
    </Rule>
</LifecycleConfiguration>

Unfortunately the Go V2 SDK uses a union type, so you can't quite set this in exactly the same way.

rabarar · 2022-06-09T17:44:55Z

as an aside, attempting to reference the Filter type LifecycleRuleAndOperator in types fails to resolve and is inconsistent with the go-docs.

…

On Jun 9, 2022, at 12:52 PM, Sean McGrail ***@***.***> wrote: I've root caused this down to the service expecting the filter API member to be serialized to the request regardless if sent or not. Works <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <AbortIncompleteMultipartUpload> <DaysAfterInitiation>1</DaysAfterInitiation> </AbortIncompleteMultipartUpload> <Expiration> <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker> </Expiration> <Filter /> <ID>abortMulti</ID> <Status>Enabled</Status> </Rule> </LifecycleConfiguration> Doesn't Work <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <AbortIncompleteMultipartUpload> <DaysAfterInitiation>1</DaysAfterInitiation> </AbortIncompleteMultipartUpload> <Expiration> <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker> </Expiration> <ID>abortMulti</ID> <Status>Enabled</Status> </Rule> </LifecycleConfiguration> — Reply to this email directly, view it on GitHub <#1722 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABB2J6EXR3D2CN2TO6HCKZLVOIOOPANCNFSM5X7GOUEQ>. You are receiving this because you were mentioned.

skmcgrail · 2022-06-09T18:00:14Z

This code snippet does appear to work, though the serialization is different. It would provide a temporary work around at least:

_, err = client.PutBucketLifecycleConfiguration(context.TODO(), &s3.PutBucketLifecycleConfigurationInput{
		Bucket: aws.String("mcgrails-test-data"),
		LifecycleConfiguration: &types.BucketLifecycleConfiguration{
			Rules: []types.LifecycleRule{
				{
					Status: types.ExpirationStatusEnabled,
					AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
						DaysAfterInitiation: 1,
					},
					Expiration: &types.LifecycleExpiration{
						ExpiredObjectDeleteMarker: true,
					},
					Filter: &types.LifecycleRuleFilterMemberPrefix{Value: ""},
					ID:     aws.String("abortMulti"),
				},
			},
		},
	})

This serializes as:

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
    <AbortIncompleteMultipartUpload>
        <DaysAfterInitiation>1</DaysAfterInitiation>
    </AbortIncompleteMultipartUpload>
    <Expiration>
        <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
    </Expiration>
    <Filter>
        <Prefix></Prefix>
    </Filter>
    <ID>abortMulti</ID>
    <Status>Enabled</Status>
</Rule>
</LifecycleConfiguration>

skmcgrail · 2022-06-09T20:01:23Z

This seems to be broken in the S3 Control model as well, for different, currently unknown, reasons...

client := s3control.NewFromConfig(cfg)
_, err = client.PutBucketLifecycleConfiguration(context.TODO(), &s3control.PutBucketLifecycleConfigurationInput{
	AccountId: aws.String("accountId"),
	Bucket:    aws.String("mcgrails-test-data"),
	LifecycleConfiguration: &types.LifecycleConfiguration{
		Rules: []types.LifecycleRule{
			{
				Status: types.ExpirationStatusEnabled,
				Filter: &types.LifecycleRuleFilter{},
				AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
					DaysAfterInitiation: 1,
				},
				Expiration: &types.LifecycleExpiration{
					ExpiredObjectDeleteMarker: true,
				},
				ID: aws.String("abortMulti"),
			},
		},
	},
})

Returns

<?xml version="1.0" encoding="UTF-8"?>
<ErrorResponse>
    <Error>
        <Code>InvalidURI</Code>
        <Message>Couldn't parse the specified URI.</Message>
        <URI>bucket/mcgrails-test-data/lifecycleconfiguration</URI>
    </Error>
    <RequestId>Z2AGJDNK1QRRVC4V</RequestId>
    <HostId>3pXY5FhtMiP+Bq0w6wEU786mNkWl4ROwekWZ9JOusTZ5UlKkZZictU0+FkpbZlVwY2oa43KmzdQ=</HostId>
</ErrorResponse>

Seeing the same behavior with the Go V1 SDK and CLI.

rabarar · 2022-06-13T12:38:39Z

Is there a timeline for a fix? Feel free to send me a t-shirt for finding this one ;)

…

On Jun 9, 2022, at 4:01 PM, Sean McGrail ***@***.***> wrote: This seems to be broken in the S3 Control model as well, for different, currently unknown, reasons... client := s3control.New(sess) _, err := client.PutBucketLifecycleConfiguration(&s3control.PutBucketLifecycleConfigurationInput{ AccountId: aws.String("<account>"), Bucket: aws.String("mcgrails-test-data"), LifecycleConfiguration: &s3control.LifecycleConfiguration{ Rules: []*s3control.LifecycleRule{ { AbortIncompleteMultipartUpload: &s3control.AbortIncompleteMultipartUpload{ DaysAfterInitiation: aws.Int64(1), }, Expiration: &s3control.LifecycleExpiration{ ExpiredObjectDeleteMarker: aws.Bool(true), }, Filter: &s3control.LifecycleRuleFilter{}, ID: aws.String("abortMulti"), Status: aws.String("Enabled"), }, }, }, }) Returns <?xml version="1.0" encoding="UTF-8"?> <ErrorResponse> <Error> <Code>InvalidURI</Code> <Message>Couldn't parse the specified URI.</Message> <URI>bucket/mcgrails-test-data/lifecycleconfiguration</URI> </Error> <RequestId>Z2AGJDNK1QRRVC4V</RequestId> <HostId>3pXY5FhtMiP+Bq0w6wEU786mNkWl4ROwekWZ9JOusTZ5UlKkZZictU0+FkpbZlVwY2oa43KmzdQ=</HostId> </ErrorResponse> Seeing the same behavior with the Go V1 SDK and CLI. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

rabarar · 2022-10-11T06:56:07Z

Definitely works but has zero relationship to the docs … not sure where types.LifecycleRuleFilterMemberPrefix comes from …

…

On Jun 9, 2022, at 2:00 PM, Sean McGrail ***@***.***> wrote: This code snippet does appear to work, though the serialization is different. It would provide a temporary work around at least: _, err = client.PutBucketLifecycleConfiguration(context.TODO(), &s3.PutBucketLifecycleConfigurationInput{ Bucket: aws.String("mcgrails-test-data"), LifecycleConfiguration: &types.BucketLifecycleConfiguration{ Rules: []types.LifecycleRule{ { Status: types.ExpirationStatusEnabled, AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{ DaysAfterInitiation: 1, }, Expiration: &types.LifecycleExpiration{ ExpiredObjectDeleteMarker: true, }, Filter: &types.LifecycleRuleFilterMemberPrefix{Value: ""}, ID: aws.String("abortMulti"), }, }, }, }) This serializes as: <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <AbortIncompleteMultipartUpload> <DaysAfterInitiation>1</DaysAfterInitiation> </AbortIncompleteMultipartUpload> <Expiration> <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker> </Expiration> <Filter> <Prefix></Prefix> </Filter> <ID>abortMulti</ID> <Status>Enabled</Status> </Rule> </LifecycleConfiguration> — Reply to this email directly, view it on GitHub <#1722 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABB2J6D2DV5GSC5ZFGIBCCTVOIWLRANCNFSM5X7GOUEQ>. You are receiving this because you were mentioned.

RanVaknin · 2022-11-30T23:03:25Z

Hi @rabarar ,

You have to specify the <Filter> tag unless you have a <Prefix> tag in your xml.

source:

Filter
The Filter is used to identify objects that a Lifecycle Rule applies to. A Filter must have exactly one of Prefix, Tag, or And specified. Filter is required if the LifecycleRule does not contain a Prefix element.
Type: LifecycleRuleFilter data type

I think this isn't a bug, just an API design choice from s3.

rabarar · 2022-11-30T23:14:30Z

Hi -The API doesn’t afford access to the xml - that is internal to the go api and therefore is a bug imho.Best,RobOn Nov 30, 2022, at 6:03 PM, Ran Vaknin ***@***.***> wrote: Hi @rabarar , You have to specify the <Filter> tag unless you have a <Prefix> tag in your xml. source: Filter The Filter is used to identify objects that a Lifecycle Rule applies to. A Filter must have exactly one of Prefix, Tag, or And specified. Filter is required if the LifecycleRule does not contain a Prefix element. Type: LifecycleRuleFilter data type I think this isn't a bug, just an API design choice from s3. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

RanVaknin · 2022-12-01T00:19:30Z

@rabarar ,

After taking a deeper look into this here is what I have found.

The SDK serializes Golang strcuts into XML. When you send a request without that field in the request you're essentially leaving it out of the XML.
When the service reads the XML body sent by the GO SDK Client and sees that your LifeCycleRule doesn't have a <Filter> tag, it looks for <Prefix> type and if both are missing it throws this exception.

This is all handled on the service-side, and is documented.

To show that its not a GO SDK bug I've tested it on JS SDK V3 with the following code:

❌ Incorrect:

import { S3Client, PutBucketLifecycleConfigurationCommand } from '@aws-sdk/client-s3';

const client = new S3Client({ region: 'us-east-1' });

const command = new PutBucketLifecycleConfigurationCommand({
    Bucket: 'mybucket123',
    LifecycleConfiguration: {
        Rules: [
            {
                Status: 'Enabled',
                ID: 'abortMulti',
                Expiration: {ExpiredObjectDeleteMarker: true},
            }
        ]
    }
});

try {
    const data = await client.send(command);
    console.log(data)
} catch (error) {
    console.log(error);
}

❌ Incorrect:

const command = new PutBucketLifecycleConfigurationCommand({
    Bucket: 'mybucket123',
    LifecycleConfiguration: {
        Rules: [
            {
                Status: 'Enabled',
                ID: 'abortMulti',
                Expiration: {ExpiredObjectDeleteMarker: true},
                Filter: {} // <- change is here
            }
        ]
    }
});

✅ Correct

const command = new PutBucketLifecycleConfigurationCommand({
    Bucket: 'mybucket123',
    LifecycleConfiguration: {
        Rules: [
            {
                Status: 'Enabled',
                ID: 'abortMulti',
                Expiration: {ExpiredObjectDeleteMarker: true},
                Filter: {Prefix: {}} // <- change is here 
            }
        ]
    }
});

==========

Golang Example:

❌ Wont Compile:

		LifecycleConfiguration: &types.BucketLifecycleConfiguration{
			Rules: []types.LifecycleRule{
				{
					Status: types.ExpirationStatusEnabled,
					Filter: &types.LifecycleRuleFilter{}, // <- this is an interface not a concrete type. 
					AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
						DaysAfterInitiation: 1,
					},
					Expiration: &types.LifecycleExpiration{
						ExpiredObjectDeleteMarker: true,
					},
					ID: aws.String("abortMulti"),
				},
			},
		},
	})

❌ Incorrect:

		LifecycleConfiguration: &types.BucketLifecycleConfiguration{
			Rules: []types.LifecycleRule{
				{
					Status: types.ExpirationStatusEnabled,
					            // <- missing Filter
					AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
						DaysAfterInitiation: 1,
					},
					Expiration: &types.LifecycleExpiration{
						ExpiredObjectDeleteMarker: true,
					},
					ID: aws.String("abortMulti"),
				},
			},
		},

✅ Correct:

LifecycleConfiguration: &types.BucketLifecycleConfiguration{
			Rules: []types.LifecycleRule{
				{
					Status: types.ExpirationStatusEnabled,
					Filter: &types.LifecycleRuleFilterMemberPrefix{},
					AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
						DaysAfterInitiation: 1,
					},
					Expiration: &types.LifecycleExpiration{
						ExpiredObjectDeleteMarker: true,
					},
					ID: aws.String("abortMulti"),
				},
			},
		},

Again quoting documentation:

Filter is required if the LifecycleRule does not contain a Prefix element.
A Filter must have exactly one of Prefix, Tag, or And specified.

The only thing I could argue is that the s3 exception is not detailed enough and requires you to dig through documentation to see what is missing in the XML that is being serialized.

Conclusion: Not a bug.

I will convert this into a discussion and keep it open so I may address more questions if you have them in the future.

Thanks again!
Ran 😄

rabarar added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jun 6, 2022

skmcgrail added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jun 8, 2022

skmcgrail added the service-api This issue is due to a problem in a service API, not the SDK implementation. label Jun 9, 2022

RanVaknin added p3 This is a minor priority issue s Effort estimation: small labels Nov 14, 2022

RanVaknin self-assigned this Nov 30, 2022

RanVaknin added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Nov 30, 2022

github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Dec 1, 2022

aws locked and limited conversation to collaborators Dec 1, 2022

RanVaknin converted this issue into discussion #1944 Dec 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

This issue was moved to a discussion.

PutBucketLifecycleConfiguration writing fails with MalformedXML #1722

PutBucketLifecycleConfiguration writing fails with MalformedXML #1722

rabarar commented Jun 6, 2022

skmcgrail commented Jun 8, 2022

rabarar commented Jun 8, 2022 via email

skmcgrail commented Jun 9, 2022

skmcgrail commented Jun 9, 2022 •

edited

rabarar commented Jun 9, 2022 via email

skmcgrail commented Jun 9, 2022

skmcgrail commented Jun 9, 2022 •

edited

rabarar commented Jun 13, 2022 via email

rabarar commented Oct 11, 2022 via email

RanVaknin commented Nov 30, 2022 •

edited

rabarar commented Nov 30, 2022 via email

RanVaknin commented Dec 1, 2022

This issue was moved to a discussion.

This issue was moved to a discussion.

PutBucketLifecycleConfiguration writing fails with MalformedXML #1722

PutBucketLifecycleConfiguration writing fails with MalformedXML #1722

Comments

rabarar commented Jun 6, 2022

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

AWS Go SDK V2 Module Versions Used

Compiler and Version used

Operating System and version

skmcgrail commented Jun 8, 2022

rabarar commented Jun 8, 2022 via email

skmcgrail commented Jun 9, 2022

skmcgrail commented Jun 9, 2022 • edited

Works

Doesn't Work

rabarar commented Jun 9, 2022 via email

skmcgrail commented Jun 9, 2022

skmcgrail commented Jun 9, 2022 • edited

rabarar commented Jun 13, 2022 via email

rabarar commented Oct 11, 2022 via email

RanVaknin commented Nov 30, 2022 • edited

rabarar commented Nov 30, 2022 via email

RanVaknin commented Dec 1, 2022

❌ Incorrect:

❌ Incorrect:

✅ Correct

Golang Example:

❌ Wont Compile:

❌ Incorrect:

✅ Correct:

Conclusion: Not a bug.

This issue was moved to a discussion.

skmcgrail commented Jun 9, 2022 •

edited

skmcgrail commented Jun 9, 2022 •

edited

RanVaknin commented Nov 30, 2022 •

edited