Get-S3ObjectAttribute doesn't return parts

[mmangiante]

Describe the bug

I tried to use Get-S3ObjectAttribute to return parts of a multipart uploaded file (via web UI) to do the checksum of these, but no parts are returned.,

Expected Behavior

When you call the Get-S3ObjectAttribute cmdlet, I expect that Parts {} array is populated if the upload is multipart, like the TotalPartsCount property indicates correctly.

Current Behavior

The Parts {} is empty, even if the property TotalPartsCount is different from 0.

Reproduction Steps

$bucket = "mybucket"
# I suppose, for this issue that my file to query is in the first position of my bucket
$obj = (Get-S3Object -BucketName $bucket)[0]
# Pass the object to the cmdlet
$obj | Get-S3ObjectAttribute -ObjectAttributes Checksum,ObjectParts -Select ObjectParts

The result is something like this:

IsTruncated : True
MaxParts : 1000
NextPartNumberMarker : 1000
PartNumberMarker : 0
Parts : {}
TotalPartsCount : 1076

Possible Solution

No response

Additional Information/Context

No response

AWS Tools for PowerShell version used

AWS Tools for PowerShell
Version 4.1.241
Copyright 2012-2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Amazon Web Services SDK for .NET
Core Runtime Version 3.7.103.8
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

PowerShell version used

Name Value

---

PSVersion 5.1.19041.2364
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
BuildVersion 10.0.19041.2364
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1

Operating System and version

Windows 10

[ashishdhingra]

AWS PowerShell Tools uses .NET SDK behind the scenes. The issue is reproducible using the below AWS SDK for .NET code:

using Amazon;
using Amazon.S3;
using Amazon.S3.Model;
using Amazon.S3.Transfer;

string filePath = "<<absolute-file-path-to-upload>>";
string bucketName = "<<bucket-name>>";
string objectKey = "<<key>>";

using (AmazonS3Client amazonS3Client = new AmazonS3Client(RegionEndpoint.USEast2))
{
    // Upload the file (tested with 234.5MB file) using TransferUtility, which would use multipart upload due to default settings.
    TransferUtilityUploadRequest transferUtilityUploadRequest = new TransferUtilityUploadRequest()
    {
        FilePath = filePath,
        BucketName= bucketName,
        Key = objectKey
    };

    TransferUtility transferUtility = new TransferUtility(amazonS3Client);
    await transferUtility.UploadAsync(transferUtilityUploadRequest);
    
    // Get the object attributes.
    GetObjectAttributesRequest getObjectAttributesRequest = new GetObjectAttributesRequest()
    {
        BucketName = bucketName,
        Key = objectKey,
        ObjectAttributes = new List<ObjectAttributes>()
        {
            new ObjectAttributes("Checksum"),
            new ObjectAttributes("ObjectParts"),
            new ObjectAttributes("ObjectSize")
        }
    };

    GetObjectAttributesResponse getObjectAttributesResponse = await amazonS3Client.GetObjectAttributesAsync(getObjectAttributesRequest);
    Console.WriteLine("Object Total Parts Count: " + getObjectAttributesResponse.ObjectParts.TotalPartsCount);
    Console.WriteLine("Object Parts Length: " + getObjectAttributesResponse.ObjectParts.Parts.Count);
}

This returns the following output:

Object Total Parts Count: 45
Object Parts Length: 0

Setting MaxParts in GetObjectAttributesRequest to, say, 10, doesn't return any part in ObjectParts.Parts.

[ashishdhingra]

Used below code at the beginning to turn on verbose logging:

Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.SystemDiagnostics;
Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());

It returned below wire logs:

...
...
Amazon Information: 0 : The environment variable AWS_ENABLE_ENDPOINT_DISCOVERY was not set with a value.
Amazon Information: 1 : The environment variable AWS_MAX_ATTEMPTS was not set with a value.
Amazon Information: 2 : The environment variable AWS_RETRY_MODE was not set with a value.
Amazon Information: 3 : The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT was not set with a value.
Amazon Information: 4 : The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE was not set with a value.
Amazon Information: 5 : The environment variable AWS_USE_DUALSTACK_ENDPOINT was not set with a value.
Amazon Information: 6 : The environment variable AWS_USE_FIPS_ENDPOINT was not set with a value.
Amazon Information: 0 : There is no defaults_mode set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 1 : There is no endpoint_discovery_enabled set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 2 : There is no retry_mode set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 3 : There is no max_attempts set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 4 : There is no ec2_metadata_service_endpoint set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 5 : There is no ec2_metadata_service_endpoint_mode set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 6 : There is no use_dualstack_endpoint set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 7 : There is no use_fips_endpoint set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 0 : Resolved DefaultConfigurationMode for RegionEndpoint [us-east-2] to [Legacy].
Amazon Information: 0 : Starting a process with the following ProcessInfo: UseShellExecute - False RedirectStandardError - True, RedirectStandardOutput - True, CreateNoWindow - True
Amazon Information: 1 : Process started
Amazon Information: 0 : Process ends with exitcode - 0
Amazon Verbose: 2 : Single encoded /test.zip with endpoint https://<<bucket-name>>.s3.us-east-2.amazonaws.com/ for canonicalization: /test.zip
Amazon Verbose: 0 : Received response (truncated to 1024 bytes): [<?xml version="1.0" encoding="UTF-8"?>
<GetObjectAttributesResponse xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><ObjectParts><PartsCount>45</PartsCount></ObjectParts><ObjectSize>234459754</ObjectSize></GetObjectAttributesResponse>]

So looks like the Parts collection per https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAttributes.html is not returned by the S3 service. Tried using MaxParts and PartNumberMarker properties, doesn't work.

[ashishdhingra]

P78562393

[ashishdhingra]

@mmangiante This might be a cross SDK issue. I have opened the issue with service team for their inputs. Moving this issue to https://github.com/aws/aws-sdk repository for better management.

[mmangiante]

@ashishdhingra I'm using a little utility now to do the checksum verification, s3verify, and it works; it has the code in github, it's written in Go: for what I see in s3verify main code he uses the sdk v2 for Go and apparently there is no problem; this only as info.

[ashovlin]

I think we do have a bug in the .NET SDK (and therefore AWS Tools for PowerShell, since it uses the SDK internally).

There is a subtle behavior acknowledged in boto/boto3#3509 (comment) that the Parts are only returned if the object was uploaded with an additional checksum. So I don't think the reproduction code in #315 will encounter the issue because a checksum is not specified on the TransferUtilityUploadRequest.

When I specify a checksum, I am able to reproduce the empty Parts collection.

I think the issue is here, our unmarshaller is looking for Part/member. If I change that to Part it unmarshalls the collection correctly.
https://github.com/aws/aws-sdk-net/blob/24d16b58f4b6250e0df7de66b1c37a18eb9e1309/sdk/src/Services/S3/Custom/Model/Internal/MarshallTransformations/GetObjectAttributesPartsUnmarshaller.cs#L79