Skip to content

Conversation

@GarrettBeatty
Copy link
Contributor

@GarrettBeatty GarrettBeatty commented Oct 12, 2025

This change creates 3 event progress trackers for SimpleUploadCommand. UploadInitiatedEvent, UploadedCompletedEvent, and UploadFailedEvent. This allows the user to subscribe to events like doing below

        private void uploadStarted(object sender, UploadInitiatedEventArgs args)
         {
             Console.WriteLine($"Upload started: {args.FilePath}");
             Console.WriteLine($"Total size: {args.TotalBytes} bytes");
             Console.WriteLine($"Bucket: {args.Request.BucketName}");
             Console.WriteLine($"Key: {args.Request.Key}");
         }


         TransferUtilityUploadRequest request = new TransferUtilityUploadRequest();
         request.UploadInitiatedEvent += uploadStarted;

These new events should only be fired 1 time during the entire lifecycle of the upload. The details of what each event should contain is in the SEP.

Description

  1. DOTNET-8275

Motivation and Context

This is being done as part of the SEP compliance.

Testing

  1. Integration tests/unit tests which demonstrate the event handler behavior
  2. dry run 0bad1273-f543-4436-bbc9-f76c591bde0a

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • My code follows the code style of this project
  • My change requires a change to the documentation
  • I have updated the documentation accordingly
  • I have read the README document
  • I have added tests to cover my changes
  • All new and existing tests passed

License

  • I confirm that this pull request can be released under the Apache 2 license

@GarrettBeatty GarrettBeatty requested a review from Copilot October 12, 2025 21:39
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements simple upload progress tracking for the AWS .NET SDK S3 Transfer Utility by adding lifecycle events (initiated, completed, failed) and creating a unified upload response class. It introduces a comprehensive testing framework with JSON-based field mapping validation to ensure response completeness.

Key Changes:

  • Added three new lifecycle events (UploadInitiatedEvent, UploadCompletedEvent, UploadFailedEvent) to TransferUtilityUploadRequest
  • Created TransferUtilityUploadResponse class to unify response data from different upload operations
  • Implemented ResponseMapper utility for mapping S3 response objects to the unified response format

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
TransferUtilityUploadRequest.cs Added lifecycle event handlers and event argument classes
TransferUtilityUploadResponse.cs New unified response class with properties mapped from S3 operations
SimpleUploadCommand.cs Added event firing logic and progress tracking
SimpleUploadCommand.async.cs Added async event firing and response mapping
ResponseMapper.cs New utility class for mapping S3 responses to unified format
ResponseMapperTests.cs Comprehensive test suite with JSON-driven validation
TransferUtilityTests.cs Integration tests for lifecycle events
AssemblyInfo.cs Added InternalsVisibleTo for unit testing
mapping.json JSON configuration defining field mappings between response types
property-aliases.json Property name aliases for backward compatibility
mapping-schema.json JSON schema validation for mapping configuration

@GarrettBeatty GarrettBeatty changed the title Gcbeatty/simpleuploadprogresstracking Add UploadIniaited, Completed, and Failed event progress trackers to SimpleUploadCommand Oct 12, 2025
@GarrettBeatty GarrettBeatty requested a review from Copilot October 13, 2025 14:07
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Comment on lines +107 to +113
// Keep track of the total transferred bytes so that we can also return this value in case of failure
long transferredBytes = Interlocked.Add(ref _totalTransferredBytes, e.IncrementTransferred - e.CompensationForRetry);
Copy link

Copilot AI Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The use of Interlocked.Add for thread-safe tracking of transferred bytes is correct, but consider potential race conditions where the final transferred bytes count might not accurately reflect the state at the exact moment of failure in multi-threaded scenarios.

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

@GarrettBeatty GarrettBeatty Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if this really matters because this is supposed to be the current moment in time. also multi part upload has been doing this same logic

long transferredBytes = Interlocked.Add(ref _totalTransferredBytes, e.IncrementTransferred - e.CompensationForRetry);

@GarrettBeatty GarrettBeatty force-pushed the getobject branch 2 times, most recently from 25ac4ec to 80394b4 Compare October 13, 2025 18:07
@GarrettBeatty GarrettBeatty force-pushed the gcbeatty/simpleuploadprogresstracking branch from 5f38ee4 to b199735 Compare October 13, 2025 18:14
@GarrettBeatty GarrettBeatty force-pushed the getobject branch 2 times, most recently from 04c81d1 to 4ff0353 Compare October 14, 2025 15:48
@GarrettBeatty GarrettBeatty force-pushed the gcbeatty/simpleuploadprogresstracking branch 2 times, most recently from 0f0e9f4 to 352afda Compare October 14, 2025 16:32
@GarrettBeatty GarrettBeatty changed the base branch from getobject to gcbeatty/tuuploadresponse October 14, 2025 16:43
@GarrettBeatty GarrettBeatty force-pushed the gcbeatty/tuuploadresponse branch from 1924fee to db89c38 Compare October 14, 2025 22:32
@GarrettBeatty GarrettBeatty force-pushed the gcbeatty/tuuploadresponse branch from 72d85c3 to 2dffdee Compare October 16, 2025 19:14
@GarrettBeatty GarrettBeatty force-pushed the gcbeatty/simpleuploadprogresstracking branch from 352afda to a5de17c Compare October 17, 2025 14:32
@GarrettBeatty GarrettBeatty requested a review from Copilot October 17, 2025 14:33
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Comment on lines 115 to 129
private void FireTransferInitiatedEvent()
{
var initiatedArgs = new UploadInitiatedEventArgs(
request: _fileTransporterRequest,
filePath: _fileTransporterRequest.FilePath,
totalBytes: _fileTransporterRequest.ContentLength
);

_fileTransporterRequest.OnRaiseTransferInitiatedEvent(initiatedArgs);
}
Copy link

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

totalBytes is sourced from _fileTransporterRequest.ContentLength, which is often 0/unspecified for file-path uploads and is set on the underlying PutObjectRequest, not on TransferUtilityUploadRequest. This causes incorrect TotalBytes (and will fail the new tests). Compute the size from the FilePath when present (FileInfo.Length), otherwise fall back to ContentLength (for stream uploads), e.g., introduce a helper GetTotalBytes() and use it here.

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this isnt a concern because in

we have logic for this

Comment on lines +41 to 43
FireTransferInitiatedEvent();

var putRequest = ConstructRequest();
Copy link

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Firing the initiated event before constructing the PutObjectRequest makes it harder to provide accurate metadata (like total bytes) if the file size or request normalization occurs during construction. Consider constructing or at least computing total size first, then firing the initiated event so the event carries accurate totals.

Suggested change
FireTransferInitiatedEvent();
var putRequest = ConstructRequest();
var putRequest = ConstructRequest();
FireTransferInitiatedEvent();

Copilot uses AI. Check for mistakes.
Comment on lines +166 to +181
// Use invalid bucket name to force failure
var invalidBucketName = "invalid-bucket-name-" + Guid.NewGuid().ToString();

try
{
UploadWithLifecycleEventsAndBucket(fileName, 5 * MEG_SIZE, invalidBucketName, null, null, eventValidator);
Assert.Fail("Expected an exception to be thrown for invalid bucket");
}
catch (AmazonS3Exception)
{
// Expected exception - the failed event should have been fired
eventValidator.AssertEventFired();
}
Copy link

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using an invalid bucket name triggers client-side validation that can throw ArgumentException instead of AmazonS3Exception, making this test brittle. Use a valid but non-existent bucket name (e.g., conforming to S3 naming rules) and catch AmazonS3Exception, or broaden the catch to Exception to ensure the failed event assertion runs regardless of where the failure occurs.

Copilot uses AI. Check for mistakes.
@GarrettBeatty GarrettBeatty force-pushed the gcbeatty/simpleuploadprogresstracking branch from 82ac159 to 810baa6 Compare October 17, 2025 16:40
@GarrettBeatty GarrettBeatty requested a review from Copilot October 17, 2025 16:41
this._fileTransporterRequest = fileTransporterRequest;

// Cache content length immediately while stream is accessible to avoid ObjectDisposedException in failure scenarios
this._cachedContentLength = this._fileTransporterRequest.ContentLength;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we actually do this in multi part upload command too

this._contentLength = this._fileTransporterRequest.ContentLength;

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

… upload

update docs and function namse

update comments

add tests/update comments

remove response from progressargs

add missing properties

update test

Refactor test

rearrange test

update tests

add commented out tests

comments

comments

update comments

fix spelling

use interlocked read and fix build

dev config

update comment

fix internals
@GarrettBeatty GarrettBeatty force-pushed the gcbeatty/simpleuploadprogresstracking branch from 3f70b05 to 9bd94c8 Compare October 18, 2025 02:41
@GarrettBeatty GarrettBeatty deleted the gcbeatty/simpleuploadprogresstracking branch October 23, 2025 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant