ENG-41215: Client Multipart Upload implementation (together-py) #159
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sbassam
force-pushed
the
feature/multipart-upload
branch
3 times, most recently
from
db7db93 to
36a5dd9
Compare
mryab
reviewed
Sep 29, 2025
src/together/lib/resources/files.py
Outdated
Comment on lines
268
to
270
| # Mock server scenario - return mock values for testing | ||
| if response.status_code == 200: | ||
| return "https://mock-upload-url.com", "mock-file-id" |
Member
There was a problem hiding this comment.
Let's mock only inside the testing code, never in the actual implementation
mryab
reviewed
Sep 29, 2025
mryab
reviewed
Sep 29, 2025
src/together/lib/resources/files.py
Outdated
| f"Unsupported file extension: '{file.suffix}'. Supported extensions: .jsonl, .parquet, .csv" | ||
| ) | ||
|
|
||
| def _calculate_parts(self, file_size: int) -> Tuple[int, int]: |
Member
There was a problem hiding this comment.
This seems duplicated across async and non-async file managers
There was a problem hiding this comment.
We can consider making it function & reuse, the function uses no state
| file_size = os.stat(file.as_posix()).st_size | ||
| file_size_gb = file_size / NUM_BYTES_IN_GB | ||
|
|
||
| if file_size_gb > MAX_FILE_SIZE_GB: |
There was a problem hiding this comment.
Is this check a duplicate for the one in get_upload_url fn? Can we have one?
src/together/lib/resources/files.py
Outdated
| f"Unsupported file extension: '{file.suffix}'. Supported extensions: .jsonl, .parquet, .csv" | ||
| ) | ||
|
|
||
| def _calculate_parts(self, file_size: int) -> Tuple[int, int]: |
There was a problem hiding this comment.
We can consider making it function & reuse, the function uses no state
|
|
||
| return part_size, num_parts | ||
|
|
||
| def _get_file_type(self, file: Path) -> str: |
There was a problem hiding this comment.
Can be a function or we can make it static
This implementation adds comprehensive multipart upload functionality to support large dataset uploads up to 25GB, with automatic routing based on file size. - **Size-based routing**: Files >5GB automatically use multipart upload - **Concurrent uploads**: Up to 4 concurrent parts for optimal performance - **Progress tracking**: Real-time progress bars for multipart uploads - **Error handling**: Robust cleanup and retry mechanisms - **Integrity verification**: SHA256 hash validation for uploaded files - **Sync multipart**: MultipartUploadManager for synchronous uploads - **Async multipart**: AsyncMultipartUploadManager using ThreadPoolExecutor - **Consistent API**: Same upload_file() interface for both approaches - **Performance optimized**: Efficient concurrent part uploads
blainekasten
force-pushed
the
feature/multipart-upload
branch
from
ad4944f to
6fb42a1
Compare
blainekasten
added a commit
that referenced
this pull request
Nov 18, 2025
* chore(api): Remove auto-generated files upload API to support custom coded version * feat(api): api update * feat(api): file upload method signature and functionality match previ… (#174) * feat(api): file upload method signature and functionality match previous version * disable test * fix locks * ENG-41215: Client Multipart Upload implementation (together-py) (#159) * feat: Add multipart upload support for large files up to 25GB This implementation adds comprehensive multipart upload functionality to support large dataset uploads up to 25GB, with automatic routing based on file size. - **Size-based routing**: Files >5GB automatically use multipart upload - **Concurrent uploads**: Up to 4 concurrent parts for optimal performance - **Progress tracking**: Real-time progress bars for multipart uploads - **Error handling**: Robust cleanup and retry mechanisms - **Integrity verification**: SHA256 hash validation for uploaded files - **Sync multipart**: MultipartUploadManager for synchronous uploads - **Async multipart**: AsyncMultipartUploadManager using ThreadPoolExecutor - **Consistent API**: Same upload_file() interface for both approaches - **Performance optimized**: Efficient concurrent part uploads * Update codebase after recent changes * reduce duplicated code in sync/async upload managers --------- Co-authored-by: Blaine Kasten <blainekasten@gmail.com> * release: 0.1.0-alpha.28 --------- Co-authored-by: stainless-app[bot] <142633134+stainless-app[bot]@users.noreply.github.com> Co-authored-by: Blaine Kasten <blainekasten@gmail.com> Co-authored-by: Soroush <soroush.bassam@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
Implements concurrent multipart file uploads (sync/async) with larger file limits, updates docs/CLI/examples, and adds tests for file upload.
MultipartUploadManagerandAsyncMultipartUploadManagerwith initiate/part PUT/complete/abort flow, ETag handling, progress bars, and error handling.UploadManager/AsyncUploadManagerto auto-select multipart vs single-part based on size; improve presigned URL handling and validations.MAX_FILE_SIZE_GBto25.0; introduce configurable multipart constants (part size/timeout/concurrency/threshold).files uploadto passfile=keyword to client.src/together/resources/files.py(sync/async) and improve header/redirect logic.purpose="fine-tune".examples/file-upload.pyand dataset samples (examples/coqa*.jsonl).test_files_resource.pyfor upload flow (presign → PUT → preprocess).Written by Cursor Bugbot for commit 6fb42a1. This will update automatically on new commits. Configure here.