Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large File Upload Task #530

Merged
merged 32 commits into from
May 2, 2024
Merged

Large File Upload Task #530

merged 32 commits into from
May 2, 2024

Conversation

shemogumbe
Copy link
Collaborator

@shemogumbe shemogumbe commented Mar 25, 2024

Closes #112

What does this PR do

Enables uploading large files using the python SDK, as chunks of bytes

Demo

Upload_chuknks2

Usage and Testing Instructions

destination_path = "path/to/your_file.txt"
file_path = "path/to/your_file.txt"


async def upload_large_file():
    try:
        file = open(file_path, 'rb')
        uploadable_properties = DriveItemUploadableProperties(
            additional_data={'@microsoft.graph.conflictBehavior': 'replace'}
        )
        upload_session_request_body = CreateUploadSessionPostRequestBody(item=uploadable_properties)
        print(f"Uploadable Properties: {uploadable_properties.additional_data}")
        # can be used for normal drive uploads
        try:

            upload_session = await user_client.drives.by_drive_id(
                "b!WtUPKhiPm0a6Bj6Z_J97a53XIv_KoNNAkdLRUrSF06lh-qW6JpABSoW62oPNb03R"
            ).items.by_drive_item_id('root:/my_docs/test_upload.txt:'
                                     ).create_upload_session.post(upload_session_request_body)
            
        except APIError as ex:
            print(f"Error creating upload session: {ex}")

        # to be used for large file uploads
        large_file_upload_session = LargeFileUploadSession(
            upload_url=upload_session.upload_url,
            expiration_date_time=datetime.now() + timedelta(days=1),
            additional_data=upload_session.additional_data,
            is_cancelled=False,
            next_expected_ranges=upload_session.next_expected_ranges
        )

        # max_slice_size = 320 * 1024 - remove to use default
        task = LargeFileUploadTask(large_file_upload_session, user_client.request_adapter, file)
        total_length = os.path.getsize(file_path)
        
        # Upload the file
        # The callback
        def progress_callback(uploaded_byte_range: tuple[int, int]):
            print(f"Uploaded {uploaded_byte_range[0]} bytes of {total_length} bytes\n\n")

        try:
            upload_result = await task.upload(progress_callback)
            print(f"Upload complete {upload_result}")
        except APIError as ex:
            print(f"Error uploading: {ex.message} - {ex.response_status_code}")
    except APIError as e:
        print(f"Error: {e}")


asyncio.run(upload_large_file())
  • Install msgraphcore and msgraph-sdk
    *Create a script with appropriate authentication and scopes
  • Run the code above, with appropriate filepath and destination address
  • Check that your logs are as per the screenshot above
  • Visit the destination folder in ondedrive and counter check that the file is there, you can us echecksum to see if the files are identical

Copy link
Contributor

github-actions bot commented Apr 8, 2024

This pull request has conflicting changes, the author must resolve the conflicts before this pull request can be merged.

Copy link
Contributor

Conflicts have been resolved. A maintainer will take a look shortly.

@shemogumbe shemogumbe self-assigned this Apr 20, 2024
@shemogumbe shemogumbe marked this pull request as ready for review April 22, 2024 14:08
Copy link
Contributor

@Ndiritu Ndiritu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Some unit tests would also be helpful for the completion scenarios on Outlook vs OneDrive/Sharepoint

src/msgraph_core/tasks/large_file_upload.py Outdated Show resolved Hide resolved
src/msgraph_core/tasks/large_file_upload.py Outdated Show resolved Hide resolved
src/msgraph_core/tasks/large_file_upload.py Outdated Show resolved Hide resolved
src/msgraph_core/tasks/large_file_upload.py Outdated Show resolved Hide resolved
src/msgraph_core/tasks/large_file_upload.py Outdated Show resolved Hide resolved
@shemogumbe shemogumbe merged commit eb0fd78 into main May 2, 2024
11 checks passed
@shemogumbe shemogumbe deleted the shem/large_file_upload branch May 2, 2024 12:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Developers should be able to access file upload support on the Python SDK
3 participants