Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge More Main Changes to Azure Storage v12 Branch #930

Merged

Conversation

wsugarman
Copy link
Contributor

@wsugarman wsugarman commented Jul 28, 2023

  • Integrate latest changes from main including the new table partition manager into the feature branch
  • Refactor TablePartitionManager to use the Table class instead of the TableClient directly
  • Rename TableLease to TablePartitionLease to align with BlobPartitionLease
  • Some async testing refactors

This change should not be squashed as I will eventually need to merge with main too.

wsugarman and others added 16 commits April 22, 2023 13:33
…ons (Azure#892)

* [AzureServiceFabric] Handle missing ExecutionStartedEvent orchestrations

---------

Co-authored-by: Shankar Reddy Sama <shsama@microsoft.com>
…zure#896)

Co-authored-by: Shankar Reddy Sama <shsama@microsoft.com>
…zure#898)

* Bumping up DurableTask.AzureServicefabric package version to 2.3.8

* [AzureServiceFabric] Fix issues related to Terminate orchestration

* Fixed comments and usings

* Fixed remote client to read http error message

---------

Co-authored-by: Shankar Reddy Sama <shsama@microsoft.com>
…onAsync and fix session query in ContainsSessionAsync (Azure#900)

* [AzureServiceFabric] Fix ArgumentNullException in CompleteOrchestrationAsync

* Use OrchestrationInstance while finding the session to avoid conflicts

---------

Co-authored-by: Shankar Reddy Sama <shsama@microsoft.com>
… to 2.3.9 (Azure#902)

Co-authored-by: Shankar Reddy Sama <shsama@microsoft.com>
…es (Azure#891)

* add randomization to blob names when uploading history, to prevent races in split-brain situations

* track all the blobs associated with an execution, and remove them when updating the tracking store after a continue-as-new

* modify blobName slightly so existing tests still pass.

* address PR feedback

* address PR feedback.

* Apply suggestions from code review

Co-authored-by: David Justo <david.justo.1996@gmail.com>

* address PR feedback

---------

Co-authored-by: David Justo <david.justo.1996@gmail.com>
Update GetMessageAsync log information, add the reason LeaseLost.
* update

* remove unnecessart pkg

* update test

* update drain process && test storage connection

* update commit

* update scaletests pm setting

* udpate commit

* update commit

* Update AzureStorageScaleTests.cs

* test UseTablePartitionManager setting

* remove typo

* change UseTablePartitionManager = true

* update null reference

* add control queue in AzureOrchestratorService

* update commit

* Update AzureStorageScaleTests.cs

* commit KillThreeWorker tests

* update GetControlQueueAsync

* update partition table name setting

* update log && commits && add new method to check alignment with owned queues and ownership lease

* remove typo and sort using

* update commit

* add tests

* update according to commits

* change test connection setting

* update according to reviews

* mark one test disabled for now cause it's flaky

* Update DurableTask.AzureStorage.csproj

* Update DurableTask.AzureStorage.csproj

* reconstruct ReadAndWriteTable method to be more readable

* udpate connection string setting

* update connection string setting

* update test according to commits and update connection string setting

* update connection string setting

* update logs

* update typo

* update tests

* update catch exceptions and logs

* remove typo

* update lease balancer and log

* remove unused package

* update commits

* Update AzureStorageScaleTests.cs for commits

* update commit

* update commits: drain process

* update checkownershiplease from void to task

* mark test disabledinci

* update according to commit

* remove test version

* Suggested edits for table partition manager (Azure#921)

* update shutdown process to avoid race condition

* update shutdown process into one loop to avoid race condition

* add commit

* More suggested edits (Azure#922)

* More suggested edits

* Sync'd with latest and added additional refactoring

* Adding a few corrections to my previous commit

* add new method CheckDrainTask to avoid redundant code

* update commit

* update shutdown wait time && KillLoop

* update commit

* Update AppLeaseManager.cs

Remove unrelated file using change.

* update AzureStorage version

* update AzureStorage version

* update commit for AcquireInterval

* update commit for AcquireInterval

* update commit

---------

Co-authored-by: Chris Gillum <cgillum@microsoft.com>
@wsugarman wsugarman changed the base branch from main to azure-storage-v12 July 28, 2023 19:39
@cgillum
Copy link
Collaborator

cgillum commented Jul 28, 2023

Adding @nytian to help review this PR and answer Will’s questions.

Copy link
Collaborator

@nytian nytian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good to me.
I just have a problem about changing TableClient to Table class. Why are we doing this, I assume is it because we cannot apply Azure.Data.Tables packages?
Thanks!

@wsugarman
Copy link
Contributor Author

The change looks good to me. I just have a problem about changing TableClient to Table class. Why are we doing this, I assume is it because we cannot apply Azure.Data.Tables packages? Thanks!

Oh, I changed it as all other interactions with Azure Storage, including the existing partition manager, leverage these client wrappers. For example, the BlobPartitionManager used by the LegacyPartitionManager uses the internal Blob type. Probably the biggest advantage is that they are configured with various policies for retrying and logging connection attempts.

@nytian nytian merged commit ec179a2 into Azure:azure-storage-v12 Aug 3, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants