external_storage: fix GCS download error, support GCS endpoints, and refactoring (#7734)

[sre-bot]

cherry-pick #7734 to release-3.1

---

What problem does this PR solve?

Issue Number: close #7625

Problem Summary:

It was found that the GCS external_storage can only download 8 KB of data. This PR fixes the issue.

Additionally, tame_gcs hard-coded the request URLs, making the custom endpoint ineffective (EmbarkStudios/tame-gcs#21). This PR also workarounds the issue through URL rewriting.

What is changed and how it works?

What's Changed:

The problem is caused by using block_on_external_io recursively. This seems to break Tokio's executor and caused the download stream to end after a single call of poll_read (where the buffer size is 8 KB).

This PR fixes the problem by removing all nested block_on_external_io and replace them by async fn.

Related changes

• Need to cherry-pick to the release branch
  • need to cherry-pick to release-4.0.

Check List

Tests

• Manual test (add detailed scripts or steps below)
  • Restore through fake-gcs-server works.

Side effects

Release note

Restoring a backup archive from GCS now works.

[sre-bot]

/run-all-tests

[kennytm]

/run-integration-ddl-test

/run-integration-compatibility-test

[5kbpers]

/merge

[sre-bot]

Your auto merge job has been accepted, waiting for:

• 7905

[sre-bot]

/run-all-tests

[5kbpers]

/merge

[5kbpers]

/merge

[5kbpers]

/merge

[sre-bot]

/run-all-tests

[sre-bot]

@sre-bot merge failed.

[kennytm]

Because of the flipped merge order, the essence of this cherry-pick has been contained in #7965 already. So this PR is much less urgent now 🤷

[5kbpers]

/merge

[sre-bot]

/run-all-tests

[sre-bot]

@sre-bot merge failed.

[5kbpers]

/merge

[5kbpers]

/merge

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 3 committers have signed the CLA.

❌ kennytm
❌ 5kbpers
❌ sre-bot
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[kennytm]

release-3.1 is no longer maintained, closing.