[FEATURE] Download backing images

[guangbochen]

Is your feature request related to a problem? Please describe.
Allowing users to download the backing images from UI, so that those images can be shared accordingly.

[innobead]

Hey team! Please add your planning poker estimate with ZenHub @jenting @PhanLe1010 @shuo-wu @joshimoo

[shuo-wu]

There are 2 major parts of this feature. The 1st one is how we can get the file data:

• The backing image manager gRPC server can manually split a file into multiple data chunks and send them to the longhorn manager API via a gRPC streaming.
• The backing image manager gRPC server can rely on a gRPC call launching a temporary HTTP server then return the URL of the HTTP server.
• Longhorn can use a new CRD to indicate the creation of a backing image "download" pod that launches a HTTP server (then expose the service if necessary).

The 2nd part of this feature is how users can download the data:

• If the gRPC streaming is applied, we need to create a new HTTP API similar to the support bundle downloading. This new API should be responsible for launching the gRPC streaming, reading the chunks from the streaming, then writing data to the HTTP (streaming) response.
• If the backing image manager will expose a temporary HTTP server, it means longhorn manager should send the gRPC call to a backing image manager, wait for the HTTP service up, then proxy it.
• If a dedicated backing image "download" pod can be launched via a new CRD (and its controller), the longhorn manager can use one HTTP API to help create the CR then expose another HTTP API to proxy it.
  Or there is no need to do proxy. Instead, we can directly put the service info in the new CR status, then users themselves are responsible for expose the service.

@joshimoo Can you provide some suggestions about this? I am not sure which one I should apply. Or are there any better ideas?

cc @innobead

[innobead]

@shuo-wu will you start this in this sprint? or move to the next sprint?

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at: Update CRDs after backing image refactoring #4050
  The PR for the chart change is at: Update CRDs after backing image refactoring #4050

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at
  Introduce file sync service & refactoring & backing image download implementation backing-image-manager#56
  Update backing image related implementation & Add backing image download feature longhorn-manager#1347

• Which areas/issues this PR might have potential impacts on?
  Area: backing image

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at feat(backing-image) Support download backing image longhorn-ui#519

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at
  The automation test case PR is at
  The issue of automation test case implementation is at (please create by the template)

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at doc: Add a test for the backing image download feature longhorn-tests#985

[innobead]

We only have a doc about creating volumes from backing images, but along with the completeness of backing image functions, it's time to add some other doc to introduce other functions like how to create a backing image from upload/remote download, create a backing image from a volume, download, etc.

[yangchiu]

verified pass on v1.3.0-rc3 by following manual test steps