Implement async storage option for DevWorkspaces

[amisevsk]

Description

"Async storage" as used by Che requires

• A shared "sync" server deployment that mounts a PVC and listens via ssh
• A sidecar "sync" container to be injected into workspaces to rsync files from the workspace to the storage server

This allows workspaces to avoid issues around backing storage for PVCs (e.g. Gluster volumes have trouble synchronizing when many files are touched -- e.g. in javascript .node_modules) while also providing persistence (unlike ephemeral volumes).

We should reuse the sync components used by Che to implement this in the DevWorkspace operator.

Requirements

A basic implementation of async storage would

• Manage the creation of ssh keypairs to allow the sidecar and storage server to communicate
• Manage a per-namespace async storage deployment + service in namespaces where workspaces exist
• Provision a sidecar container, secret, etc. for DevWorkspaces
• Correctly manage cleanup when workspaces are deleted:
  • Remove unowned resources when appropriate (ssh key for deleted workspace)
  • Clean up PVC when a workspace is deleted
  • Remove async deployment when all workspaces are deleted

Scope

For the first implementation, we should target

• One workspace per namespace
• Async deployment is removed when workspace is deleted
• PVC cleanup runs as it does for the common strategy (depends on async deployment being removed)

This is mainly to avoid edge cases in managing out-of-devworkspace resources when there are multiple devworkspaces:

• What happens if a secret is deleted and we have to regenerate an ssh keypair?
• How do we clean up a subpath in the PVC when it's mounted by the async server (if e.g. two workspaces are running and one is deleted)
• How do we keep the authorized_keys used by the sync server in sync with existing workspaces?

Additional info

• workspace-data-sync repo
• async storage sidecar image
• async storage server image

[sleshchenko]

Please add the expected way how users will configure async storage for volume or for whole devfile.

[amisevsk]

Please add the expected way how users will configure async storage for volume or for whole devfile.

With a PR open now, I can answer this question :)

• Per volume configuration is not permitted; it does not make sense to have some volumes use common and other using async. Any volume can specify ephemeral: true, which is a separate option from storage type.
• Per devfile configuration is done via the annotation controller.devfile.io/storage-type, with common and async being currently supported (ephemeral to come)