Direct upload to Azure/Local

[MGibson1]

Overview

This PR is the second half of #1174. It creates endpoints to allow direct upload to Azure blob storage and local storage. It also provides renewed upload URLs if permissions expire, validates file upload sizes, and locks down Send writes once size is validated.

Files Changed

• Api.csproj: Add event grid nuget dependency. This package allows for easier integration with Azure Event Grid, which is used to verify Send file sizes.
• SendsController.cs: New endpoints:
  • PostFile (v2): Creates a file type send and get a URL/SendResponse to upload the file to. FileUploadType is used to specify Azure vs Direct upload. This request makes and validates a send based on a promised file size. This is checked upon actual file upload.
  • RenewFileUploadUrl: Verifies a Send exists, belongs to the user, and has not yet been written (validated == false), then provides a new upload URL to allow for extended file operations.
  • PostFileForExistingSend: The API endpoint for localStorage deployments. This maintains the 100MB upload limit, uploads the file to local storage. SendService also validates the length of the received file stream is within a grace size.
  • AzureValidateFile: Azure event webhook endpoint. Accepts Azure Event Grid Event POSTs and handles blobCreate events, validating the final blob size. This method blocks the client's return from blob create writes.
• ApiHelpers.cs: Azure requires a response verification code to prove we own the webhook endpoint we specify. This helper automatically handles Verification events and accepts a Dictionary of event handlers to call for other Azure events.
• MultipartFormDataHelper.cs: Add overload to GetSendFileAsync where formData contains only file data now since the URL has the send Id and file ID that have already been created. The old overload still exists to support old clients.
• FileUploadType.cs: Specifies the method of uploading a file. Direct is direct to Bitwarden. Azure is an Azure Upload. This is used by the client to upload in the appropriate manner.
• SendRequestModel.cs: Optionally provide a hint of the File Size that will be uploaded for this Send. The FileLength is required for File Sends and ignored for Text sends.
• SendFileUploadDataResponseModel: Specifies the Send URL, UploadType, and Generated Send. This is the response to a request to make a new File Type Send.
• SendFileData: Add Validated bool to data model. This defaults to true so existing sends cannot be updated by requesting an upload URL for them. Send Data Creations must, unfortunately, specify this value as false upon instantiating new SendFileData.
• ISendService/SendService.cs:
  • FileUploadType: specify the file upload method to the client. Current options are Direct and Azure.
  • SaveFileSendAsync: Creates a File Send without file data and responds with a string to upload the file data to.
  • UploadFileToExistingSend: The local storage endpoint. Should not be used for Azure deployments. Validates existence of Send, stores the stream, and validates its size prior to returning. Throws in the event of a size mismatch since the client will correctly interpret that throw as a toast error.
  • ValidateSendFile.cs: Simply validates the specified file and deletes the send if it does not mesh within a reasonable leeway (set to 1MB). Updates Send's data with size and sets validation to true.
• AzureSendFileStorageService.cs:
  • GetSendUploadUrl: Uses SAS string to authorize a 1 min create permission to the specified blob. This requires a few Azure settings
    1. CORS for blob storage needs to be set to allow any origin PUT
    2. The sendfiles container should have permissions set to off
  • ValidateFile: Called from the Azure event webhook upon blob creation (which occurs after the file is finished uploading). Pulls the blob's size and sets some metadata/property attributes that could not be set prior to creation. We could offload this to the clients, but it makes sense to keep this file organization stuff in the server. If the size is not roughly equal to expected, returns false to indicate the Send should be removed due to un-validated file size. Always returns size of the blob (null if not found). This allows SendService to do some additional work updating db
• LocalSendFileStorageService: Validates that file on disk equals expected file size. We need to write to disk first since HttpStreams have no known length until they're read and we don't want to risk storing a huge file in memory to figure out the size.
• NoopSendFileStorageService: Noop implementations.

Required Azure Changes

1. CORS for blob storage needs to be set to allow any orgin for GET and PUT methods (upload is PUT only, but GET is needed for the download portion).
2. The sendfiles container should have permissions set to off
3. Specify webhook event for blob creation with filters defined by @joseph-flinn. Endpoint is <<apiUrl>>/sends/file/validate/azure

Testing requirements

Need to test upload and download to both local and Azure hosted environments.

Need to test using both old and new client, to make sure original Send upload implementation is not broken.

I've tested lying about file size, but a second set of eyes on that is a good idea.

The above require access to bitwarden/web#875 for the new client implementation

[cscharf]

Looking good, just a few comments