fix(mrf): attachment v2

[KenLSM]

Problem

Encrypted content of MRF with attachments is huge as it contains the attachments contents.

Solution

Overview

First, we introduce a new backward-compatibility key mrfVersion in the multirespondent submission object. This is initialized as 1 for all new MRF submissions. Here are the differences between the two values.

Old version: mrfVersion === undefined

• On submission
  • Response + attachments is saved to DB encrypted with submission key.
  • Attachment is also saved to S3 encrypted with form key.
• On retrieval
  • Respondent flow (R2 onwards): response + attachment is retrieved from DB and decrypted with submission key.
  • Admin flow (individual response page, csv download): attachment is retrieved from S3 and decrypted with form key.

New version: mrfVersion === 1

• On submission
  • Response is saved to DB encrypted with submission key.
  • Attachment is saved to S3 encrypted with submission key.
• On retrieval
  • Respondent flow (R2 onwards): response is retrieved from DB and decrypted with submission key. attachment is retrieved from S3 and decrypted with submission key.
  • Admin flow (individual response page, csv download): attachment is retrieved from S3 and decrypted with form key (via encrypted submission key).

Changelog

Therefore, this PR does the following changes.

• On submission
  1. Exclude attachment contents in the saved response encryptedContent
  2. Encrypt attachments using the submission key instead.
• On retrieval (all of these must be backward compatible, thus gated by mrfVersion === 1 check)
  • Respondent flow
    1. a new type MultirespondentSubmissionDtoWithAttachments is created to allow for rehydration of the encrypted attachments
    2. Add rehydration of attachments during getMultirespondentSubmissionById. (This will slow down the initial load, but provides compatibility quickly. If the UX is slow, it might make sense for us to provide a loading screen to handle this.)
    3. evaluation of which attachment to attach as defaults to the existing form is streamlined into the useEffect in PublicFormProvider, via the previousAttachments variable, based on whether mrfVersion === 1
  • Admin flow
    1. Change Admin Response Page to use Submission Key to decrypt files for mrfVersion === 1 responses, since attachments are now encrypted with a different key

Breaking Changes

• No - this PR is backwards compatible
  • Backwards compatibility is handled through mrfVersion: 1 on the encryptedContent.

Tests

Before deployment tests

Prepare for backward compatibility tests

• Create an MRF with an attachment field (of less than 4MB).
• Make a new submission with an attachment. This should be successful.
• Run the following attachment loading tests:
  • In the admin individual response page, download the attachment. This should be successful.
  • In the admin individual response page, download all attachments as ZIP. This should be successful.
  • In the admin panel, export all form responses as CSV with attachments. This should be successful.
  • As respondent 2, load the public form. Download the attachment from the form page. This should be successful.

After deployment tests

Backward compatibility tests

For the submission that was made earlier,

• Run the attachment loading tests defined above

End-to-end tests for the new protocol

• Now, make a new submission with an attachment. This should be successful.
• Run the attachment loading tests defined above
• As respondent 2, submit the form with the default attachment. This should be successful.
• Run the attachment loading tests defined above again

Ensure that attachments up to 20MB are supported
(note: this is not testable on staging as a result of infra-level limitations. test this on production as a manual test once the new version is released)

• Adjust the form to allow two attachments of up to 10MB each
• Submit two 9MB attachments. This should be successful.
• Run the attachment loading tests defined above again

[KenLSM]

@justynoh LGTM for the admin changes! I couldn't approve this PR myself

[justynoh]

lgtm'ed by ken