[Record Movies] Enable SharedArrayBuffer for non-dev environment

[schoinh]

Use Case

As a developer, I want to enable SharedArrayBuffer so I can use ffmpeg.wasm to convert webm video output to mp4 format. (See note under "Browser" in the ffmpeg readme.) In a dev environment, I can just supply the necessary CORS headers to webpack.dev.js like this:

    devServer: {
        headers: {
            'Cross-Origin-Embedder-Policy': 'require-corp',
            "Cross-Origin-Opener-Policy": "same-origin"
        }
    },

But I'm not sure how to enable it for non-dev environment.

Acceptance Criteria

I can go to https://simularium.github.io/simularium-viewer/, record a video, and download the video in mp4 format (which has been converted from webm to mp4 with ffmpeg.wasm).

Details

MDN: SharedArrayBuffer - Security requirements

[interim17]

I've played around with MediaRecorder API a bit before and was able to natively output directly to mp4 without a need to use ffmpeg to convert. Currently trying to understand why this was built to output webm then convert to mp4, maybe something I'm missing, but we could circumvent this issue if direct to mp4 is possible. I'll try it.

[toloudis]

@interim17 see this draft PR in case it helps #251
It looks like maybe webm was the preferred mime type to specify in the MediaRecorder?

[interim17]

State of the issue: CORS and Shared Array Buffer.

tl;dr we can either output WEBM instead of mp4 or we probably need to host ffmpeg on our own server and i will need backend/devops help with that

SharedArrayBuffer: allows the browser to implement some of the capabilities of multi-threaded languages via WebAssembly ports of libraries. For our purposes, the relevant library is ffmpeg.wasm, which does the video conversions we need. We want users to be able to download mp4 video of their trajectories, and MediaRecorder can output mp4 but not with the right codecs, it really wants to put out WEBM for video.

SharedArrayBuffer introduces security risks, and so is disabled in Chrome (and other browsers) unless cross origin isolation is enabled. Not allowing cross origin requests makes using SharedArrayBuffer safe because we know that we will not respond to malicious third party APIs and domains that make requests to our domain.

We can achieve cross origin isolation by applying certain headers to our site on the server side.

That being said, benevolent domains and APIs will also be restricted. I don’t know that the Simularium website requires a lot of cross origin requests at this point. A list of potential features/tools that could be interrupted:
- loading trajectories from any storage other than our primary domain
- MediaRecorder API (more below)
- embedded images on the front page from other websites
- analytics and metrics that use APIs
- as yet unanticipated things

The non-technical question at the heart of the issue: How much effort and engineering is this feature, outputting mp4 video, worth?

Options

1. One low effort workaround would be to simply output WEBM videos and let users worry about converting them to their desired formats. This is the only pure front end dev solution I have right now.

2. Another option that avoids CORS problems is to host ffmpeg on our own server. We should be able to access it as an API within our domain, but will have to host and maintain that service, will take a bit of testing. This becomes somewhat of a backend/dev-ops issue.

3. Really, changing the CORS headings will also be a backend/dev-ops issue. I am happy to pursue it, but I think I may need to muck around in AWS and other places where I could cause trouble. There are some low-consequence steps we can take if we go this route. There are “report-only” CORS headings that will help us figure out what services/features would be interrupted by a change in CORS headers without actually applying that change.

The Achilles heel of this option: the existing solution for grabbing these data blobs in the first place is the MediaRecorder API. It’s what the previous dev implemented, and what I would have done. I’m pretty sure that cross origin isolation might prevent this from working, which means the whole CORS change is moot and we are back to options 1 and 2.

[blairlyons]

@interim17 thanks for finding all the great info! The main issue with WEBM is that it's not supported by Powerpoint, which is the main use case we imagine for recording movies, so you can put them in a presentation. We don't want to make users have to convert first. Maybe if we found a service we could recommend to them that is super easy to use, then that could work. But we should probably investigate setting up our own ffmpeg service. Maybe @ascibisz could look into this as far as the back end requirements?

[interim17]

I'll see what @ascibisz and I can figure out about hosting if we put our heads together a bit, I like that route and would like to learn more about our backend.

[jsavieo]

Reaching out to Andy for help on CORS

[anl]

Let me know how I can help, either with CORS or with hosting.

[blairlyons]

using another method