Recording: replace Kurento as the recording adapter

[prlanzarin]

Is your feature request related to a problem? Please describe.

Kurento is being used as a recorder for mediasoup as an intermediate solution.
It needs to be replaced by something else that is 1) lightweight/low footprint 2) mantained 3) portable

Describe the solution you'd like

There are few options. Those that I have in mind revolve around using an "external" RTP endpoint provider
that would receive streams from mediasoup and then mux those into media containers (always preferably webm or mkv).
Being an RTP endpoint means it would handle packet loss/jitter/et al. The inner workings of such an endpoint provider largely
depends on which platform is chosen (ie.: would it be a decoupled, controllable service with an HTTP interface? would it
be controlled by webrtc-sfu via processes or FFIs? ...)

The alternative would be to extend mediasoup itself to do that. The problem is that mediasoup transports do not
act as endpoints (AFAIK), so it would not be just a straightforward "dump the producer's streams into a container" because
everything an endpoint does (packet loss/jitter/et al.) would need to be accounted for.

Describe alternatives you've considered

Focusing on the RTP endpoint approach:

• ffmpeg: that would be a good option if it worked reliably enough since we already ship ffmpeg with BBB. It wouldn't
  be that much of pain to use via a Node.js application. That was prototyped and can currently be enabled in webrtc-sfu. It is controlled via ffmpeg processes, so high level and not that flexible. Very lightweight, though. There are a few problems with the ffmpeg prototype that might be not so straightforward to fix.
  • Packet loss handling: current prototype isn't handling rtcp-fb properly, meaning packet loss handling is flaky. That brings the need to manually handle keyframe requests by parsing ffmpeg processes' output (doable but not sure how stable that would be) or just figure out whether rtcp-fb can be bolted in/activated in ffmpeg when re-muxing a live RTP/SDP stream.
  • First frame wallclock time: have not figured out a way to easily infer that info yet, meaning recordings are prone to heavy desync since BBB recording start/stop events' timestamps need to be as accurate as they can be in relation to the frame wallclock time.
• gstreamer: I'd not exactly call this lightweight, but I'm sure it could work quite reliably. Kurento is basically an assortment of gstreamer elements (plus some custom elements that stitch a bunch of gstreamer elements together). The idea would be to basically reconstruct a minimal gstreamer pipeline that act as a RTP endpoint->muxer which would closely resemble what Kurento does minus the appendages.
  • That could be isolated as a separate service that would be controlled by webrtc-sfu through RPC interfaces (either through HTTP or redis).
  • It could also be implemented in a bunch of languages (any that have acceptable gstreamer interfaces, really; Java, C, C++, Rust, Python, ...)
  • The muxer part of Kurento is quite lightweight so I'd suppose doing this with gstreamer would not bring any performance issues (unless someone wants to record 1000 streams in a single host which is not a realistic scenario)
• Pion: as far as I know RTP endpoint->muxing is quite doable with Pion, which is Go-based so it would be fairly straightforward to build as a separate service. Idea is pretty much the same as the aforementioned so won't delve on it further.

Additional information

Stems from #12894 (comment)

[ffdixon]

Thanks Paulo for (as usual) a very well written issue that provides a model for others.

[alfredonodo]

Any update on this? Thank you

[prlanzarin]

SFU/Recorder integration draft

• This is a generic API draft for a 3rd-party recorder app
• API transport: Redis pubsub (initially)
• Payload format: JSON
• PUBSUB channels (POV recorder):
  • pub: `from-bbb-webrtc-recorder`,
  • sub: `to-bbb-webrtc-recorder` (provisional)
    • These need to be configurable in the application
• Base recording file path needs to be configurable in the
  application

Recorder API (draft)

startRecording (SFU -> Recorder)

{
  id: 'startRecording',
  recordingSessionId: 'aebcdf', // <String>, requester-defined - error out if collision
  sdp: 'base64offer', // <String>, Base64 SDP offer ({ type: 'offer', sdp })
  fileName: 'screenshare/meetingId/etc.webm', // <String>, file name INCLUDING format (e.g.: .webm), relative
}

startRecordingResponse (Recorder -> SFU)

Notes

• fileName: undefined if status === 'failed', otherwise it is the absolute recording filename (including the parent recording directory defined in the recorder application)
• sdp: undefined if status === 'failed', Base64 SDP answer otherwise
• error: undefined if status === 'ok', String otherwise

{
  id: 'startRecordingResponse',
  recordingSessionId: 'aebcdf',
  fileName: '/var/recorder/screenshare/meetingId/etc.webm', // <String | undefined>, *absolute file name*
  status: 'ok|failed',
  sdp: 'base64answer', // <String | undefined>, Base64 SDP answer ({ type: 'answer', sdp })
  error: 'etc', // <String | undefined>, implementer-defined
}

recordingRtpStatusChanged (Recorder -> SFU)

Notes

• This event should only be emitted on status state transitions

{
  id: 'recordingRtpStatusChanged', // media started or stopped flowing
  status: 'flowing|not_flowing',
  recordingSessionId: 'aebcdf',
timestampUTC: '1676487150880', // last written frame timestamp, UTC, wall clock
  timestampHR: '22668783783' // last written frame timestamp, system, high resolution
}

stopRecording (SFU -> Recorder)

{
  id: 'stopRecording',
  recordingSessionId: 'aebcdf',
}

recordingStopped (Recorder -> SFU)

{
  id: 'recordingStopped',
  recordingSessionId: 'aebcdf',
  reason: 'SOME_TERMINATION_REASON', // implementer-defined
  timestampUTC: '1676487150880', // last written frame timestamp, UTC, wall clock
  timestampHR: '22668783783' // last written frame timestamp, system, high resolution
}

Example sequence diagrams

[prlanzarin]

For tracking, a few things I'll looking into currently in the new recorder - a few benign, others breaking:

• RAP processing is currently borked due to a few issues in the raw files
  • Missing SeekHead section in webm files (~)
  • Missing or borked Cues section
  • Audio tracks are created even if there's no audio
  • Borked media file permissions

Things I plan on looking into afterwards, shortest term:

• Crash @ Jitterbuffer.Add@L26 - invalid index
• Reimplement PLI/keyframe request triggers
• H.264 support
• VP9 support (Add VP9 codec support to BBB #8869)
• Mem consumption
• Review logging and config system
• TBD...

[prlanzarin]

The replacement is already usable under a feature flag in 2.7 and is the default in the 3.0 branch.
I consider this done. I'm starting to roll field trials on my end - any subsequent bugs found by anyone should be reported via separate issues as done for other shipped features.