Custom netplay implementation

[n-at]

I've created a simple retro game library with built-in ability to play with EmulatorJS and built-in netplay: https://github.com/n-at/playtime

On the user's side, netplay works like this:

1. Game host uploads a game and enables netplay in the game settings:
   
   

2. The game host starts the game and opens the netplay menu. There are link and QR code that can be shared with other players. No registration is required to join:
   
   

3. Other players (clients) start as a spectators and cannot yet control the game. The game host can assign their player id (i.e. controller number):
   
   

4. Players (clients) in the netplay menu can also change their display names:
   

On the technical side, netplay is implemented using WebSocket and WebRTC:

• No netplay code from EmulatorJS is used (neither client or server), everything is custom. All emulation cores should be supported.
• WebSocket used to share state (clients connecting/disconnecting, player ids and names) and exchange messages between clients (for RTC signalling). Server-side code is written on Go and is mostly straightforward: https://github.com/n-at/playtime/blob/master/web/handlers_netplay.go#L54
• Audio and video are streamed from the game host (i.e. owner of the game, the only one who has access to the game ROM and emulation settings). The video stream is captured from the canvas and the audio stream is from AL gain nodes. Stream is one-directional: from the game host to clients.
• Control, i.e. passing the controller state from the client to the host, including button ids and values, is done with RTC Data Channels. It is a direct connection, so input lag will be minimal. Although Data Channels are bidirectional, data is transmitted in only one direction: from the client to the game host.
• Client-side code is organized into three parts:
  • https://github.com/n-at/playtime/blob/master/assets/netplay.js - a common library where all netplay functionality is gathered (client state, WebSocket connection and RTC connection management)
  • https://github.com/n-at/playtime/blob/master/assets/netplay-ui-client.js - client code that connects the netplay library with the client UI (also contains custom virtual controller implementation)
  • https://github.com/n-at/playtime/blob/master/assets/netplay-ui-host.js - host code that connects the netplay library with game host UI (and emulator)

Maybe this can be helpful for an official netplay implementation.

[allancoding]

I really like this method of doing netplay. I have a question how is the audio captured? This is a cool project, I think this would be an awesome client to replace linuxserver.io docker image. If you would be ok with that. Another thing is how are the delays and what are the draw backs of always depending on the host. Like what is the extra performance on the host that is needed? I think the server code would need to be written in node.js but that would be not too hard. Do you have discord? If so would you mind joining the server for easier communication and collaboration.
Thanks

[ethanaobrien]

I've tried netplay from this angle, but it's never really turned out well, er, worked.
My main concern is webrtc. The old version of netplay (emulatorjs 3.0) used webrtc, and it "worked." The main issue of using both webrtc and websockets is that the webrtc connection would seem to randomly fail, leaving the websocket connection open, but the user disconnected.

My other concerns are latency and bandwidth. I have a goal for netplay to be optimized as much as possible, to be playable on 3G speeds, and to not have to have the worry of any data cap, which is why I haven't gone the approach of streaming the video.

Our current take on netplay is this. We're planning on basically reverse engineering how netplay works in retroarch and putting it in here. I think the main issue was that game state prediction is unimplimented at the moment, after that we should be able to optimize latency and delay as much as possible, the same way retroarch does it. @allancoding is assigned with this task, and I got no clue where the progress is at.

Now my question, how did you get an audio stream again? I've tried and tried but I've never been able to get an audio stream. Could you try recording it with a mediarecorder? Because that's my current goal that I haven't been able to accomplish

[n-at]

About audio capture. It's code here: https://github.com/n-at/playtime/blob/master/assets/netplay.js#L394 AL exposes GainNode in AL.currentCtx.audioCtx.sources - in EmulatorJS it used to control game volume, but it's output can also be connected to something else. I've connected it to the ChannelMergerNode (in case if there is multiple gain nodes) and merger's output connected to the MediaStreamAudioDestinationNode - it produces MediaStream that can be sent via WebRTC to clients. I think that the merger node is not really needed, and there will always be only one GainNode (so code may be simplified). There is one rare glitch - sometimes audio stream is empty and I have not yet found a cause (maybe audio capture started too early and not initialized properly).

MediaRecorder works. I just took function from netplay (with little edit) and passed collected MediaStream to recorder:

function collectMediaTracks() {
        let videoTrack = null;
        const videoTracks = document.querySelector(`${window.EJS_player} canvas`).captureStream().getVideoTracks();
        if (videoTracks.length !== 0) {
            videoTrack = videoTracks[0];
        } else {
            console.error('Unable to capture video stream');
            return null;
        }

        let audioTrack = null;
        if (window.AL && window.AL.currentCtx && window.AL.currentCtx.audioCtx) {
            const alContext = window.AL.currentCtx;
            const audioContext = alContext.audioCtx;

            const gainNodes = [];
            for (let sourceIdx in alContext.sources) {
                gainNodes.push(alContext.sources[sourceIdx].gain);
            }

            const merger = audioContext.createChannelMerger(gainNodes.length);
            gainNodes.forEach(node => node.connect(merger));

            const destination = audioContext.createMediaStreamDestination();
            merger.connect(destination);

            const audioTracks = destination.stream.getAudioTracks();
            if (audioTracks.length !== 0) {
                audioTrack = audioTracks[0];
            }
        }

        const stream = new MediaStream();
        if (videoTrack && videoTrack.readyState === 'live') {
            stream.addTrack(videoTrack);
        }
        if (audioTrack && audioTrack.readyState === 'live') {
            stream.addTrack(audioTrack);
        }
        return stream;
    }

    setTimeout(() => {
        const chunks = [];
        const recorder = new MediaRecorder(collectMediaTracks());
        recorder.addEventListener('dataavailable', e => {
            chunks.push(e.data);
        });
        recorder.addEventListener('stop', () => {
            const blob = new Blob(chunks);
            const url = URL.createObjectURL(blob);
            const a = document.createElement('a');
            a.href = url;
            a.download = 'video.webm';
            a.click();
        });
        recorder.start();

        setTimeout(() => { recorder.stop()}, 10000);
    }, 5000);

And this video been recorded:

video.webm

Video looks fine but the game canvas refresh rate reduced to 1-2 FPS during recording. Maybe the canvas should not be captured directly (for example use another invisible canvas of fixed size and paint game canvas image on it).

[allancoding]

depending on the host it will have to make a peer to peer connection to each client individually.

[ethanaobrien]

What

[n-at]

@n-at would you like to try integrating this with EmulatorJS netplay? Stripping out (what you need)? If you have any questions or need access to the current emulatorjs netplay repo (which I expect) just let me know.

Sorry, but I couldn't do this due to my current workload. Appreciate your understanding.

Also, do we need a webrtc turn server for this?

Yes, WebRTC requires a TURN/STUN/ICE server to connect peers over the internet.

[ethanaobrien]

Sorry, but I couldn't do this due to my current workload. Appreciate your understanding.

Thats alright, no problem there. I did get some other idea's I'd like to play around with for netplay so I'll let you know how those go

[n-at]

Video looks fine but the game canvas refresh rate reduced to 1-2 FPS during recording. Maybe the canvas should not be captured directly (for example use another invisible canvas of fixed size and paint game canvas image on it).

Capture video with additional canvas works. Game frame rate is good and video is recording.

function collectMediaTracks(canvasEl, fps) {
        let videoTrack = null;
        const videoTracks = canvasEl.captureStream(fps).getVideoTracks();
        if (videoTracks.length !== 0) {
            videoTrack = videoTracks[0];
        } else {
            console.error('Unable to capture video stream');
            return null;
        }

        let audioTrack = null;
        if (window.AL && window.AL.currentCtx && window.AL.currentCtx.audioCtx) {
            const alContext = window.AL.currentCtx;
            const audioContext = alContext.audioCtx;

            const gainNodes = [];
            for (let sourceIdx in alContext.sources) {
                gainNodes.push(alContext.sources[sourceIdx].gain);
            }

            const merger = audioContext.createChannelMerger(gainNodes.length);
            gainNodes.forEach(node => node.connect(merger));

            const destination = audioContext.createMediaStreamDestination();
            merger.connect(destination);

            const audioTracks = destination.stream.getAudioTracks();
            if (audioTracks.length !== 0) {
                audioTrack = audioTracks[0];
            }
        }

        const stream = new MediaStream();
        if (videoTrack && videoTrack.readyState === 'live') {
            stream.addTrack(videoTrack);
        }
        if (audioTrack && audioTrack.readyState === 'live') {
            stream.addTrack(audioTrack);
        }
        return stream;
    }

    const CAPTURE_SCREEN_WIDTH = 800;
    const CAPTURE_SCREEN_HEIGHT = 600;
    const CAPTURE_FPS = 30;

    const captureCanvas = document.createElement('canvas');
    captureCanvas.width = CAPTURE_SCREEN_WIDTH;
    captureCanvas.height = CAPTURE_SCREEN_HEIGHT;
    captureCanvas.style.position = 'absolute';
    captureCanvas.style.top = '-999px';
    captureCanvas.style.bottom = '-999px';
    document.getElementsByTagName('body')[0].append(captureCanvas);

    const captureCtx = captureCanvas.getContext('2d');
    captureCtx.fillStyle = '#000';

    setTimeout(() => {
        const gameCanvas = document.querySelector(`${window.EJS_player} canvas`);
        const interval = setInterval(() => {
            const scaleX = CAPTURE_SCREEN_WIDTH / gameCanvas.width;
            const scaleY = CAPTURE_SCREEN_HEIGHT / gameCanvas.height;
            const scale = Math.max(scaleY, scaleX);
            const width = gameCanvas.width * scale;
            const height = gameCanvas.height * scale;
            const startX = (CAPTURE_SCREEN_WIDTH - width) / 2;
            const startY = (CAPTURE_SCREEN_HEIGHT - height) / 2;
            captureCtx.drawImage(gameCanvas, startX, startY, width, height);
        }, 1000 / 60);

        const chunks = [];
        const recorder = new MediaRecorder(collectMediaTracks(captureCanvas, CAPTURE_FPS), {
            videoBitsPerSecond: 2 * 1024 * 1014,
            audioBitsPerSecond: 256 * 1024,
        });
        recorder.addEventListener('dataavailable', e => {
            chunks.push(e.data);
        });
        recorder.addEventListener('stop', () => {
            const blob = new Blob(chunks);
            const url = URL.createObjectURL(blob);
            const a = document.createElement('a');
            a.href = url;
            a.download = 'video.webm';
            a.click();

            clearInterval(interval);
            captureCanvas.remove();
        });
        recorder.start();

        setTimeout(() => { recorder.stop()}, 10000);
    }, 5000);

video.9.webm

[Cattn]

I also like this method of netplay, and some friend of mine & me have made a similar solution in the past (albiet much much more crude lol as it was meant for only about 4 of us), but without webRTC (due to complications)

Was very little compared to this