Skip to content

Commit

Permalink
Support for capturing screen + audio (#385)
Browse files Browse the repository at this point in the history
* Add AUDIO_SCREEN recording type

* Add sv-perm icon (for tests, differentiation)

* getRecorderMode: Improve conditionals to catch mixed modes first

* Record Screencast + Audio stream

This requires using .addTrack to grab the microphone from getUserMedia,
the ordering / context and so on matter.

* Add tests for AUDIO_SCREEN

* Add example for audio-screen

* README: Update for screen capturing

* add changelog entry
  • Loading branch information
tony authored and thijstriemstra committed Jul 16, 2019
1 parent e6da555 commit 0943657
Show file tree
Hide file tree
Showing 13 changed files with 232 additions and 24 deletions.
1 change: 1 addition & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ videojs-record changelog

- New ffmpeg.js plugin: convert recorded data into other audio/video file formats
in the browser (#201)
- Support for capturing screen + audio (#385 by @tony)
- Support for specifying third-party plugin settings using the `pluginLibraryOptions`
option (#383)
- New options: `videoBitRate` and `videoFrameRate` (currently only used in the
Expand Down
9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ Add the extra stylesheet for the plugin that includes a

### Audio/video/image

When recording either audio/video, video-only, animated GIF or a single image,
When recording either audio/video, video-only, screen-only, audio/screen, animated GIF or a single image,
include a `video` element:

```html
Expand Down Expand Up @@ -178,7 +178,8 @@ Examples
- audio-only example ([demo](https://collab-project.github.io/videojs-record/examples/audio-only.html) / [source](https://github.com/collab-project/videojs-record/blob/master/examples/audio-only.html))
- image ([demo](https://collab-project.github.io/videojs-record/examples/image-only.html) / [source](https://github.com/collab-project/videojs-record/blob/master/examples/image-only.html))
- animated GIF ([demo](https://collab-project.github.io/videojs-record/examples/animated-gif.html) / [source](https://github.com/collab-project/videojs-record/blob/master/examples/animated-gif.html))
- screen capture ([demo](https://collab-project.github.io/videojs-record/examples/screen-only.html) / [source](https://github.com/collab-project/videojs-record/blob/master/examples/screen-only.html))
- screen-only ([demo](https://collab-project.github.io/videojs-record/examples/screen-only.html) / [source](https://github.com/collab-project/videojs-record/blob/master/examples/screen-only.html))
- audio/screen ([demo](https://collab-project.github.io/videojs-record/examples/audio-screen.html) / [source](https://github.com/collab-project/videojs-record/blob/master/examples/audio-screen.html))

To try out the examples locally, download the [zip-file](https://github.com/collab-project/videojs-record/archive/master.zip)
and unpack it, or checkout the repository using Git:
Expand Down Expand Up @@ -245,7 +246,7 @@ The available options for this plugin are:
| `audio` | boolean or object | `false` | Include audio in the recorded clip. |
| `video` | boolean or object | `false` | Include video in the recorded clip. |
| `animation` | boolean or object | `false` | Animated GIF without audio. |
| `screen` | boolean or object | `false` | Screen capture without audio. |
| `screen` | boolean or object | `false` | Include screen capture in the recorded clip. |
| `debug` | boolean | `false` | Enables console log messages during recording for debugging purposes. |
| `pip` | boolean | `false` | Enables [Picture-in-Picture support](#picture-in-picture). Enable to add Picture-in-Picture button to controlbar. |
| `maxLength` | float | `10` | Maximum length of the recorded clip. |
Expand Down Expand Up @@ -293,7 +294,7 @@ player.record().destroy();
| Method | Description |
| --- | --- |
| `isRecording` | Returns a boolean indicating whether recording is active or not. |
| `getRecordType` | Get recorder type as string. Either `image_only`, `animation`, `screen_only`, `audio_only`, `video_only` or `audio_video`. |
| `getRecordType` | Get recorder type as string. Either `image_only`, `animation`, `audio_only`, `video_only`, `audio_video`, `screen_only` or `audio_screen`. |
| `saveAs` | Show save as dialog in browser so the user can [store the recorded media locally](#save-data). |
| `destroy` | Destroys the recorder instance and children (including the video.js player). |
| `reset` | Not as destructive as `destroy`: use this if you want to reset the player interface and recorder state. |
Expand Down
84 changes: 84 additions & 0 deletions examples/audio-screen.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>Audio/Screen Example - Record Plugin for Video.js</title>

<link href="../node_modules/video.js/dist/video-js.min.css" rel="stylesheet">
<link href="../dist/css/videojs.record.css" rel="stylesheet">
<link href="assets/css/examples.css" rel="stylesheet">

<script src="../node_modules/video.js/dist/video.min.js"></script>
<script src="../node_modules/recordrtc/RecordRTC.js"></script>
<script src="../node_modules/webrtc-adapter/out/adapter.js"></script>

<script src="../dist/videojs.record.js"></script>

<script src="browser-workarounds.js"></script>

<style>
/* change player background color */
#myVideo {
background-color: #9ab87a;
}
</style>
</head>
<body>

<video id="myScreenAudio" playsinline class="video-js vjs-default-skin"></video>

<script>
var options = {
controls: true,
width: 800,
height: 450,
fluid: false,
controlBar: {
volumePanel: false,
fullscreenToggle: false
},
plugins: {
record: {
audio: true,
screen: true,
debug: true
}
}
};

// apply some workarounds for opera browser
applyVideoWorkaround();
applyScreenWorkaround();

var player = videojs('myScreenAudio', options, function() {
// print version information at startup
var msg = 'Using video.js ' + videojs.VERSION +
' with videojs-record ' + videojs.getPluginVersion('record') +
' and recordrtc ' + RecordRTC.version;
videojs.log(msg);
});

// error handling
player.on('deviceError', function() {
console.log('device error:', player.deviceErrorCode);
});

player.on('error', function(element, error) {
console.error(error);
});

// user clicked the record button and started recording
player.on('startRecord', function() {
console.log('started recording!');
});

// user completed recording and stream is available
player.on('finishRecord', function() {
// the blob object contains the recorded data that
// can be downloaded by the user, stored on server etc.
console.log('screen+audio capture ready: ', player.recordedData);
});
</script>

</body>
</html>
1 change: 1 addition & 0 deletions src/css/_icons.scss
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ $icon-font-family: videojs-record;
// http://sass-lang.com/documentation/file.SASS_REFERENCE.html#maps
$icons: (
av-perm: 'f101',
sv-perm: 'f104',
video-perm: 'f102',
audio-perm: 'f103',
screen-perm: 'f104',
Expand Down
1 change: 1 addition & 0 deletions src/css/components/device-button.scss
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
}

.vjs-record button.vjs-device-button.vjs-control.vjs-icon-av-perm:before,
.vjs-record button.vjs-device-button.vjs-control.vjs-icon-sv-perm:before,
.vjs-record button.vjs-device-button.vjs-control.vjs-icon-audio-perm:before,
.vjs-record button.vjs-device-button.vjs-control.vjs-icon-video-perm:before,
.vjs-record button.vjs-device-button.vjs-control.vjs-icon-screen-perm:before,
Expand Down
4 changes: 4 additions & 0 deletions src/fonts/icons.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@
"name": "av-perm",
"svg": "action/svg/production/ic_perm_camera_mic_48px.svg"
},
{
"name": "sv-perm",
"svg": "communication/svg/production/ic_screen_share_48px.svg"
},
{
"name": "video-perm",
"svg": "av/svg/production/ic_videocam_48px.svg"
Expand Down
3 changes: 3 additions & 0 deletions src/fonts/videojs-record.svg
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
14 changes: 9 additions & 5 deletions src/js/engine/record-mode.js
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ const IMAGE_ONLY = 'image_only';
const AUDIO_ONLY = 'audio_only';
const VIDEO_ONLY = 'video_only';
const AUDIO_VIDEO = 'audio_video';
const AUDIO_SCREEN = 'audio_screen';
const ANIMATION = 'animation';
const SCREEN_ONLY = 'screen_only';

Expand All @@ -19,15 +20,18 @@ const getRecorderMode = function(image, audio, video, animation, screen) {
} else if (isModeEnabled(animation)) {
return ANIMATION;

} else if (isModeEnabled(screen)) {
} else if (isModeEnabled(audio) && isModeEnabled(video)) {
return AUDIO_VIDEO;

} else if (isModeEnabled(audio) && isModeEnabled(screen)) {
return AUDIO_SCREEN;

} else if (!isModeEnabled(audio) && isModeEnabled(screen)) {
return SCREEN_ONLY;

} else if (isModeEnabled(audio) && !isModeEnabled(video)) {
return AUDIO_ONLY;

} else if (isModeEnabled(audio) && isModeEnabled(video)) {
return AUDIO_VIDEO;

} else if (!isModeEnabled(audio) && isModeEnabled(video)) {
return VIDEO_ONLY;
}
Expand All @@ -46,5 +50,5 @@ const isModeEnabled = function(mode) {

export {
getRecorderMode,
IMAGE_ONLY, AUDIO_ONLY, VIDEO_ONLY, AUDIO_VIDEO, ANIMATION, SCREEN_ONLY
IMAGE_ONLY, AUDIO_ONLY, VIDEO_ONLY, AUDIO_VIDEO, ANIMATION, SCREEN_ONLY, AUDIO_SCREEN
};
3 changes: 2 additions & 1 deletion src/js/engine/record-rtc.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import RecordRTC from 'recordrtc';
import Event from '../event';
import {RecordEngine} from './record-engine';
import {isChrome} from '../utils/detect-browser';
import {IMAGE_ONLY, AUDIO_ONLY, VIDEO_ONLY, AUDIO_VIDEO, ANIMATION, SCREEN_ONLY} from './record-mode';
import {IMAGE_ONLY, AUDIO_ONLY, VIDEO_ONLY, AUDIO_VIDEO, ANIMATION, SCREEN_ONLY, AUDIO_SCREEN} from './record-mode';

const Component = videojs.getComponent('Component');

Expand Down Expand Up @@ -168,6 +168,7 @@ class RecordRTCEngine extends RecordEngine {

case VIDEO_ONLY:
case AUDIO_VIDEO:
case AUDIO_SCREEN:
case SCREEN_ONLY:
// recordrtc returns a single blob that includes both audio
// and video data
Expand Down
37 changes: 34 additions & 3 deletions src/js/videojs.record.js
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import setSrcObject from './utils/browser-shim';
import {detectBrowser} from './utils/detect-browser';

import {getAudioEngine, isAudioPluginActive, getVideoEngine, getConvertEngine} from './engine/engine-loader';
import {IMAGE_ONLY, AUDIO_ONLY, VIDEO_ONLY, AUDIO_VIDEO, ANIMATION, SCREEN_ONLY, getRecorderMode} from './engine/record-mode';
import {IMAGE_ONLY, AUDIO_ONLY, VIDEO_ONLY, AUDIO_VIDEO, AUDIO_SCREEN, ANIMATION, SCREEN_ONLY, getRecorderMode} from './engine/record-mode';

const Plugin = videojs.getPlugin('plugin');
const Player = videojs.getComponent('Player');
Expand Down Expand Up @@ -68,6 +68,7 @@ class Record extends Plugin {

// add device button with icon based on type
let deviceIcon = 'av-perm';

switch (this.getRecordType()) {
case IMAGE_ONLY:
case VIDEO_ONLY:
Expand All @@ -80,6 +81,9 @@ class Record extends Plugin {
case SCREEN_ONLY:
deviceIcon = 'screen-perm';
break;
case AUDIO_SCREEN:
deviceIcon = 'sv-perm';
break;
}

// add custom interface elements
Expand Down Expand Up @@ -235,6 +239,7 @@ class Record extends Plugin {
case AUDIO_VIDEO:
case ANIMATION:
case SCREEN_ONLY:
case AUDIO_SCREEN:
// customize controls
this.player.bigPlayButton.hide();

Expand Down Expand Up @@ -362,7 +367,7 @@ class Record extends Plugin {
// check for support because some browsers still do not support
// getDisplayMedia or getUserMedia (like Chrome iOS, see:
// https://bugs.chromium.org/p/chromium/issues/detail?id=752458)
if (this.getRecordType() === SCREEN_ONLY) {
if (this.getRecordType() === SCREEN_ONLY || this.getRecordType() === AUDIO_SCREEN) {
if (navigator.mediaDevices === undefined ||
navigator.mediaDevices.getDisplayMedia === undefined) {
this.player.trigger(Event.ERROR,
Expand Down Expand Up @@ -450,6 +455,25 @@ class Record extends Plugin {
);
break;

case AUDIO_SCREEN:
// setup camera and microphone
this.mediaType = {
audio: (this.audioRecorderType === AUTO) ? true : this.audioRecorderType,
video: (this.videoRecorderType === AUTO) ? true : this.videoRecorderType
};
navigator.mediaDevices.getDisplayMedia({
video: true // This needs to be true for Firefox to work
}).then(screenStream => {
navigator.mediaDevices.getUserMedia({ audio:this.recordAudio }).then((mic) => {
// Join microphone track with screencast stream (order matters)
screenStream.addTrack(mic.getTracks()[0]);
this.onDeviceReady.bind(this)(screenStream);
});
}).catch(
this.onDeviceError.bind(this)
);
break;

case AUDIO_VIDEO:
// setup camera and microphone
this.mediaType = {
Expand Down Expand Up @@ -754,6 +778,7 @@ class Record extends Plugin {

case VIDEO_ONLY:
case AUDIO_VIDEO:
case AUDIO_SCREEN:
case SCREEN_ONLY:
// preview video stream in video element
this.startVideoPreview();
Expand Down Expand Up @@ -796,6 +821,7 @@ class Record extends Plugin {

case VIDEO_ONLY:
case AUDIO_VIDEO:
case AUDIO_SCREEN:
case ANIMATION:
case SCREEN_ONLY:
// wait for media stream on video element to actually load
Expand Down Expand Up @@ -985,6 +1011,7 @@ class Record extends Plugin {

case VIDEO_ONLY:
case AUDIO_VIDEO:
case AUDIO_SCREEN:
case SCREEN_ONLY:
// pausing the player so we can visualize the recorded data
// will trigger an async video.js 'pause' event that we
Expand All @@ -1006,7 +1033,7 @@ class Record extends Plugin {
this.playbackTimeUpdate);

// unmute local audio during playback
if (this.getRecordType() === AUDIO_VIDEO) {
if (this.getRecordType() === AUDIO_VIDEO || this.getRecordType() === AUDIO_SCREEN) {
this.mediaElement.muted = false;

// show the volume bar when it's unmuted
Expand Down Expand Up @@ -1117,6 +1144,7 @@ class Record extends Plugin {

case VIDEO_ONLY:
case AUDIO_VIDEO:
case AUDIO_SCREEN:
case ANIMATION:
case SCREEN_ONLY:
if (this.player.controlBar.currentTimeDisplay &&
Expand Down Expand Up @@ -1161,6 +1189,7 @@ class Record extends Plugin {

case VIDEO_ONLY:
case AUDIO_VIDEO:
case AUDIO_SCREEN:
case ANIMATION:
case SCREEN_ONLY:
// update duration display component
Expand Down Expand Up @@ -1190,6 +1219,7 @@ class Record extends Plugin {
case IMAGE_ONLY:
case VIDEO_ONLY:
case AUDIO_VIDEO:
case AUDIO_SCREEN:
case ANIMATION:
case SCREEN_ONLY:
if (url instanceof Blob || url instanceof File) {
Expand Down Expand Up @@ -1368,6 +1398,7 @@ class Record extends Plugin {
*/
muteTracks(mute) {
if ((this.getRecordType() === AUDIO_ONLY ||
this.getRecordType() === AUDIO_SCREEN ||
this.getRecordType() === AUDIO_VIDEO) &&
this.stream.getAudioTracks().length > 0) {
this.stream.getAudioTracks()[0].enabled = !mute;
Expand Down
7 changes: 5 additions & 2 deletions test/engine/record-mode.spec.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
* @since 2.2.0
*/

import {IMAGE_ONLY, AUDIO_ONLY, VIDEO_ONLY, AUDIO_VIDEO, ANIMATION, SCREEN_ONLY, getRecorderMode} from '../../src/js/engine/record-mode';
import {IMAGE_ONLY, AUDIO_ONLY, VIDEO_ONLY, AUDIO_VIDEO, ANIMATION, SCREEN_ONLY, AUDIO_SCREEN, getRecorderMode} from '../../src/js/engine/record-mode';

/** @test {record-mode} */
describe('engine.record-mode', () => {
Expand All @@ -14,6 +14,7 @@ describe('engine.record-mode', () => {
expect(AUDIO_VIDEO).toEqual('audio_video');
expect(ANIMATION).toEqual('animation');
expect(SCREEN_ONLY).toEqual('screen_only');
expect(AUDIO_SCREEN).toEqual('audio_screen');
});

it('returns the correct recorder mode', () => {
Expand All @@ -24,6 +25,7 @@ describe('engine.record-mode', () => {
expect(getRecorderMode(false, true, false, false, false)).toEqual(AUDIO_ONLY);
expect(getRecorderMode(true, false, false, false, false)).toEqual(IMAGE_ONLY);
expect(getRecorderMode(false, true, true, false, false)).toEqual(AUDIO_VIDEO);
expect(getRecorderMode(false, true, false, false, true)).toEqual(AUDIO_SCREEN);
expect(getRecorderMode(false, false, false, false, false)).toBeUndefined();

// and object
Expand All @@ -33,5 +35,6 @@ describe('engine.record-mode', () => {
expect(getRecorderMode(false, {}, false, false, false)).toEqual(AUDIO_ONLY);
expect(getRecorderMode({}, false, false, false, false)).toEqual(IMAGE_ONLY);
expect(getRecorderMode(false, {}, {}, false, false)).toEqual(AUDIO_VIDEO);
expect(getRecorderMode(false, {}, false, false, {})).toEqual(AUDIO_SCREEN);
});
});
});

0 comments on commit 0943657

Please sign in to comment.