-
Notifications
You must be signed in to change notification settings - Fork 9
/
tutorial-simple-manipulation.html
133 lines (117 loc) · 11.5 KB
/
tutorial-simple-manipulation.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="X-UA-Compatible" content="chrome=1">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="format-detection" content="telephone=no">
<meta name="viewport" content="user-scalable=no, initial-scale=1, maximum-scale=1, minimum-scale=1, width=device-width">
<title>rtc.io</title>
<link rel="stylesheet" type="text/css" href="css/reset.css">
<link rel="stylesheet" type="text/css" href="css/main.css">
<!-- responsive -->
<link rel="stylesheet" media="screen and (max-width: 960px)" href="css/tablet.css">
<link rel="stylesheet" media="screen and (max-width: 710px)" href="css/phone.css">
<link rel="stylesheet" type="text/css" href="fonts/source-sans/stylesheet.css">
<link rel="stylesheet" type="text/css" href="css/code.css">
</head>
<body>
<a class="scroll-point pt-top" name="top"></a>
<header>
<a href="https://github.com/rtc-io"><img class="fork" src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png" alt="Fork me on GitHub"></a>
<a class="scroll-point pt-top" name="top"></a>
<div class="site">
<div class="mascot">
<img src="images/artsio.png">
</div>
<div class="logo" data-subtext="OpenSource WebRTC">
<a href="index.html">rtc.io</a>
</div>
<nav>
<ul>
<li><a href="index.html">About</a></li>
<li><a href="tutorials.html">Tutorials</a></li>
<li><a href="demos.html">Demos</a></li>
<li><a href="modules.html">Modules</a></li>
</ul>
</nav>
</div>
<div class="shadow"></div>
</header>
<div class="main" role="content"><h1 id="processing-video-streams">Processing Video Streams</h1>
<p>One of the really "fun" things that you can do with WebRTC and media capture in general, is to do some post-processing on the captured video. This is a reasonably simple process whereby, you draw the video frames to a <code><canvas></code> element and then get the pixel data, manipulate the data and then push it back to the canvas.</p>
<p>As with other parts of the rtc.io suite, we've tried to make this as simple as possible, and much of the magic is achieved by using the <a href="module-rtc-canvas.html">rtc-canvas</a> helper module.</p>
<p>The <code>rtc-canvas</code> module allows you to render a captured <code>rtc-media</code> stream to a virtual <code><video></code> element which redraws it's content on a canvas. This virtual video element also has a processing pipeline that you can add manipulation methods to using the <code>.pipeline.add(handler)</code> method.</p>
<p>For example, here is what you will commonly include if you wish to do some video manipulation using the <code>rtc-canvas</code> module:</p>
<div class="highlight"><pre><span class="kd">var</span> <span class="nx">crel</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'crel'</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">media</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'rtc-media'</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">videoproc</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'rtc-videoproc'</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">video</span> <span class="o">=</span> <span class="nx">crel</span><span class="p">(</span><span class="s1">'video'</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">canvas</span> <span class="o">=</span> <span class="nx">crel</span><span class="p">(</span><span class="s1">'canvas'</span><span class="p">);</span>
<span class="c1">// set up the video processing</span>
<span class="nx">videoproc</span><span class="p">(</span><span class="nx">video</span><span class="p">,</span> <span class="nx">canvas</span><span class="p">,</span> <span class="p">{</span>
<span class="nx">filter</span><span class="o">:</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'rtc-filter-grayscale'</span><span class="p">)</span>
<span class="p">});</span>
<span class="c1">// capture and render the video</span>
<span class="nx">media</span><span class="p">().</span><span class="nx">render</span><span class="p">(</span><span class="nx">video</span><span class="p">);</span>
<span class="c1">// add the canvas to the document body</span>
<span class="nb">document</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nx">canvas</span><span class="p">);</span>
</pre></div>
<p>In the code above, you can see that we are using a node style <code>require</code> statement to include a grayscale filter into our processing code. Let's have a look at the implementation of that small module now:</p>
<div class="highlight"><pre><span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">imageData</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">channels</span> <span class="o">=</span> <span class="nx">imageData</span><span class="p">.</span><span class="nx">data</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">channelCount</span> <span class="o">=</span> <span class="nx">channels</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span>
<span class="c1">// iterate through the data</span>
<span class="k">for</span> <span class="p">(</span><span class="kd">var</span> <span class="nx">ii</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">ii</span> <span class="o"><</span> <span class="nx">channelCount</span><span class="p">;</span> <span class="nx">ii</span> <span class="o">+=</span> <span class="mi">4</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// update the values to the rgb average</span>
<span class="nx">channels</span><span class="p">[</span><span class="nx">ii</span><span class="p">]</span> <span class="o">=</span> <span class="c1">// update R</span>
<span class="nx">channels</span><span class="p">[</span><span class="nx">ii</span> <span class="o">+</span> <span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="c1">// update G</span>
<span class="nx">channels</span><span class="p">[</span><span class="nx">ii</span> <span class="o">+</span> <span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="c1">// update B</span>
<span class="p">(</span><span class="nx">channels</span><span class="p">[</span><span class="nx">ii</span><span class="p">]</span> <span class="o">+</span> <span class="nx">channels</span><span class="p">[</span><span class="nx">ii</span> <span class="o">+</span> <span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="nx">channels</span><span class="p">[</span><span class="nx">ii</span> <span class="o">+</span> <span class="mi">2</span><span class="p">]</span> <span class="p">)</span> <span class="o">/</span> <span class="mi">3</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// return true to flag that we want to write our pixel data</span>
<span class="c1">// back to the canvas</span>
<span class="k">return</span> <span class="kc">true</span><span class="p">;</span>
<span class="p">};</span>
</pre></div>
<p>This code takes the image data that has already been extracted via the canvas 2d context <a href="http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#dom-context-2d-getimagedata">getImageData</a> method and applies some processing to it (in this case a simple grayscale filter).</p>
<p>By returning true at the end of the function, we are flagging to the <code>rtc-canvas</code> module that modifications have been made to the reference pixel data and that a <a href="http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#dom-context-2d-putimagedata">putImageData</a> call should be made to replace the contents of the canvas.</p>
<p><a class="sample" data-sample="simple-manipulation" href="#">Run Sample</a></p>
<h2 id="adjusting-the-capture-rate">Adjusting the Capture Rate</h2>
<p>By default, the canvas is set to capture at 25 fps, but this can be adjusted if you feel like giving your CPU a bit of a rest:</p>
<div class="highlight"><pre><span class="kd">var</span> <span class="nx">crel</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'crel'</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">media</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'rtc-media'</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">videoproc</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'rtc-videoproc'</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">video</span> <span class="o">=</span> <span class="nx">crel</span><span class="p">(</span><span class="s1">'video'</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">canvas</span> <span class="o">=</span> <span class="nx">crel</span><span class="p">(</span><span class="s1">'canvas'</span><span class="p">);</span>
<span class="c1">// set up the video processing</span>
<span class="nx">videoproc</span><span class="p">(</span><span class="nx">video</span><span class="p">,</span> <span class="nx">canvas</span><span class="p">,</span> <span class="p">{</span>
<span class="nx">fps</span><span class="o">:</span> <span class="mi">5</span><span class="p">,</span>
<span class="nx">filter</span><span class="o">:</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'rtc-filter-grayscale'</span><span class="p">)</span>
<span class="p">});</span>
<span class="c1">// capture and render the video</span>
<span class="nx">media</span><span class="p">().</span><span class="nx">render</span><span class="p">(</span><span class="nx">video</span><span class="p">);</span>
<span class="c1">// add the canvas to the document body</span>
<span class="nb">document</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nx">canvas</span><span class="p">);</span>
</pre></div>
<p><a class="sample" data-sample="simple-manipulation-5fps" href="#">Run Sample</a></p>
</div>
<footer>
<p>
<a href="http://nicta.com.au">
<img src="images/nicta-logo.gif" alt="NICTA logo">
</a>© NICTA 2013 - 2014
</p>
<p class="license">Project source code is licensed under the <a href="https://github.com/rtc-io/rtc/blob/master/LICENSE">Apache 2.0</a>.</p>
<a class="closing" href="#top"></a>
</footer>
</body>
<script src="js/app.js"></script>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-26567546-2', 'rtc.io');
ga('send', 'pageview');
</script>
</html>