Zoom optimisation for Waves and matching implementation for Spectrogr…

…ams (#2646)

* Used setTimeout() method to require a minimum time period passes before
redrawing/recalculating to reduce CPU requirements while zooming in on
waveforms.
Currently displaying a waveform image in example/zoom which will be used
to hide the lack of wave drawing while zooming.

* Implemented a backimage mimicking the wave to display zoom level faster
TO DO:
Hide the real wave while zooming
Hide the backimage after zooming
Zoom in on current timestamp while zooming on backimage
FUTURE:
Split or crop backimage for longer waves
Re-render backimage to match the colour style of the wave with selected
timestamp

* Added hiding/showing of elements while zooming
Since the coloured section of the wave is also handled by canvas objects
within a wave I am going to move the backimage functionality into the
canvasentry.js file so that each canvas handles it's own image, which
should resolve the issue as well as segment the image into sections.

* backimages are stored and drawn for each canvas entry. Current issue is
the left offsets of each backimage are not being calculated correctly
and images are overlapping.

* Zooming now correctly handles progress positions.
Creates backimages for progress wave
Left offset of canvas elements are handled correctly.

* Zooming out now works without any issues
Zooming in needs a viewable region boundary implemented so that canvases
don't become too large.
Progress tracking and regions plugin work without any issues now.

* Zoom optimisations have been made
Canvases not in frame are cleared and not rendered
Canvases that become too large are clipped
Progress location is maintained

* Added priority drawing for canvases in view
Currently, changing zoom level before all canvases have been loaded
causes issues, my next goal is to fix this by either improving the
handling of missing canvases or preventing zoom change until all
canvases have loaded.

* Zoom slider locks up if rendering isn't finished
I'll use and hopefully instead of locking up I can return to a backup
render of the wave.

* Allows zooming immediately after release
This is done by cancelling loading and showing only loaded sections

* Improved backup image that can handle large waves
Zooming out on partially loaded waves loads a backup image, this
features works regardless of wave size.

* Optimisation checks if there is a wrapper
With this change the branch now passes the included tests

* Fixed issues with back image generation
Low zoom level previously caused issue with back image generation, the
issue has been resolved.

* Added zooming to spectrogram example
This will allow testing of the intended multicanvas implementation

* Spectrogram successfully split into two canvases
Canvases are equally sized
Calculation is on the whole spectrogram and then converted to seperate
canvases
This is a starting point for scaling up to an adaptive number of
canvases, similar to the multicanvas drawer used for the main wave.
Ideally calculation can be moved such that it is done for each canvas,
this would mean speed could be improved by calcuating visible
spectrogram canvases.
Right now this offers no noticeable speed improvement, but could improve
the ability to display zoomable spectrograms without creating images
too large for browsers.

* Working for arbitrary number of canvases
Image still generated first then split, limiting max size
Committing a working version before attempting fix

* spectrogram sections are generated per-canvas
Currently trying to draw all canvases at once causes failure
Next I'm going to implement staggering of the canvases to hopefully
allowing further zooming even without performance improvements

* Spectrogram can now display at unlimited width
Fixed an issue where resample() was creating full-size arrays instead of
limiting to the canvas size
Fixed an issue where render() was being called twice as a result of an
incorrect event fire in wavesurfer.js
Removed placeholder text in zoom example

* Spectrogram only draws viewable canvases
After this I will implement delays rendering of low-priority canvases

* Rearranged variables to allow for setting delays

* Canvases stretch to zoom

* Zoom now scales correctly against waveform
Fixed issue where progress wasn't accurate during zoom

* Improved zooming functionality for wavesurfer
Removed functions that were no longer necessary
backimages are no longer required, existing canvases are stretched
during zoom.
backup image is no longer required, canvases can continue rendering
during the zooming process.

* Implemented priority rendering for spectrogram
Removed/changed comments referring to the removed backimage/backupimage

* Added timeout clearing to prevent drawing old canvases after zoom level
changes

* Removed unecessary class variables

* Patched in a fix so that original zooming behaviour is preserved
Upon first call of the stretchCanvases() function the drawer will switch
over to the optimised zoom functionality

* Removed unecessary changes to formatting

* Removed more unecessary changes

---------

Co-authored-by: Luke Harrison <s4587520@student.uq.edu.au>

example/spectrogram/app.js

@@ -22,15 +22,6 @@ function initAndLoadSpectrogram(colorMap) {
         ]
     };
 
-    if (location.search.match('scroll')) {
-        options.minPxPerSec = 100;
-        options.scrollParent = true;
-    }
-
-    if (location.search.match('normalize')) {
-        options.normalize = true;
-    }
-
     wavesurfer = WaveSurfer.create(options);
 
     /* Progress bar */
@@ -54,6 +45,29 @@ function initAndLoadSpectrogram(colorMap) {
     })();
 
     wavesurfer.load('../media/demo.wav');
+
+    // Zoom slider
+    let slider = document.querySelector('[data-action="zoom"]');
+
+    slider.value = wavesurfer.params.minPxPerSec;
+    slider.min = wavesurfer.params.minPxPerSec;
+    slider.max = 250;
+
+
+    slider.addEventListener('input', function() {
+        wavesurfer.zooming(slider.value);
+    });
+    slider.addEventListener('mouseup', function() {
+        wavesurfer.zoom(slider.value);
+
+        let desiredWidth = Math.max(parseInt(wavesurfer.container.offsetWidth * window.devicePixelRatio),
+            parseInt(wavesurfer.getDuration() * slider.value * window.devicePixelRatio));
+        wavesurfer.spectrogram.width = desiredWidth;
+        wavesurfer.spectrogram.render();
+    });
+
+    // set initial zoom to match slider value
+    wavesurfer.zoom(slider.value);
 }
 
 document.addEventListener('DOMContentLoaded', function() {

example/spectrogram/index.html

@@ -31,13 +31,6 @@
     <body itemscope itemtype="http://schema.org/WebApplication">
         <div class="container">
             <div class="header">
-                <noindex>
-                <ul class="nav nav-pills pull-right">
-                    <li><a href="?fill">Fill</a></li>
-                    <li><a href="?scroll">Scroll</a></li>
-                </ul>
-                </noindex>
-
                 <h1 itemprop="name"><a href="http://wavesurfer-js.org">wavesurfer.js</a><noindex> + Spectrogram</noindex></h1>
             </div>
 
@@ -59,6 +52,17 @@ <h1 itemprop="name"><a href="http://wavesurfer-js.org">wavesurfer.js</a><noindex
                         <i class="glyphicon glyphicon-pause"></i>
                         Pause
                     </button>
+                    <div class="col-sm-1">
+                        <i class="glyphicon glyphicon-zoom-in"></i>
+                    </div>
+
+                    <div class="col-sm-3">
+                      <input data-action="zoom" type="range" min="1" max="200" value="0" style="width: 100%" />
+                    </div>
+
+                    <div class="col-sm-1">
+                        <i class="glyphicon glyphicon-zoom-out"></i>
+                    </div>
                 </div>
             </div>
 

example/zoom/index.html

@@ -82,9 +82,13 @@ <h3>How to Zoom</h3>
     backend: 'MediaElement'
 });
 
-document.querySelector('#slider').oninput = function () {
-    wavesurfer.zoom(Number(this.value));
-};
+let slider = document.querySelector('[data-action="zoom"]');
+slider.addEventListener('input', function() {
+    wavesurfer.zooming(slider.value);
+});
+slider.addEventListener('mouseup', function() {
+    wavesurfer.zoom(slider.value);
+});
 </code></pre>
                 </p>
             </div>

example/zoom/main.js

@@ -49,6 +49,9 @@ document.addEventListener('DOMContentLoaded', function() {
     slider.max = 1000;
 
     slider.addEventListener('input', function() {
+        wavesurfer.zooming(Number(this.value));
+    });
+    slider.addEventListener('mouseup', function() {
         wavesurfer.zoom(Number(this.value));
     });
 

src/drawer.canvasentry.js

@@ -66,6 +66,11 @@ export default class CanvasEntry {
          * @type {object}
          */
         this.canvasContextAttributes = {};
+        /**
+         * The Timeout id used to track this canvas entry.
+         */
+        this.drawTimeout = null;
+
     }
 
     /**
@@ -125,21 +130,27 @@ export default class CanvasEntry {
      */
     clearWave() {
         // wave
+        this.waveCtx.save();
+        this.waveCtx.setTransform(1, 0, 0, 1, 0, 0);
         this.waveCtx.clearRect(
             0,
             0,
             this.waveCtx.canvas.width,
             this.waveCtx.canvas.height
         );
+        this.waveCtx.restore();
 
         // progress
         if (this.hasProgressCanvas) {
+            this.progressCtx.save();
+            this.progressCtx.setTransform(1, 0, 0, 1, 0, 0);
             this.progressCtx.clearRect(
                 0,
                 0,
                 this.progressCtx.canvas.width,
                 this.progressCtx.canvas.height
             );
+            this.progressCtx.restore();
         }
     }
 
@@ -424,4 +435,36 @@ export default class CanvasEntry {
             return this.wave.toDataURL(format, quality);
         }
     }
+
+    /**
+     * Stretches existing canvas
+     * @param {Number} newTotalWidth total width of wave in pixels
+     */
+    stretchCanvas(newTotalWidth) {
+        //Calculate the start and width of this canvas
+        let start = Math.round(this.start * newTotalWidth);
+        let width = Math.round(this.end * newTotalWidth - start);
+
+        //Stretch canvas
+        let elementSize = { width: width + 'px' };
+        let elementStart = {left: start + 'px'};
+        style(this.wave, elementSize);
+        style(this.wave, elementStart);
+        if (this.hasProgressCanvas) {
+            style(this.progress, elementSize);
+            style(this.progress, elementStart);
+        }
+    }
+
+    /**
+     * Set the left offset of the canvas
+     * @param {Number} position in px for the canvas to start
+     */
+    setLeft(position) {
+        let elementStart = {left: position + 'px'};
+        style(this.wave, elementStart);
+        if (this.hasProgressCanvas) {
+            style(this.progress, elementStart);
+        }
+    }
 }

src/drawer.js

@@ -272,10 +272,6 @@ export default class Drawer extends util.Observer {
      * @return {boolean} Whether the width of the container was updated or not
      */
     setWidth(width) {
-        if (this.width == width) {
-            return false;
-        }
-
         this.width = width;
 
         if (this.params.fillParent || this.params.scrollParent) {

src/drawer.multicanvas.js

@@ -89,6 +89,14 @@ export default class MultiCanvas extends Drawer {
          * @type {boolean}
          */
         this.vertical = params.vertical;
+
+        /**
+         * Whether to use the optimsized zoom rendering
+         * Automatically toggles to true if stretchCanvases() function is called
+         *
+         * @type {boolean}
+         */
+        this.optimiseZoom = false;
     }
 
     /**
@@ -157,10 +165,16 @@ export default class MultiCanvas extends Drawer {
 
         let canvasWidth = this.maxCanvasWidth + this.overlap;
         const lastCanvas = this.canvases.length - 1;
+        let leftOffset = 0;
         this.canvases.forEach((entry, i) => {
             if (i == lastCanvas) {
                 canvasWidth = this.width - this.maxCanvasWidth * lastCanvas;
             }
+
+            //Set left offset and add to next entry
+            entry.setLeft(leftOffset);
+            leftOffset += canvasWidth / this.params.pixelRatio;
+
             this.updateDimensions(entry, canvasWidth, this.height);
 
             entry.clearWave();
@@ -410,7 +424,39 @@ export default class MultiCanvas extends Drawer {
         this.canvases.forEach((entry, i) => {
             this.setFillStyles(entry, waveColor, progressColor);
             this.applyCanvasTransforms(entry, this.params.vertical);
-            entry.drawLines(peaks, absmax, halfH, offsetY, start, end);
+
+            if (this.optimiseZoom) {
+                //Optimising zoom functionality
+                //If there's a wrapper, optimise for the view
+                let priority = 0;
+                if (this.wrapper) {
+                    let canvasRect = entry.wave.getBoundingClientRect();
+                    let wrapperRect = this.wrapper.getBoundingClientRect();
+
+                    //Determine whether canvas is in viewframe or not and assign priority
+                    if (Math.floor(canvasRect['left']) > Math.ceil(wrapperRect['right'])) {
+                        //Canvas is to the right of view window
+                        let distance = canvasRect['left'] - wrapperRect['right'];
+                        priority = Math.ceil(distance / wrapperRect['width']);
+                    } else if (Math.ceil(canvasRect['right']) < Math.floor(wrapperRect['left'])) {
+                        //Canvas is to the left of the view window
+                        let distance = wrapperRect['left'] - canvasRect['right'];
+                        priority = Math.ceil(distance / wrapperRect['width']);
+                    }
+                } else {
+                    //Everything is equal priority
+                }
+
+                //This staggers the drawing of canvases so they don't all draw at once
+                entry.clearWave();
+                clearTimeout(entry.drawTimeout);
+                entry.drawTimeout = setTimeout(function(){
+                    entry.drawLines(peaks, absmax, halfH, offsetY, start, end);
+                    entry.drawTimeout = null;
+                }, 25 * priority);
+            } else {
+                entry.drawLines(peaks, absmax, halfH, offsetY, start, end);
+            }
         });
     }
 
@@ -606,6 +652,30 @@ export default class MultiCanvas extends Drawer {
         }
     }
 
+    /**
+     * Stretches the canvases to mimic zoom without recalculation
+     *
+     * @param {Number} desiredWidth new width of the wave display
+     * @param {Number} progress Value between 0 and 1 for wave progress
+     */
+    stretchCanvases(desiredWidth, progress) {
+        if (!this.optimiseZoom) {
+            //Enable optimsed zooming
+            this.optimiseZoom = true;
+        }
+        let totalCanvasWidth = Math.round(desiredWidth / this.params.pixelRatio);
+        this.width = desiredWidth;
+
+        for (let i = 0; i < this.canvases.length; i++) {
+            this.canvases[i].stretchCanvas(totalCanvasWidth);
+        }
+
+        //Update progress
+        let progressPos = progress * totalCanvasWidth;
+        this.updateProgress(progressPos);
+        this.recenterOnPosition(progressPos, true);
+    }
+
     /**
      * Render the new progress
      *