Skip to content

Commit

Permalink
Created the AnkiMediaQueue for JavaScript media support
Browse files Browse the repository at this point in the history
This adds the global `ankimedia` javascript object to every card. By default, it does not do anything unless the user explicitly calls for `ankimedia.setup()` inside a script tag. For example:

![image](https://user-images.githubusercontent.com/5332158/80847514-5bf8ea80-8be6-11ea-93f0-7fccfcd4bc72.png)

**front**
```html
What is the <b>past participle</b> of the verb to <b>{{Verb}}</b>?</span><br>
<span style="display: flex; align-items: center;">
    <input type="button" value="x1.2" onclick="setAnkiMedia( media => { media.playbackRate = 1.2; } )" />
    <input type="button" value="x1.0" onclick="setAnkiMedia( media => { media.playbackRate = 1.0; } )" />
    <input type="button" value="x0.6" onclick="setAnkiMedia( media => { media.playbackRate = 0.6; } )" />
    <audio data-file="{{VerbAudio}}" controlsList="nodownload" controls></audio>
</span>
{{type:PastParticiple}}

<script type="text/javascript">
ankimedia.setup();
ankimedia.add("front", "{{VerbAudio}}", 1.2);
☺</script>
```

**style**
```css
audio {
    width: 100%;
}
```

**style**
```html
{{FrontSide}}
<hr id=answer>

<span style="display: flex; align-items: center;">
    <input type="button" value="x1.2" onclick="setAnkiMedia( media => { media.playbackRate = 1.2; } )" />
    <input type="button" value="x1.0" onclick="setAnkiMedia( media => { media.playbackRate = 1.0; } )" />
    <input type="button" value="x0.6" onclick="setAnkiMedia( media => { media.playbackRate = 0.6; } )" />
    <audio data-file="{{PastParticipleAudio}}" controlsList="nodownload" controls></audio>
</span>
{{PastParticiplePhonetic}}

<script type="text/javascript">
ankimedia.add( "back", "{{PastParticipleAudio}}", 1.2);
</script>
```

This will require the usage of the argument `fileonly` to the sound tag (#540 - Added arguments as [sound:argument1 argument2]).

**fields**
```java
Verb: bumb
VerbPhonetic: /bumb/
VerbAudio: [sound:bumb.mp3|fileonly]
PastParticiple: bamb
PastParticiplePhonetic: /bamb/
PastParticipleAudio: [sound:bamb.mp3|fileonly]
```

Instead of calling `ankimedia.add("front", "{{VerbAudio}}", 1.2)` / `ankimedia.add( "back", "{{PastParticipleAudio}}", 1.2)` you can just call `ankimedia.autoadd( "front" )` / `ankimedia.autoadd( "back" )` which automatically detect all HTML5 media elements and plays them sequentially.

These is the documentation for all public methods of the `ankimedia` global object:
```sql
/**
 * Find all audio and video tags and run them through the callback parameter.
 * @param {Function} callback - to be called on each media.
 * @param {Array}    initial  - an additional list of items to be iterated over.
 */
function setAnkiMedia(callback, initial = undefined)

/**
 * Automatically add all media elements found and start playing them sequentially.
 * @param {string} where - pass "front" if this is being called on the card-front,
 *        otherwise, pass "back" if it is being called on the card-back.
 * @param {number} speed - the speed to play the audio, where 1.0 is the default speed.
 *        Each media element can also have an attribute as `data-speed="1.0"` indicating
 *        the speed it should play. The `data-speed` value has precedence over this parameter.
 */
ankimedia.addall(where, speed = 1.0)

/**
 * Add an audio file to the playing queue and immediately starts playing, if not playing already.
 * @param {string} filename - an audio filename for playing
 * @param {string} where    - pass "front" if this is being called on the card-front,
 *        otherwise, pass "back" as it is being called on the card-back.
 * @param {number} speed    - the speed to play the audio, where 1.0 is the default speed.
 *        Each media element can also have an attribute as `data-speed="1.0"` indicating
 *        the speed it should play. The `data-speed` value has precedence over this parameter.
 */
ankimedia.add(where, filename, speed = 1.0)

/**
 * Call this on your front-card before adding new medias to the playing queue.
 * You can call this function as `setup({delay: 5, wait: false})`.
 *
 * @param {number} delay   - how many seconds to time to wait before playing the next audio.
 * @param {boolean} wait   - if true (default), wait the question audio to play
 *        when the answer was showed before it had finished playing.
 * @param {function} extra - a function(media) to be run on each media of the page.
 * @param {array} medias   - an array of initial values to be passed to setAnkiMedia() calls.
 */
ankimedia.setup(parameters: any = {})
```

You can see the audio tests running if you go do the directory `anki/qt`, run `export PUPPETEER_HEADLESS=false` and run `make check`:

![keepinput](https://user-images.githubusercontent.com/5332158/80896475-faf41400-8cc4-11ea-9dcc-553569eb567b.gif)

```java
(pyenv) F:\anki\qt\ts>set PUPPETEER_HEADLESS=false

(pyenv) F:\anki\qt\ts>cmd /c "cd .. && make .build/testjs"
(cd ts && npm run build)

> anki-dtop-js@1.0.0 build F:\anki\qt\ts
> tsc --build

(cd ts && npm run gulp)

> anki-dtop-js@1.0.0 gulp F:\anki\qt\ts
> gulp reviewer

[19:13:06] Using gulpfile F:\anki\qt\ts\gulpfile.js
[19:13:06] Starting 'reviewer'...
[19:13:17] Finished 'reviewer' after 11 s
(cd ts && npm run test)

> anki-dtop-js@1.0.0 test F:\anki\qt\ts
> jest --verbose --runInBand

Determining test suites to run... Running static file server on 'http://127.0.0.1:58254'...
 Environment 'slowMo=undefined' 'headless=true' 'args=--window-position=960,10'

DevTools listening on ws://127.0.0.1:58255/devtools/browser/2b4a8e8c-7d76-4ce8-a01c-61754b7231e0
 PASS  src/reviewer.test.ts (15.415s)
  Test question and answer audios
    √ Showing a question should play its audio file automatically:
front silence1.mp3 'ankimedia.setup(); ankimedia.addall( "front" );'... (870ms)
    √ Showing a question should play its audio file automatically:
front silence1.mp3 'ankimedia.setup({delay: 0, wait: false, medias: []}); ankimedia.addall( "front" );'... (590ms)
    √ Showing a question should play its audio file automatically:
front silence2.mp3 'ankimedia.setup(); ankimedia.add( "front", "silence2.mp3" );'... (574ms)
    √ Showing a question should play its audio file automatically:
front silence2.mp3 'ankimedia.setup({delay: 0, wait: false, medias: []}); ankimedia.add( "front", "silence2.mp3" );'... (560ms)
    √ Showing a question should play its audio file automatically:
front silence2.mp3 'ankimedia.setup({delay: 0, wait: false, medias: []}); ankimedia.add( "front", "silence2.mp3" );'... (320ms)
    √ Showing a new question should play its audio automatically:
front silence1.mp3 'ankimedia.setup(); ankimedia.addall( "front" );',
refront silence2.mp3 'ankimedia.setup(); ankimedia.addall( "front" );'... (1083ms)
    √ Showing a new question should play its audio automatically:
front silence1.mp3 'ankimedia.setup(); ankimedia.add( "front", "silence1.mp3" );',
refront silence2.mp3 'ankimedia.setup(); ankimedia.add( "front", "silence2.mp3" );'... (1097ms)
    √ Showing an answer with the same id as the question should only play the answer audio:
front silence1.mp3 'ankimedia.setup(); ankimedia.addall( "front" );',
back silence1.mp3 'ankimedia.setup(); ankimedia.addall( "back" );'... (1263ms)
    √ Showing an answer with the same id as the question should only play the answer audio:
front silence1.mp3 'ankimedia.setup(); ankimedia.addall( "front" );',
back silence2.mp3 'ankimedia.setup(); ankimedia.addall( "back" );'... (1268ms)
    √ Showing an answer with the same id as the question should only play the answer audio:
front silence1.mp3 'ankimedia.setup(); ankimedia.add( "front", "silence1.mp3" );',
back silence1.mp3 'ankimedia.setup(); ankimedia.add( "back", "silence1.mp3" );'... (1294ms)
    √ Showing an answer with the same id as the question should only play the answer audio:
front silence1.mp3 'ankimedia.setup(); ankimedia.add( "front", "silence1.mp3" );',
back silence2.mp3 'ankimedia.setup(); ankimedia.add( "back", "silence2.mp3" );'... (1303ms)

 PASS  src/reviewer-exceptions.test.ts
  Test question and answer exception handling
    √ ankimedia.setup() with invalid parameters (24ms)
    √ ankimedia.setup() with bad parameters (20ms)
    √ do not call setup() before other methods (3ms)
    √ do not pass the correct value of where (4ms)
    √ do not add media files with the correct speed or file name (10ms)
    √ Calling functions with invalid arguments count (6ms)
    √ setAnkiMedia() with invalid callback parameters (4ms)

Test Suites: 2 passed, 2 total
Tests:       18 passed, 18 total
Snapshots:   0 total
Time:        17.026s, estimated 18s
Ran all test suites.

(pyenv) F:\anki\qt\ts>
```
  • Loading branch information
evandrocoan committed May 3, 2020
1 parent 6046bbc commit a778a19
Show file tree
Hide file tree
Showing 20 changed files with 9,704 additions and 119 deletions.
4 changes: 4 additions & 0 deletions .github/workflows/checks.yml
Expand Up @@ -293,11 +293,15 @@ jobs:
set -x
sudo apt update
sudo apt install portaudio19-dev gettext
# https://github.com/BurntSushi/ripgrep/issues/1232
# sudo apt-get install ripgrep
curl -LO https://github.com/BurntSushi/ripgrep/releases/download/11.0.2/ripgrep_11.0.2_amd64.deb
sudo dpkg -i ripgrep_11.0.2_amd64.deb
# https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md#chrome-headless-doesnt-launch-on-unix
sudo apt-get install ca-certificates fonts-liberation gconf-service libappindicator1 libasound2 libatk-bridge2.0-0 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgbm1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libnss3 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 lsb-release wget xdg-utils
- name: Set up brew ripgrep, pyaudio, gettext
if: matrix.os == 'macos-latest'
run: |
Expand Down
1 change: 1 addition & 0 deletions qt/.gitignore
Expand Up @@ -13,6 +13,7 @@ aqt/forms
tools/runanki.system
ts/node_modules
aqt_data/locale
aqt_data/web/ankimedia.js
aqt_data/web/deckbrowser.js
aqt_data/web/editor.js
aqt_data/web/overview.js
Expand Down
37 changes: 27 additions & 10 deletions qt/Makefile
Expand Up @@ -52,10 +52,15 @@ all: check
./scripts/copy-qt-files)
@touch $@

TSDEPS := $(wildcard ts/src/*.ts) $(wildcard ts/scss/*.scss)
TSDEPS := $(shell "${FIND}" ts/src ! -name "*.test.ts" -name '*.ts') ts/tsconfig.json ts/package.json

.build/js: $(TSDEPS)
(cd ts && npm i && npm run build)
.build/js: .build/scss $(TSDEPS)
(cd ts && npm run build)
(cd ts && npm run gulp)
@touch $@

.build/scss: .build/prodDependencies $(wildcard ts/scss/*.scss)
(cd ts && npm run sass)
python ./tools/extract_scss_colors.py
@touch $@

Expand All @@ -73,22 +78,20 @@ BUILD_STEPS := .build/vernum .build/run-deps .build/dev-deps .build/js .build/ui
check: $(BUILD_STEPS) .build/mypy .build/test .build/fmt .build/imports .build/lint .build/ts-fmt

.PHONY: fix
fix: $(BUILD_STEPS)
fix: .build/devDependencies $(BUILD_STEPS)
isort $(ISORTARGS)
python -m black $(BLACKARGS)
(cd ts && npm run pretty)

.PHONY: clean
clean:
rm -rf .build aqt.egg-info build dist
rm -rf .build ts/node_modules aqt.egg-info build dist

# Checking Typescript
######################

JSDEPS := $(patsubst ts/src/%.ts, web/%.js, $(TSDEPS))

.build/ts-fmt: $(TSDEPS)
(cd ts && npm i && npm run check-pretty)
.build/ts-fmt: .build/devDependencies $(TSDEPS)
(cd ts && npm run check-pretty)
@touch $@

# Checking python
Expand All @@ -102,10 +105,24 @@ CHECKDEPS := $(shell "${FIND}" aqt tests -name '*.py' | grep -v buildinfo.py)
python -m mypy ${MYPY_ARGS} aqt
@touch $@

.build/test: $(CHECKDEPS)
.build/test: $(CHECKDEPS) .build/testjs
python -m pytest -s
@touch $@

.build/testjs: .build/js .build/devDependencies $(wildcard ts/src/*.?s) $(wildcard ts/*.js)
(cd ts && npm run test)
@touch $@

.build/devDependencies: ts/package.json
# https://github.com/fsevents/fsevents/issues/321
(cd ts && npm install --no-optional --only=dev)
@touch $@

.build/prodDependencies: ts/package.json
# https://github.com/fsevents/fsevents/issues/321
(cd ts && npm install --no-optional --only=prod)
@touch $@

.build/lint: $(CHECKDEPS)
python -m pylint -j 0 --rcfile=.pylintrc -f colorized ${PYLINT_ARGS} \
--extension-pkg-whitelist=PyQt5,ankirspy aqt tests setup.py
Expand Down
6 changes: 6 additions & 0 deletions qt/aqt/clayout.py
Expand Up @@ -224,6 +224,12 @@ def setupWebviews(self):
pform.frontWeb.set_bridge_command(self._on_bridge_cmd, self)
pform.backWeb.set_bridge_command(self._on_bridge_cmd, self)

# Enable the Javascript audio play only on the back-card to avoid double play
# https://anki.tenderapp.com/discussions/beta-testing/1858-can-you-pass-autoplay-policyno-user-gesture-required-to-chrome-engine
pform.backWeb._page.settings().setAttribute(
QWebEngineSettings.PlaybackRequiresUserGesture, False
)

def _on_bridge_cmd(self, cmd: str) -> Any:
if cmd.startswith("play:"):
play_clicked_audio(cmd, self.card)
Expand Down
4 changes: 4 additions & 0 deletions qt/aqt/main.py
Expand Up @@ -779,6 +779,10 @@ def setupMainWindow(self) -> None:
o._domReady = False
o._page.setContent(bytes("", "ascii"))

# Enable the Javascript audio auto play on the main web view
# https://anki.tenderapp.com/discussions/beta-testing/1858-can-you-pass-autoplay-policyno-user-gesture-required-to-chrome-engine
self.web._page.settings().setAttribute(QWebEngineSettings.PlaybackRequiresUserGesture, False) # type: ignore

def closeAllWindows(self, onsuccess: Callable) -> None:
aqt.dialogs.closeAll(onsuccess)

Expand Down
22 changes: 22 additions & 0 deletions qt/ts/.vscode/launch.json
@@ -0,0 +1,22 @@
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
// https://jestjs.io/docs/en/troubleshooting.html
"version": "0.2.0",
"configurations": [
{
"name": "Debug Jest Tests",
"type": "node",
"request": "launch",
"runtimeArgs": [
"--inspect-brk",
"${workspaceRoot}/node_modules/jest/bin/jest.js",
"--runInBand"
],
"console": "internalConsole",
"internalConsoleOptions": "neverOpen",
"port": 9229
}
]
}
86 changes: 86 additions & 0 deletions qt/ts/globalSetup.js
@@ -0,0 +1,86 @@
/* Copyright: Ankitects Pty Ltd and contributors
* License: GNU AGPL, version 3 or later; http://www.gnu.org/licenses/agpl.html */
const express = require('express');
const puppeteer = require('puppeteer');

module.exports = async () => {
await setupExpress();
await setupPuppeteer();
};

/**
* Convert a any object to true/false to be used on puppeteer headless option.
* @return {boolean} true if value is undefined, 'true', 'yes', has length 0, or is a nonzero.
* false if value is 'false', 'no', the zero number or any other string.
*/
function stringToBoolean(value) {
value = String(value);
switch(value.toLowerCase().trim()){
case "true": case "yes": return true;
case "false": case "no": return false;
default: {
if(value.length === 0 || value == "undefined") {
return true;
}
return !!parseInt(value);
}
}
}

async function setupExpress() {
let server;
const app = express();

await new Promise(function(resolve) {
server = app.listen(0, "127.0.0.1", function() {
let address = server.address();
process.env.SERVER_ADDRESS = `http://${address.address}:${address.port}`;
console.log(` Running static file server on '${process.env.SERVER_ADDRESS}'...`);
resolve();
});
});

global.server = server;
app.get('/favicon.ico', (req, res) => res.sendStatus(200));
app.use(express.static('./testfiles'));
app.use(express.static('../aqt_data/web'));
}

async function setupPuppeteer() {
let PUPPETEER_SLOWMO = parseInt(process.env.PUPPETEER_SLOWMO) || undefined;
let PUPPETEER_HEADLESS = stringToBoolean(process.env.PUPPETEER_HEADLESS);
let PUPPETEER_CHROME_ARGS = process.env.PUPPETEER_CHROME_ARGS;
let chrome_args = [
// "--start-maximized",
// "--window-position=960,10",
"--autoplay-policy=no-user-gesture-required",
// Puppeteer with headless:true is extremely slow
// https://github.com/puppeteer/puppeteer/issues/1718
"--proxy-server='direct://'",
'--proxy-bypass-list=*',
];

if(PUPPETEER_CHROME_ARGS) {
PUPPETEER_CHROME_ARGS = PUPPETEER_CHROME_ARGS.trim();
if(PUPPETEER_CHROME_ARGS.length > 0) {
chrome_args.push(...PUPPETEER_CHROME_ARGS.split(' '));
}
}

console.log(` Environment 'slowMo=${PUPPETEER_SLOWMO}' 'headless=${PUPPETEER_HEADLESS}' 'args=${PUPPETEER_CHROME_ARGS}'`);
let browser = await puppeteer.launch({
dumpio: true, // https://github.com/puppeteer/puppeteer/issues/4253
headless: PUPPETEER_HEADLESS, // show the Chrome window
slowMo: PUPPETEER_SLOWMO, // slow things down e.g. by 250 ms
ignoreDefaultArgs: [
"--mute-audio",
],
args: chrome_args,
});

// http://${host}:${port}/json/version
// ws://${host}:${port}/devtools/browser/<id>
// https://chromedevtools.github.io/devtools-protocol
process.env.PUPPETEER_BROWSER_ENDPOINT = browser.wsEndpoint();
browser.disconnect();
}
13 changes: 13 additions & 0 deletions qt/ts/globalTeardown.js
@@ -0,0 +1,13 @@
/* Copyright: Ankitects Pty Ltd and contributors
* License: GNU AGPL, version 3 or later; http://www.gnu.org/licenses/agpl.html */
const puppeteer = require('puppeteer');

module.exports = async () => {
global.server.close();
let PUPPETEER_BROWSER_ENDPOINT = process.env.PUPPETEER_BROWSER_ENDPOINT;

if(PUPPETEER_BROWSER_ENDPOINT) {
let browser = await puppeteer.connect({browserWSEndpoint: PUPPETEER_BROWSER_ENDPOINT});
await browser.close();
}
};
22 changes: 22 additions & 0 deletions qt/ts/gulpfile.js
@@ -0,0 +1,22 @@
/* Copyright: Ankitects Pty Ltd and contributors
* License: GNU AGPL, version 3 or later; http://www.gnu.org/licenses/agpl.html */
var gulp = require("gulp");
var concat = require('gulp-concat');
var resolveDependencies = require('gulp-resolve-dependencies');

var ts = require("gulp-typescript");
var tsProject = ts.createProject("tsconfig.json");

gulp.task("reviewer", function() {
return gulp
.src(["src/reviewer.ts"])
.pipe(resolveDependencies({
pattern: /^\s*\/\/\/\s*<\s*reference\s*path\s*=\s*(?:"|')([^'"\n]+)/gm
}))
.on('error', function(err) {
console.log(err.message);
})
.pipe(tsProject())
.pipe(concat('reviewer.js'))
.pipe(gulp.dest("../aqt_data/web/"));
});

0 comments on commit a778a19

Please sign in to comment.