Web MIDI

[ad8e]

After prompting from #1, I explored Web MIDI a bit. It does seem convenient.

Fluidsynth appears to be a usable MIDI device that I have now installed.

Firefox doesn't support Web MIDI without a permission add-on. Also, it appears to have issues with Snaps on Ubuntu.
Chromium on Ubuntu Linux doesn't support Web MIDI, another bug report here.
My phone has Safari, which doesn't support Web MIDI.

Bitwig Studio in Windows 10 on VMWare is unresponsive; clicks take 30 seconds to register.
Fluidsynth fails to create a MIDI thread in Windows 10 on VMWare.

Windows 8: Bitwig Studio doesn't expose any virtual midi device that Chrome can see, even after adding a generic keyboard in its settings
Windows 8: Fluidsynth (XP) fails to create any MIDI devices, outputs some errors

So I'm still having compatibility issues getting Web MIDI to work. Otherwise, it should be pretty easy to output MIDI; the code to do it is short, and I know the calculations already.

[davidhaley]

Firefox doesn't support Web MIDI without a plugin

It's working for me; however, I had to serve the page from a server instead of loading it directly in the browser.

(I use yarn, but npm works as well)

1. npm init
2. yarn add http-server
3. Add this to package.json:

  "scripts": {
    "start": "http-server"
  },

4. yarn start

Visit localhost:8080, and then open the keyboard.html from there (and then accept Firefox's prompt to allow MIDI access).

[ad8e]

The keyboard in this repo should support Web MIDI, and is ready for testing. 10% chance it works, since I can't test it.

If MIDI devices are found, a dropdown will appear right below the piano black-and-white dashes. Selecting one of those should output MIDI to the device.

The diff is at https://github.com/ad8e/Just-intonation-keyboard/compare/6a0fd0d..main

I ran sudo modprobe snd-seq-dummy ports=4, and aconnect -i -o does list the 4 new dummy MIDI ports, but this test website in Firefox still can't see them. (I don't expect you to debug my system; feel free to ignore this.) In my keyboard in a local webserver, it complains "Silently denying site request for MIDI access because no devices were detected. You may need to restart your browser after connecting a new device.", then "Web MIDI error: DOMException: WebMIDI requires a site permission add-on to activate". I didn't set the midi HTTP Permission Policy as MDN tells me to, but it seems your local webserver version didn't need either the Permission Policy or the site permission add-on.

[davidhaley]

Good evening,

I pulled the latest code and tested it out. My midi devices are appearing in the dropdown:

However, if I select a device and play a note, the active device is reset and thinks the selected index is 0, and then returns undefined trying to access -1. I think what's happening is that playing a note causes a state change event, which reinits midi and the DOM (losing the selected input).

The midi note does send to Bitwig fine (it just doesn't turn off at the moment due to the exception).

If you're okay with me contributing code, I should have more time to look at it later this week.

[ad8e]

I'm totally okay to accept your code; getting patches from you is great. I'll take a look at the statechange in the meantime.

[ad8e]

https://webaudio.github.io/web-midi-api/#dom-midiaccess-onstatechange:
onstatechange. The handler called when a new port is connected or an existing port changes the state attribute.

MIDIPort's state: https://developer.mozilla.org/en-US/docs/Web/API/MIDIPort/state
Either "disconnected" or "connected".

So the Web MIDI docs don't give a hint as to why the midi message can cause the state to change. They seem to say that onstatechange should only fire for big events, like disconnecting your DAW. So your testing is quite invaluable!

[davidhaley]

I'm totally okay to accept your code; getting patches from you is great. I'll take a look at the statechange in the meantime.

Cool, sounds great!

So the Web MIDI docs don't give a hint as to why the midi message can cause the state to change. They seem to say that onstatechange should only fire for big events, like disconnecting your DAW. So your testing is quite invaluable!

I haven't tested this yet, but I wonder if the port connects lazily (i.e. first note send to the device truly activates the connection).

---

Just a thought, I wonder if it makes sense to have a separate toggle/checkbox input: Enable synth (or similar, defaulted to checked), so that the in-page synth is completely independent from MIDI. A use-case is that the user can immediately jam/record MIDI to their DAW without worrying about choosing/configuring a VST/synth (i.e. they can monitor w/low-latency using the page's synth, and then capture the session in a DAW). Since it would be optional, they could also mute the in-page synth, save on performance, and just emit MIDI notes. What do you think?

[ad8e]

Just a thought, I wonder if it makes sense to have a separate toggle/checkbox input: Enable synth (or similar, defaulted to checked), so that the in-page synth is completely independent from MIDI. A use-case is that the user can immediately jam/record MIDI to their DAW without worrying about choosing/configuring a VST/synth (i.e. they can monitor w/low-latency using the page's synth, and then capture the session in a DAW). Since it would be optional, they could also mute the in-page synth, save on performance, and just emit MIDI notes. What do you think?

It depends on what workflows musicians use, and you understand those better than I do. Since you think the use-case is worthwhile, I'll implement it. I'll think a bit about the UI, since there's a state that doesn't make sense: "no in-page synth active + no MIDI output active = nothing happens". I'll try to find some elegant UI representation.

[ad8e]

I added your checkbox.

Since you're already running a local web server, some quality + latency improvements are possible, so I rearranged files and wrote instructions at https://github.com/ad8e/Just-intonation-keyboard#readme.

The web server is optional; it'll degrade gracefully back to the sine wave if you're not running a web server. In that case, the various .js files are also optional; keyboard.html will work on its own for the sine wave.

[davidhaley]

Good evening,

I tested out MIDI/sound today and both are working well - thank you.

---

I tested and noticed that both Firefox/Chrome logged a (caught) exception when the audio-context was being created:

An AudioContext was prevented from starting automatically. It must be created or resumed after a user gesture on the page

I think for security reasons, the browser wants the code that creates the audio-context to run in response to the user interacting with the page. I was able to fix it by moving callMain() and postRun() into a new start() that's exported and called when the user enables sound checkbox. The keyboard.js file was minified, so I unminified it to make the changes (and so that you could read the changes).

Since the user controls enabling sound, it's now disabled by default. I then improved the UI to enable the keyboard if sound or MIDI is on (if the user has MIDI devices), and disable it otherwise (and display a message). I also updated the styles of the configuration panel.

I can open a PR with the changes, so that you can see them.

Edit: link to PR here: #4

Have a great night.

[ad8e]

The reason the MIDI is fixed is probably a nice side effect of preserving the chosen option on MIDI statechange. I still don't know why statechange is firing; perhaps it's as you said, it runs lazily on the first MIDI message.

keyboard.js is actually not editable; it's generated automatically by emscripten. So I can't preserve any changes to it. (That's also why it's minified - all the code hand-written by me is in keyboard.html, and the various .js and wasm files are auto-generated.)

On button press, the AudioContext automatically works, so it doesn't create any issues other than the unpleasant console warning. It's actually more efficient to start it on page load: my testing notes say that starting the AudioContext on first user input adds noticeable latency to the first click, most likely because the AudioWorklet can only be started after the AudioContext.

I can add your UI to this github version of the keyboard. There's no point in starting the keyboard disabled, since pressing anything on the keyboard will enable the AudioContext automatically. But if you often want to keep the sound off, I can add some state to localStorage that preserves the "Sound off" mode. In the prior version, the MIDI dropdown simply doesn't appear unless a MIDI device is selected.

The other changes:

1. renaming synthCheckbox to soundCheckbox. It makes no difference to me, so I'll accept your change.
2. keyboard disabled grey-out + warning. I would prefer to prevent the keyboard from being disabled, rather than show a warning when it is disabled. To accomplish this, if MIDI is turned off, the sound turns itself back on. From what I can see, this will only cause an issue if you unplug all your MIDI devices, then plug one back in. Is this switching an issue for you?
3. Font change to Inter in the config. I didn't have Inter installed, so this changed nothing for me. After I installed Inter, Firefox's Snap sandbox got in the way, haha. Anyway, I can accept this even though I can't see its results. But without vendoring fonts, only people who have Inter installed will see it.
4. The headings and descriptions in the option menu are mostly redundant, so I'll remove them. For example: "Sound", "Configuration options for the in-page synthesizer", "Enable sound".
5. Reset options on page refresh: I think it's better if the page remembers your pre-refresh options. However, I have no way to test if the MIDI options will actually be preserved on refresh. I'll check that the dropdowns and states don't de-sync.
6. Two placeholder options: I'm not seeing the point to having both a "Select a MIDI output device" and "MIDI output off". I think there could be just one placeholder, saying "No MIDI devices available", and changing to "MIDI output off" when a MIDI device is found. "MIDI output off" creates the affordance that MIDI is off, and they can turn it on by clicking the menu.

[ad8e]

Changes merged. Some of the CSS looked odd; I didn't understand the purpose of some lines.

I believe the alignment of options you wanted is possible with CSS Grid. Learning it was too tedious for me at the moment, but I would accept a patch using it. It didn't seem important.

[davidhaley]

keyboard.js is actually not editable; it's generated automatically by emscripten

I thought it might have been generated, but I didn't realize it was by emscripten, cool.

On button press, the AudioContext automatically works, so it doesn't create any issues other than the unpleasant console warning.
But if you often want to keep the sound off, I can add some state to localStorage that preserves the "Sound off" mode

If we're okay with the console warning, then we can do what it did before (run on load, enabled by default). Most of the changes were predicted on us not being okay with that warning. It sounds like we gain performance, and the happy-path bonus of having sound working immediately. I think that's a good idea.

Font change to Inter in the config

This is an artifact of me tinkering with some styles in a web-font that I'm familar with. Apologies - I'll remove that in a PR that I'm about to push (flattening the SCSS to CSS for browser compatibility - perhaps I could set up a build script sometime so that we can benefit from SCSS tooling)

The headings and descriptions in the option menu are mostly redundant

Sounds good!

Reset options on page refresh

Since the new logic requires the user to enable sound, and Firefox remembers the input state, and starting the module initialization required user-input, I worked around that by resetting the state. Since we won't need to do that anymore (reverting to the previous behaviour), we can remove that.

"MIDI output off" creates the affordance that MIDI is off, and they can turn it on by clicking the menu.

That's a very good point. I see what you mean, and agree. Let's do that.

---

I'll submit a PR with the existing SCSS converted to CSS (my mistake - not used to writing plain CSS these days!).

[ad8e]

The CSS is pared down in the latest commit. If you see something it doesn't accomplish anymore because I removed too much (like the now-missing CSS Grid alignment on the Options) or something odd/extra, feel free to point it out. I know little CSS.

[ad8e]

Also, I can leave Inter in if you like it. It creates no harm. I noticed that the default system font stack I took from https://systemfontstack.com/ looks horrendous; Helvetica gets converted to Liberation Sans (EDIT: Nimbus Sans), which is far worse than Noto Sans to my eyes. So there was no intelligent decision-making on my side in the old font decisions.

[davidhaley]

The CSS is pared down in the latest commit. If you see something it doesn't accomplish anymore because I removed too much (like the now-missing CSS Grid alignment on the Options) or something odd/extra, feel free to point it out. I know little CSS.

Sounds great - no worries, it looks fine, I just submitted a PR to flatten the SCSS (browsers don't like SCSS and we would require a build step to compile SCSS into CSS, which is probably not something we want to do yet).

Also, I can leave Inter in if you like it. It creates no harm. I noticed that the default system font stack I took from https://systemfontstack.com/ looks horrendous; Helvetica gets converted to Liberation Sans, which is far worse than Noto Sans to my eyes. So there was no intelligent decision-making on my side in the old font decisions.

No worries as well - Inter is a modern web-safe font that looks okay. We don't have to switch though!

[ad8e]

Neat! I added credits to you in the source. Thanks for all your help!