feat(developer): Keyman Developer Server

[mcdurdin]

Keyman Developer Server

Initial version of Keyman Developer Server, running on Node.js. This PR has a few too many changes -- sorry -- but hopefully a large proportion of them are boilerplate! Happy to walk reviewers through code.

This starts the process of moving the internal web server in Keyman Developer IDE to a standalone module. The initial version supports the KeymanWeb keyboard debugger, which is the primary module which is currently needed outside of Keyman Developer IDE.

Rationale

I was working on updating the existing web keyboard debugger following feedback from ELTW. The existing debugger had a Delphi backend and a HTML/JS frontend. I was finding that I had reached the limits of the existing design, and the whole module really needed rewriting: the code was pretty rudimentary. I was also hitting some limitations with the Delphi backend, in particular:

• There was no support for WebSockets (meaning 'live reload' capability would need to poll);
• The server code is multithreaded, with each request running on a separate thread, which means managing locking -- more difficult in practice than it might seem (essentially we had One Giant Lock which made the multithreaded approach kinda useless anyway);
• Working with JSON is verbose and somewhat fiddly in Delphi.

Given the need to rewrite, I felt that choosing Node.js made the most sense:

• We already use Node.js for the lexical model compiler (as well as some of our websites, and some KeymanWeb tests), so we had an embedded Node.js executable bundled with the project;
• Javascript/Typescript is one of our primary dev languages;
• It's open source and cross-platform.

Benefits

This new server module has a number of benefits:

1. It can run standalone from the IDE.
2. It can run on Windows, macOS and Linux.
3. It has a coherent API which I will publish in due course (accessible only on localhost).
4. It uses WebSockets to notify users of updated keyboards, models, fonts and packages -- as opposed to polling which is both noisy and unstable.
5. It optionally integrates with ngrok, which provides an ephemeral public https:// address so that:
   • You can easily access your test projects from any online device -- without needing to know about network settings.
   • You can share a test keyboard or model with another user simply by sharing the ngrok address with them.
6. It becomes practical to enable multiple instances of the IDE (e.g. one for each project); this is a setting in the Keyman Developer Options dialog.
7. It maintains the testing keyboards, models, packages and fonts (aka debug objects) across sessions -- making it much easier to test keyboard and model interactions.
8. It avoids file locking issues with the compiler by caching all debug objects.
9. As it is written in Node, it has no dependencies on closed source development tools.

Features

1. Seamless integration with the IDE -- start / stop / test keyboard / recompile
2. Live reload as soon as you compile a debug object
3. Drag-drop any debug object onto the page to load it in (or use the Upload button)
4. Uses Bootstrap for clean responsive design
5. API for interacting with debug server
6. Tray icon to shutdown / control server (on Windows)
7. Integrated all functionality into a single page (previously package downloads were a separate page)
8. Download link for latest (stable) version of Keyman for current device

Architecture

This is intended to be the first step in the way to having a fully cross-platform Keyman Developer. As such, Keyman Developer Server runs in Node.js and I have already tested on Linux (macOS not yet). Longer term, we may publish it as an npm module (it is currently called @keymanapp/developer-server).

Keyman Developer Server is located in /developer/server/ and the Keyman Developer build process (still under /windows/src/developer/ for now) will build the server when do a release build.

For your own builds, /developer/server/build.sh [--production] will build the server, to dist/. The server can be then run with node . If you specify to build to production, the build will create a build/ folder, and this will be a self-contained version, ready to deploy. The Keyman Developer build bundles this folder, both into the Keyman Developer installer, and into the kmcomp.zip archive for other platforms.

The server runs in its own process, and its Node.js console window will be hidden by default when started from Keyman Developer. On Windows, the process runs a taskbar icon which allows you to shutdown the server, or show/hide the console window.

When you start testing a keyboard, model or package, or when you compile one of these which is already under test, the server will be notified of the updated file via an api, /api/<object-type>/register. The server caches the updated file to a data folder, so it will be available even if the original file is removed, and across sessions. The server automatically expires old cached objects, with a limit of 12 of each object type in cache (this number may be too high?).

The server will then notify all client web browsers, via WebSockets, of the updated files (a 'refresh' notification).

The client web browser (viewing e.g. http://localhost:8008) will reload the list of keyboards and models from a pretty awful /inc/keyboards.js endpoint (see TODO), and then read the debug objects themselves from /data/<object-type>/<filename>.

ngrok Integration

Keyman Developer Server includes integration with ngrok, which provides a public URL that works through firewalls and NAT. This is not enabled by default, and ngrok is not bundled with Keyman Developer at this time, but this can all be configured in Keyman Developer Options dialog (see screenshot below).

The user will need to:

1. Download ngrok (use the button in the Options dialog) -- this downloads to %appdata%\Keyman\Keyman Developer\ngrok.exe
2. Create a free ngrok account (use the button in the Options dialog)
3. Paste the auth token from their ngrok account (presented after creating the account, or accessible through the link in the Options dialog)
4. Optionally, configure region -- helps with performance to choose the closest region

Once ngrok is configured and running, the IDE will include the public URL in the Tools/Web Debugger menu, and also in the available test addresses within the Keyboard, Package and Model editors (see screenshots).

This public URL will be available for the duration of the Keyman Developer Server session (use the option Leave server running after closing IDE for a longer session!). You can share this URL to your devices or even with other users (who need not be in the same location as you), which enables an instant user test-feedback cycle.

Note: if the user runs ngrok for other purposes, they may prefer to configure ngrok externally. In this case, Keyman Developer will not show the ngrok URL in the UI, but otherwise it should still work in the same way.

Screenshots

• Refreshed web debugger - running on a 'phone'

• Refreshed web debugger - normal web interface

• Keyman Developer Options

• Tools/Web Debugger menu

• Running remotely with a publicly-accessible URL:

• Listing available endpoints for testing (and yes, QR Code works for ngrok addresses):

For upcoming pull requests

• Documentation
• Refactor the old /inc/ endpoints (currently all a bit ... dodgy)
• Further test on fonts -- are they working correctly?
• Move ngrok.exe download to %appdata%\Keyman\Keyman Developer\Server
• Hide Upload button, disable drag-drop if not on localhost (/upload already blocked); don't forget to document this

Future work

• Move the 'start test' IDE-centric UI action to a non-UI action (then it will be available from kmcomp, project view, etc).
• Support watching a folder or hierarchy for changing debug objects (good for cross-platform run)
• Use Typescript instead of Javascript for /src/site
• Consider using npm for /src/site (avoid bundling Bootstrap, etc)
• We currently bundle compiled versions of two native code packages: node-windows-trayicon and hetrodo-node-hide-console-window-napi. This reduces the build complexity but we need a better approach before publishing to npm registry
• Publish to npm registry

[keymanapp-test-bot]

User Test Results

Test specification and instructions

✅ SUITE_BASIC_USAGE: Basic usage tests

• ✅ TEST_SERVER (PASSED): The icon does show up in the notification area.
  (notes)
• ✅ TEST_BASIC_KEYBOARD (PASSED)
• ✅ TEST_BASIC_PACKAGE (PASSED): The .kps loaded and built without any issue. The package does appear and can be downloaded for installation.
• ✅ TEST_BASIC_MODEL (PASSED): Since the ZWSP doesn't get output, the model is crippled in away, but generally speaking it's kind of work in and of itself so long as you keep picking from the suggestion. :)

✅ SUITE_WEB_TEST_FUNCTIONALITY: Testing aspects of the user interface in the Web Test page

• ✅ TEST_ABOUT (PASSED): The About link does work. It shows the version number of the KD, copyright line and a link to help which is broken for the time being.
  (notes)
• ✅ TEST_UPLOAD (PASSED): When we re-tested in a localhost scenario, it worked correctly.
• ✅ TEST_STATUS (PASSED): It does reflect what's selected.

✅ SUITE_NGROK: Testing the ngrok integration

• ✅ TEST_NGROK_INSTALL (PASSED): The ngrok setup is not straightforward. I was having an issue with either the token or the account verification part which resulted in not having it working. After rubber ducky with Marc, this was sorted out and the url for web test with "ngrok.io" shows up.

✅ SUITE_INTEROPERABILITY: Verify that various processes function correctly

• ✅ TEST_RELOAD (PASSED): The change reflects on the web test page right after compiling the keyboard.
• ✅ TEST_LEAVE_LOADED (PASSED): The KD server icon is not seen in the notification area, but the web test page is still working for all previously built keyboards.
• ✅ TEST_MOBILE (PASSED): Keyboards and models work as expected. One thing that might be a separate issue is that a key cap text with a custom font is shown up as tofu. (notes)

Test Artifacts

• Keyman for Android apk
• FirstVoices Keyboards for Android apk
• KMSample1 apk
• KMSample2 apk
• Keyman for macOS
• KeymanWeb Test Home
• Keyman for iOS (simulator image)
• FirstVoices Keyboards for iOS (simulator image)
• Keyman for Windows
• Keyman Developer
• FirstVoices Keyboards for Windows

[MakaraSok]

SUITE_WEB_TEST_FUNCTIONALITY: Testing aspects of the user interface in the Web Test page

• TEST_ABOUT: PASSED The About link does work. It shows the version number of the KD, copyright line and a link to help which is broken for the time being.
  
  About page
  
  
  
  Page not found on the Help link
  
  
  
• TEST_UPLOAD: FAILED Drop a .js file and .kmp of a keyboard on the test page and it doesn't reconize the files.
  
  
  
• TEST_STATUS: PASSED It does reflect what's selected.

SUITE_NGROK: Testing the ngrok integration

• TEST_NGROK_INSTALL: FAILED No url shown in the box has ngrok.io after clicking on "test Keyboard on web".
  
  
  

SUITE_INTEROPERABILITY: Verify that various processes function correctly

• TEST_RELOAD: PASSED The change reflects on the web test page right after compiling the keyboard.
• TEST_LEAVE_LOADED: PASSED The KD server icon is not seen in the notification area, but the web test page is still working for all previously built keyboards.
• TEST_MOBILE: BLOCKED No url with ngrok.io found in the web address box. This is blocked by the issue with ngrok found above.

[MakaraSok]

SUITE_NGROK: Testing the ngrok integration

• TEST_NGROK_INSTALL: PASSED The ngrok setup is not straightforward. I was having an issue with either the token or the account verification part which resulted in not having it working. After rubber ducky with Marc, this was sorted out and the url for web test with "ngrok.io" shows up.

SUITE_INTEROPERABILITY: Verify that various processes function correctly

• TEST_MOBILE: PASSED Keyboards and models work as expected. One thing that might be a separate issue is that a key cap text with a custom font is shown up as tofu.

[mcdurdin]

• TEST UPLOAD: FAILED Drop a .js file and .kmp of a keyboard on the test page and it doesn't reconize the files.

It turns out that this is because the page is only allowed to accept dropped files when it is loaded on localhost, for security reasons. The drag+drop interface and upload menu item should be disabled in this context (see checkbox in 'For upcoming pull requests' section).

Documentation required on this limitation!

SUITE_WEB_TEST_FUNCTIONALITY

• TEST_UPLOAD: PASSED When we re-tested in a localhost scenario, it worked correctly.

[darcywong00]

Note that if you run with your own x64 version of Node, you need to be running v17;

Is it fine to use the current LTS of Node v16?

[mcdurdin]

Note that if you run with your own x64 version of Node, you need to be running v17;

Is it fine to use the current LTS of Node v16?

Good catch. This note is now irrelevant, and I have removed it, as we are able to build the necessary modules on demand. You should not need to run a specific version of Node, but should be on Node v16 (LTS) of course because that's our current supported version.

[rc-swag]

Just some comments re dead code

[darcywong00]

Is there a way on the test page for the user to test portrait vs landscape layout?

[mcdurdin]

Is there a way on the test page for the user to test portrait vs landscape layout?

This is not currently supported. Given the keyboard functions identically, I'm not inclined to prioritise this at this time. At some point we can add a rotate button.

[darcywong00]

lgtm

[darcywong00]

I don't think the iOS build fail is related to this. Maybe certificate issue?

[rc-swag]

lgtm

[keyman-server]

Changes in this pull request will be available for download in Keyman version 15.0.188-alpha