A High speed web server extension for InterSystems Cache/IRIS, YottaDB and JavaScript.
Chris Munt cmunt@mgateway.com
21 June 2024, MGateway Ltd http://www.mgateway.com
- Current Release: Version: 2.7; Revision 36.
- Release Notes can be found at the end of this document.
mg_web provides a high-performance minimalistic interface between three popular web servers ( Microsoft IIS, Apache and Nginx ) and a choice of either M-like DB Servers ( YottaDB, InterSystems IRIS and Cache ) or JavaScript.
A longer read on the rationale behind mg_web can be found here. This document was prepared with my colleague Rob Tweed. Rob has many years' experience in creating and using web application development frameworks.
For those wishing to use mg_web with JavaScript, the essential components can be found in the mg_web_js repository here. A complete library of documentation setting out the rationale for using mg_web with JavaScript can be found here. You will also find some fairly staggering benchmark results together with a number of Docker containers showcasing what the mg_web line of products can do.
mg_web is compliant with HTTP version 1.1 and 2.0 and WebSockets are supported. mg_web can connect to a local DB Server via its high-performance API or to local or remote DB Servers via the network.
Full documentation for installing and configuring mg_web can be found here.
If you are familiar with WebLink, WebLink Developer or EWD then this document will help you get started with mg_web and will explain how existing applications created with those technologies can be run through mg_web. Thanks are due to Rob Tweed and Mike Clayton for designing this interface.
HTTP requests passed to the DB Server via mg_web are processed by a simple function of the form:
Response = DBServerFunction(CGI, Content, System)
Where CGI represents an array of CGI Environment Variables, Content represents the request payload and System is reserved for mg_web use.
A simple 'Hello World' function would look something like the following pseudo-code:
DBServerFunction(CGI, Content, System)
{
// Create HTTP response headers
Response = "HTTP/1.1 200 OK" + crlf
Response = Response + "Content-type: text/html" + crlf
Response = Response + crlf
//
// Add the HTML content
Response = Response + "<html>" + crlf
Response = Response + "<head><title>" + crlf
Response = Response + "Hello World" + crlf
Response = Response + "</title></head>" + crlf
Response = Response + "<h1>Hello World</h1>" + crlf
return Response
}
mg_web also provides a mode through which response content can be streamed back to the client using DB Server write statements.
DBServerFunction(CGI, Content, System)
{
stream = startstream(ByRef System)
// Create HTTP response headers
Write "HTTP/1.1 200 OK" + crlf
Write "Content-type: text/html" + crlf
Write crlf
//
// Add the HTML content
Write "<html>" + crlf
Write "<head><title>" + crlf
Write "Hello World" + crlf
Write "</title></head>" + crlf
Write "<h1>Hello World</h1>" + crlf
return stream
}
In production, the above functions would, of course, be crafted in the scripting language provided by the DB Server.
-
A supported web server. Currently mg_web supports Microsoft IIS, Apache and Nginx.
-
Node.js version 20 (or later) if JavaScript is used.
-
A database. InterSystems Cache/IRIS or YottaDB (or similar M DB Server):
https://www.intersystems.com/ https://yottadb.com/ -
A suitable C compiler if building from the source code.
mg_web is written in standard C (or C++ for IIS). The GNU C compiler (gcc) can be used for Linux systems:
Ubuntu:
apt-get install gcc
Red Hat and CentOS:
yum install gcc
Apple OS X can use the freely available Xcode development environment.
Windows can use the free "Microsoft Visual Studio Community" edition of Visual Studio for building the SIG:
- Microsoft Visual Studio Community: https://www.visualstudio.com/vs/community/
There are built Windows binaries available from:
Currently, you will find built solutions for Windows IIS (x64) and Apache (x86 and x64).
Full documentation for building, deploying and using mg_web will be found in the package: /doc/mg_web.pdf
Copyright (c) 2019-2024 MGateway Ltd,
Surrey UK.
All rights reserved.
http://www.mgateway.com
Email: cmunt@mgateway.com
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
- Initial Release
- Improve the parsing and validation of the mg_web configuration file (mgweb.conf).
- Correct a fault in the network connectivity code between mg_web and InterSystems databases under UNIX.
- Correct a fault in the API-based connectivity code between mg_web and YottaDB under UNIX.
- Introduce an Event Log facility that can be controlled by the log_level configuration parameter. See the documentation (section: 'General mg_web configuration').
- Introduce the ability to stream response content back to the client using M Write statements (InterSystems Databases) or by using the supplied write^%zmgsis() procedure (YottaDB and InterSystems Databases).
- Include the configuration path and server names used for the request in the DB Server function's %system array.
- Introduce HTTP version 2 compliance.
- Introduce WebSocket support.
- Introduce a "stream ASCII" mode. This mode will enable both InterSystems DB Servers and YottaDB to return web response content using the embedded DB Server write commands.
- Reset the UCI/Namespace after completing each web request (and before re-using the same DB server process for processing the next web request).
- Insert a default HTTP response header (Content-type: text/html) if the application doesn't return one.
- Correct a fault in WebSocket connectivity for Nginx under UNIX.
- Introduce a configuration parameter ('chunking') to control the level at which HTTP chunked transfer is used. Chunking can be completely disabled ('chunking off'), or set to only be used if the response payload exceeds a certain size (e.g. 'chunking 250KB').
- Introduce the ability to define custom HTML pages to be returned on mg_web error conditions. Custom pages (specified as full URLs) can be defined for the following error conditions.
- DB Server unavailable (parameter: custompage_dbserver_unavailable)
- DB Server busy (parameter: custompage_dbserver_busy).
- DB Server disabled (parameter: custompage_dbserver_disabled)
- Introduce the functionality to support load balancing and fail-over.
- Web Path Configuration Parameters:
- load_balancing [on|off] (default is off)
- server_affinity variable:[variable(s)] cookie:[name]
- Web Path Configuration Parameters:
- Correct a fault that led to response payloads being truncated when connecting to YottaDB via its API.
- Correct a fault that led to web server worker processes crashing if an error occurred while processing a request via a DB Server API. Such error conditions will now be handled gracefully and an appropriate HTTP error code returned to the client.
- Correct a memory leak that could potentially occur when using Cookies to implement server affinity in a multi-server configuration.
- Introduce a global-level configuration parameter to allow administrators to set the size of the working buffer for handling requests and their response (parameter: request_buffer_size).
- Correct a fault that led to mg_web not working correctly under Nginx on the Raspberry Pi.
- Ensure that API bindings to the DB Server are gracefully closed when the web server terminates the hosting worker process. This correction does not affect configurations using network based connectivity between mg_web and the DB Server.
- Improved error reporting (to the event log).
- Miscellaneous minor bug fixes.
- Correct a regression in the generation of chunked responses (regression introduced in v2.1.13).
- Add functionality to parse multipart MIME content. In addition to the inclusion of 'helper functions' to parse multipart content in the DB Superserver code base, this update will also parse such content on the web server side to extract variables used for Server Affinity.
- This enhancement requires DB Superserver version 4.2; Revision 21 (or later).
- Introduce a mechanism through which DB Servers can be excluded from Load-Balancing and Failover. DB Servers marked in this way are usually reserved to enable specific applications to be accessed through the hosting path.
- Make DB Server names case-insensitive in the configuration and in any server affinity variables.
- Introduce support for request payloads that exceed the maximum string length of the target DB Server.
- This enhancement requires DB Superserver version 4.3; Revision 22 (or later).
- Mark all DB Servers for a path as being 'online' after a request fails on account of all servers being being marked 'offline'. This will allow subsequent requests to succeed if, in the meantime, a participating DB Server becomes available.
- Introduce a configuration parameter (health_check) to instruct mg_web to retry connecting to DB Servers marked as 'offline' after the specified period of time.
- Example: health_check 600 - with this setting the DB Server will, if marked 'offline', be retried after 600 seconds (i.e. after 10 minutes have elapsed since marking the DB Server 'offline').
- Introduce support for TLS-secured connectivity between mg_web and the DB Superserver.
- This enhancement is available for InterSystems DB Servers only.
- This enhancement requires DB Superserver version 4.4; Revision 23 (or later).
- Support the renamed TLS libraries introduced with OpenSSL v1.1.
- libeay32.dll was renamed as libcrypto.dll (or libcrypto-1_1-x64.dll under x64 Windows and libcrypto-1_1.dll for x86 Windows).
- ssleay32.dll was renamed as libssl.dll (or libssl-1_1-x64.dll under x64 Windows and libssl-1_1.dll for x86 Windows).
- Correct a memory initialization fault that could occasionally lead to connectivity failures between mg_web and the DB Superserver.
- Recommend that mg_web is not used with DB Superserver v4.4.23.
- Make DB Server names case-insensitive in any server affinity cookie values (the same change was applied to server affinity variables in v2.1.17).
- Improve the validation of values assigned to server affinity variables.
- Introduce a verbose (v) log level. If set, the key processing steps involved in extracting DB server affinity variables (and cookies). The DB server name chosen will be recorded for each request.
- Introduce Administrator Facilities, implemented as REST requests. The following operations are included:
- Listing the internal status of mg_web.
- Retrieving the configuration file.
- Retrieving the event log file.
- Marking individual servers online/offline.
- Improve the granularity of error reporting.
- Introduce a DB Server configuration parameter (connection_retries) to control the number of attempts (and the total time spent) in connecting to a DB Server before marking it offline.
- connection_retries number_of_connection_retries/total_time_allowed
- Introduce a DB Server configuration parameter (idle_timeout) to limit the amount of time that a network connection will remain in the pool without receiving any work. This parameter should be included in server configuration blocks.
- idle_timeout timeout_in_seconds
- Example: idle_timeout 300 (close down network connections that have been idle for more than 5 minutes.
- This enhancement requires DB Superserver version 4.5; Revision 26 (or later).
- Correct a regression that led to request payloads not being correctly transmitted to the DB Server from Nginx-based mg_web installations.
- This regression was introduced in v2.2.18.
- This change only affects mg_web for Nginx.
- Remove an unnecessary "mg_web: Bad request" error that was previously recorded in the Apache event log for requests that were destined to be served by modules other than mg_web.
- Correct a fault that led to mg_web connections erroneously closing down if no value was specified for the (optional) idle_timeout configuration parameter.
- Documentation update.
- Introduce a configuration parameter (DB Server section) to limit the number of connections created to a DB Server:
- max_connections
- When the limit is reached, and all connections in the pool are busy, additional requests will queue for a period up to the time allowed in the timeout setting.
- Correct a fault in memory management for the Nginx solution.
- This fault resulted in requests occasionally failing with SIGSEGV errors.
- Improve the mechanism through which WebSocket functions are invoked. Instead of embedding the WebSocket function name and path in the client-side script, a mapping must be created in the appropriate location block of the configuration. For example:
- websocket mywebsocket.mgw websocket^webroutine
- Use '/[location_path]/mywebsocket.mgw' in the client-side script. When invoked, this URL will map to 'websocket^webroutine' on the DB Server.
- Introduce support for Server-Sent Events (SSE).
- Note that in order to use this facility, DB Superserver v4.5.32 must be installed.
- Ensure that the 'administrator: off' configuration setting is properly honoured.
- Return a HTTP status code of '504 Gateway Timeout' if a request to the DB Server times-out.
- Previous versions would return '500 Internal Server Error' on response timeout.
- A custom form can be created to override the default response by defining the form to be returned in parameter: custompage_dbserver_timeout
- Correct a fault in the management of SSE channels that could lead to infinite loops on channel closure - particularly when used with the JavaScript Superserver.
- Update the WebLink compatibiity shim code listed here. This update corrects a fault in the code for processing multi-part requests.
- Record (in the event log) extra error information when connections made through the YottaDB API fail.
- For UNIX systems, record at initialisation-time the user and group under which the hosting web server worker process is running.
- For example: configuration: /opt/nginx1261/conf/mgweb.conf (user=nobody; group=nogroup).
- Correct a fault that resulted in the build for the Apache module failing under UNIX.
- This regression was introduced in v2.7.35.