sack.vfs

Quick summary of features/subsystems.

• WebSocket/HTTP/HTTPS network library. UDP sockets (http,UDP replacment).
• JSON/JSON6 (stream)parser,
• JSOX (streaming) parser,
• Node addon for a custom virtual and physical file system interface (fs replacement).
• USB keyboard raw input (windows)
• Graph Database and Object Storage.
• Primitive Natural Language tokenizer, and attributed text type.
• OpenSSL/LibreSSL Certificate Generation (https, wss).
• Salty Random Generator - procedural random content generator.
• Simple Play a Sound. (windows, but can port to openAl and cross platform)
• Process control; launch and control tasks launched.
• Sqlite/ODBC database driver; minimal SQL interface, open and do.
• Option/Configuration Database support using sqlite/odbc interface; stores in a heirarchy of options per program and global.
• Monitor filesystem for changes to files (caching server update).
• Configuration file parser/log analyzer based on word token matches.
• opt-out per-worker access to tasks and filesystem access.

Requirements

All Systems

• Node.
• npm install -g node-gyp
• a working build environment
  • (win32) npm install -g windows-build-tools - if these are not initially installed, it will require a system restart, without any visible notice; but the install will not complete until restarted.
  • ninja, gcc
• (opt) npm install -g cmake-js and cmake.

ODBC and uuid support. (can be disabled)

ODBC connectivity is provided as an alternative to the sqlite interface, providing connectivity with all existing databases. It is not required; Sqlite itself is not 'required' but special steps have to be taken to disable it. When enabled, the SACK library uses it to configure some runtime options. Options may be controlled by an 'interface.conf' configuration file.

Various Linux

• yum install gcc-c++ libuuid-devel unixodbc-devel
  • (uuid/uuid.h sql.h)
• apt-get install uuid-dev unixodbc-dev
  • (uuid/uuid.h sql.h)
• pacman -S unixodbc util-linux
  • (sql.h uuid/uuid.h(probably already available, fs2util) )
• (?)emerge unixodbc

Mac

• (ODBC might be optioned out; just uses sqlite typically)
• brew (brew install unixODBC)

Windows

• None additional

Usage

const sack = require( 'sack.vfs' );

import {SACK} from "sack.vfs";

Overview of SACK Namespace Object

{
    ComPort(comport) - access to com ports.
     - A jsox (JavaScript Object eXchange) parser. (JSON5/6 input compatible)
        parse(string) - result with a V8 object created from the json string.  
        begin( cb ) - begin parsing JSOX stream; callback is called as each value is completed.
        stringifier() - create a reusable stringifier which can be used for custom types
        stringify(object,replacer,pretty) - stringify an object; same API as JSON.
    JSOX - A jsox parser. (JSON5 input compatible)
        parse(string) - result with a V8 object created from the json string.  
        begin( cb ) - begin parsing JSOX stream; callback is called as each value is completed.
        stringify(obj [,stringifer [,formatter])
    JSON6 - A json6 parser. (JSON5 input compatible)
        parse(string [,reviver]) - result with a V8 object created from the json string.  
        begin( cb ) - begin parsing JSON6 stream; callback is called as each value is completed.
        stringify(obj [,stringifer [,formatter])
    JSON - A json parser.
        parse(string) - result with a V8 object created from the json string.  
        begin( cb ) - begin parsing JSON stream; callback is called as each value is completed.
    Sqlite(dbName) - an interface to sqlite which can open databases in the volume.
            parameters - ( databaseName )
                databaseName maybe a simple ODBC name; if the name ends with .db, it is assumed to be 
                a sqlite database.  If ODBC is not available, it may be assumed that the name will 
                just be a sqlite database name. Extra syntax - if the name starts with a $, then the 
                next word after $ and before the next @ or $ is used to select a sqlite vfs name.
                ( https://sqlite.org/vfs.html )  
                if the name is 'sack' then after @ and before the next $ is specified a mounted filesystem name.
        Sqlite has methods available on it to access native program options.
        Sqlite.op( opName, defaultValue ) - read/write default option database option.
        Sqlite.so( opName, newValue ) - write new value to default option database.
    ObjectStorage( fileName [, version] ) - open a JS object storage database. 
    Volume([mountName,]fileName[,version]) - a virtual disk partition that holds files.
          mountName - the mount name used to mount to volume as a filesystem; it may be referenced 
                later in the string passed to Sqlite.  It may be `null` if it is anonymous mount.
          if no parameters are passed, a Volume object representing the native filesystem is returned.
    Volume.mkdir(pathname) - utility function to make directories which might not exist before volume does; 
            (Volume() auto creates directories now if specified path to filename does not exist)
            parameters - (pathname)
                path to create- will created missing parent path parts too.
    Volume.readAsString( filename ) - memory map map a file from native filesystem as a utf8 string.
    Volume.mapFile( filename, loaded_callback ) - memory map map a file from native filesystem as an arrayBuffer.
            if loaded callback is specified, a thread is created that touches every page in the file, when it's done
            the callback is called with no parameters.
    File - some native filsystem utility methods(?)
    FileMonitor - receive event notifications when files change
    SaltyRNG(feed salt callback) - creates a random number generator
    TLS - namespace for utilities to generate certificates/keys
        genkey( length [,password]) - Generates a keypair
        pubkey( {options} ) - gets public key of a keypair or certificate
        gencert( { options } ) - Generates a self signed certificate.
        genreq( {options} ) - Generates a certification request
        signcert( {options} ) - uses a certificate to sign a certificate request
        validate( {options} ) - validate a certificate against a certificate chain.
        expiration( certificate ) - gets the expiration of a certificate as a Date().     
    WebSocket - Websocket interface
        Client( ... ) - create a websocket client
        Server( ... ) - create a websocket server        
        Thread - Helper functions to transport accepted clients to other threads.
            post( accepted_socket, unique_destination_string ) - posts a socket to another thread
            accept( unique_destination_string, ( newSocket )=>{} ) - receives posted accepted sockets
    HTTP - HTTP Request Method
        get( {options} ) - synchronous http request
    HTTPS - HTTPS Request Method
        get( {options} ) - synchronous https request
    Network - Raw network utilities
        Address( address [,port] ) - holder for network addresses.
        UDP( ... ) - UDP Socket to send/received datagrams.
    Config - Process configuration files; also streaming text parsing
        (methods)[#Config_Methods]
    ComPort - (see below)
    
    log(string) - log a string.
    memDump() - log memory stats (track module memory leaks)
    mkdir() - make a directory in the current path; handles recursive directory creation.
    u8xor(s,k) - utility function to apply a string mask.
    b64xor(s,k) - utility function to just xor a value into a base64 string without expanding the values.
    id() - generate a unique ID (256 bits, 32 bytes, 44chars, trailing '=').
    Id() - generate a short unique ID (12 bytes, 16chars).
    loadComplete() - Indicate to SACK system that this is completed loading (task summoner support;linux;deprecated)

    System - Namespace for SACK system interface routines (The above methods should be moved into this namespace)
	createMemory(name,byte size) - creates a named memory region; memory regions by name are shared on the system.
	openMemory(name) - opens an existing names region; returns an ArrayBuffer which can be mapped to a typed array by application.
	enableThreadFileSystem() - enables mounting file systems specifically for this thread
        allowSpawn() - returns task allowed state
        disallowSpawn() - disble task spawns for this thread
	dumpRegisteredNames() - dumps internal procedure/interface registry
        reboot() - on windows, trigger a reboot.
    Task(options) - an interface for createing and monitoring tasks.
        Task constructor takes an option object.
        end() - cause a task to exit..
        write() - send something to a task.
        send() - send something to a task.
        terminate() - terminate created task.
    // windows only
    registry - an interface to windows registry options
    	set( path, value ) - set a new value in the registry
        get( path )  - get a value from the registry.
    hid - raw keyboard interface, allows identification of different physical keyboard devices.
    
}	

Interfaces

• Volume
• SQLite
• Object Storage
• JSON/JSON6/JSOX
• Websocket Server
  • HTTP Server
• Websocket Client
• HTTP Requests
• UDP - Low level system UDP sockets.
• TCP - Low level system TCP sockets.
• ICMP - ICMP interface (name lookup and ping)
• TLS - certificate creation and validation
• ComPort - COM port IO
• File Changes - Monitors files for changes
• Misc... - Other

GUI Interfaces

These are available when building the GUI version.

• PSI; Image, Image.Color, Render, Intershell, Frame, Registration.

Quick App Helpers

• Http/Websocket - simple script to handle basic resource requests, and provide a websocket endpoint interface. User supplies accept() and connect() callbacks. Also includes basic express-like interface addon.

• PRNG - Small seedable RNGs.

• Events - Small event subscription/dispatch class

• dbUtil - database utilities to read schema from database; builds JS objects representing table structures which can be queried for structure.

• summoner - (experimental/example) this launches and tracks other tasks.

• task-manager - Runs tasks, provides a simple UI which can get the tasks' logs and control run state of tasks.

Node JS Loader Support

Node loader hooks for .json6 and .jsox file types. This loader does an import of sack.vfs and sets globalThis.SACK with the result; and then also assigns globalThis.JSOX and globalThis.JSON6.

The loader hooks are loaded with the option

    --experimental-loader=node_modules/sack.vfs/import.mjs

Previously support for .json6 and .jsox were only provided for require().

Loading either JS version of JSON6 or JSOX should be done after this, allowing them to replace the globalThis versions. The sack.vfs version is still available via SACK.JSOX or SACK.JSON6.

Performance-wise the JS versions have advantages if the information to parse is sourced in JS, while the SACK versions can operate directly on the array buffers loaded from SACK databases, Volumes and files. Which limits the copy one side; otherwise there's a conversion to string from binary and a copy of that string from JS to C potentially.

Loader also supports fetching files over HTTP:// and HTTPS:// (or http:// and https://). This performs a request, and then sends the result as the loaded module.

As a further feature, since web files often just use .js as the extension for modules, and everything loaded from a type="module" script is loaded as a module, the loader supports loading from module://./ (the '.' is important, since really the host is always this computer, and becomes part of the requests when a module loads another module). The module://./ prefix is substituted, and uses environment variables RESOURCE_PATH for where to load default files, if the file path starts with /common then COMMON_PATH environment variable is prepended to the file, and /node_modules file prefix uses NODE_MODULE_PATH environment variable to load modules; native modules will probably not load this way, but typical javascript modules can be loaded this way.

There is a race condition, if http(s):// is used from the local server, then the loading might stall waiting for the server to become available; this is the reason that module:// support was added. TODO: Fix stall, workaround, use module:// instead of http(s):// for local files loaded from a self contained server.

---

Changelog

• 1.3.118(in progress)

• 1.3.117

  • Make TLS hostname checks less optional. Add hooks to TCP for TLS host specification.
  • Make default server module check for SSL_HOST environment when SSL_PATH is specified for certificate chain.
  • Improve specification of hosts and certificates for TCP connections with TLS enabled.
  • Fix/improve chunked transfer-encoding handling for HTTP.

• 1.3.116

  • Set blank headers object if the request had no headers, instead of not setting headers to a value.
  • Fix getting server name extension for TLS connections to be ready in time for 'connect' event.
  • Removed noisy debug logging.
  • (Other minor tweaks and improvements)

• 1.3.115

  • Make HTTP response parser more pessimistic about line endings.
  • Fixed memory double deallocation with multiple onmessage handlers registered.
  • respect command line parameter option to open port in example HTTP server.
  • revised image loader to just return promises (http-ws).
  • removed some noisy logging.

• 1.3.114

  • minimize published package.

• 1.3.113

  • Fix sack source; tcp network async event selection on windows for connect() sockets.

• 1.3.112

  • implement network error callback (SSL fallback).
  • add null check to client websocket protocol send.
  • add ports accessor on ComPort interface.
  • Update LibreSSL from 3.8.0 to 3.9.2.
  • Update libssh2 sources to current (as of 2024/8/23).
  • add ports accessor on TCP interface to get list of open ports and their processes.
  • Improve network stability on windows and linux (SACK).
  • try to auto handle ENETUNREACH, EHOSTUNREACH by trying to resolve IPV4 when IPv6 fails. Added options to prefer V4 or v6 IP addresses.

• 1.3.111

  • fixed missing sources (added network addresses to TCP sockets)

• 1.3.11

  • implement SSL/TLS on TCP sockets.
  • fixes error message releasing JSOX parsed messages.

• 1.3.0(1.3.1)

  • change version, mostly to have a cleaner version point, but there are some significant changes to this, that really obsoletes everything before
  • Add simple TCP network interface.

• 1.1.822

  • Enable option to load ssl certificates for HTTPS app service.
  • JSOX decode fixes. Handle '': as not an error. which also means {: is not an error and it's a zero length string.
  • Enable System Tray interface on windows (linux doesn't have a standard API; and tool doesnt' exist)
  • Fix rare JSOX parsing ref follow where part of path is used and then forks to previous data.
  • Add client SSH module; using libssh2 for support.
  • Add ability to open remote listen channels.
  • Added hook between a channel and the SACK Websocket/HTTP server to a remote connection.
  • fixed some issues with async SQL queries done in a quick batch.
  • improved name lookup from address.
  • Added uv_unref() to async SQL event; allows application to close when code is done.
  • Added ability to resolve names; the websocket is initialized with options on whether to get MAC and or resolve IP to name.
  • Added local and remote MAC addresses to websocket connection for clients and accepted server sockets.
  • Applied some minor fixes to build under termux on android.
  • Added isRunning() method to tasks to return if a task is still running.
  • Added retries and timeout parameters to HTTP request option object.
  • Added ready() method to client-protocol utility class.
  • Updated import.mjs import registration for JSOX, JSON6 for Node 20+.
  • Improved async DB.run() method for SQL databases; removed dead code from old do() only method.
  • Apply auto inherit of stdio, stdout, stderr handles from node under linux.
  • Added some tests for Task() objects; fixed some minor memory leaks.
  • Modified Task send() method to send direct buffer, without format support; previously behaved like printf format.
  • Handle task write/read during close better; may generate an EPIPE, which can be caught with process.on( "EPIPE", ...) in JS script.
  • (JSOX)Quote strings that have a '#' in them.
  • (JSOX)allow '#' to start a comment until end-of-line.
  • (JSOX)fix cummulative error in column count on a comma after a number.

• 1.1.821

  • Linux build fix; don't use windows admin option.

• 1.1.820

  • Fix ability to launch task as admin on windows.
  • Added some tests for windows tasks (testing admin).
  • Added some windows shell utility interfaces; set windows shell, add logout, enable/disable task manager.
  • Added some app utility scripts for syslog scanner.
  • Fix environment variable for windows '~/' path to USERPROFILE instead of HOMEPATH

• 1.1.819

  • published final parser fixes.

• 1.1.818.1-4

  • minor fixes for common language parser multi-rule matching

• 1.1.818

  • Fix fatal error with non-async SQL command that generated an error.
  • Fix race condition crash with new SQL open callback; the db object was deallocated before uv_close event happened.
  • Improved import.mjs experimental loader support; added module://./ support for it.
  • Fixed SQL result error when no records returned for non-promised query (returned undefined instead of empty array).
  • Fix failure to generate callback opening Sqlite databases.
  • Fixed configuration parser; added syslog scanner to generate bans by IP.

• More in CHANGELOG.md