Skip to content

v3.0.0

Latest
Latest

Choose a tag to compare

View all tags
@dev-embedthis dev-embedthis released this 19 Dec 03:23
· 32 commits to main since this release
v3.0.0
e1115ec

Ioto Device Agent v3.0.0 Release Notes

Release Date: December 9, 2025

Recommended Action

  • Optional Upgrade -- Upgrade only if convenient
  • Recommended Upgrade -- Upgrade recommended but not essential
  • Essential Upgrade -- All users strongly advised to upgrade

Overview

Version 3.0.0 is a major feature and security release that adds HTTP Basic and Digest authentication support to the embedded web server, client-side cache control, event-driven non-blocking I/O for 10x connection scalability, and pre-compressed content serving. This release delivers significant performance optimizations including socket accept improvements, zero-copy body reading, and dynamic buffer management that yield substantial throughput gains under high-concurrency workloads.

This release also incorporates important security hardening discovered through comprehensive fuzzing campaigns, resource management improvements, stability fixes, and new runtime APIs for enhanced fiber and socket control. Quality assurance has been strengthened with an extensive test suite encompassing unit testing, fuzz testing, leak detection, and performance benchmarking across all supported platforms.

Major Features

Web Server Authentication

  • HTTP Basic Authentication - Industry-standard username/password authentication with Base64 encoding
    • Configurable TLS enforcement for secure transmission
    • Support for SHA-256 password hashing
    • Per-route authentication configuration
  • HTTP Digest Authentication - More secure challenge-response authentication
    • MD5 and SHA-256 algorithm support
    • Server-generated nonce values for replay protection
    • Quality of Protection (qop) support
    • Per-route authentication configuration
  • Password Management - New password command-line tool for generating hashed passwords
    • SHA-256 and MD5 password hash generation
    • Integration with authentication configuration

Client-Side Cache Control

  • Route-Based Cache Headers - Configure HTTP cache control headers per route for improved browser and proxy caching
    • Cache-Control, Expires, and Pragma headers per route
    • Extension-based filtering (e.g., cache only .css, .js, .png files)
    • Multiple cache directives: public, private, no-cache, no-store, must-revalidate
    • Natural time string parsing (e.g., 1week, 5mins, 1day)
    • HTTP/1.0 compatibility with automatic Expires header generation

Event-Driven Non-Blocking I/O

  • 10x Connection Scalability - Major architecture enhancement that frees fibers during keep-alive idle periods
    • Support 10x more concurrent connections with the same memory
    • Saves 64-256KB per idle connection (fiber stack)
    • Zero performance impact on active request processing
    • Full backward compatibility - no API changes required

Pre-Compressed Content Serving

  • Automatic Compressed File Serving - Serve pre-compressed .gz and .br files when available
    • Content negotiation based on Accept-Encoding header
    • Prioritizes Brotli (.br) over gzip (.gz)
    • Falls back to uncompressed content if compressed version unavailable
    • Proper Content-Encoding headers

Network Improvements

  • Dual-Stack Network Support - Fixed IPv4/IPv6 dual-stack listening to properly support both protocols simultaneously

Socket Accept and Buffer Optimizations

  • Optimized Socket Accept - Refactored accept to run on main fiber with socket setup in spawned fiber for improved performance
    • New R_WAIT_MAIN_FIBER flag to run wait handlers on main fiber
    • Better resource utilization during connection acceptance
  • Dynamic Buffer Growth - New rGrowBufSize() API to grow buffers to a specific size
  • Runtime Socket Limits - New APIs to get and set socket limits at runtime
    • rGetSocketLimit() - Query current socket limit
    • rSetSocketLimit() - Adjust socket limit dynamically
  • Zero-Copy Body Reading - New webReadDirect() for zero-copy HTTP body reading
  • Static Response Strings - New webWriteResponseString() for efficient static string responses
  • Dynamic Poll Table Growth - Windows/WSAPoll poll table now grows dynamically as needed

Fiber Control Limits

  • Runtime Fiber Limits - Added fiber control limits for better resource management in high-concurrency scenarios

Performance Optimizations

This release includes substantial performance improvements across the stack:

  • 10x Connection Scalability - Event-driven non-blocking I/O frees fiber resources during keep-alive idle periods, enabling 10x more concurrent connections with identical memory footprint
  • Optimized Socket Accept Path - Connection acceptance refactored to run on main fiber with socket setup delegated to spawned fibers, reducing context switching overhead
  • Zero-Copy Body Reading - New webReadDirect() API eliminates buffer copies for HTTP body data, significantly improving throughput for large request bodies
  • Dynamic Buffer Management - Intelligent buffer growth with rGrowBufSize() reduces memory allocation overhead and fragmentation
  • Static Response Optimization - webWriteResponseString() provides fast-path for static string responses without buffer allocation
  • Dynamic Poll Table Growth - Windows/WSAPoll poll table grows on-demand, eliminating artificial connection limits
  • Reduced Memory Pressure - Fiber stack memory (64-256KB per connection) reclaimed during idle periods

Benchmarking Results:

  • High-concurrency workloads show significant throughput improvements
  • Memory efficiency improved under sustained connection load
  • Reduced latency for connection acceptance under load
  • Windows platform performance now comparable to Linux/macOS

New Runtime APIs

  • Socket Linger Control - New APIs for fine-grained socket connection control and graceful shutdown
    • rSetSocketLinger() - Configure socket linger behavior for clean connection termination
    • urlSetLinger() - Control URL client connection linger settings
  • Fiber Block Management - New rStartFiberBlock() API for managing fiber execution blocks and better control flow
  • Time API Enhancements
    • rGetHttpDate() - Format time as RFC 7231 HTTP date string
    • rMakeTime() - Convert struct tm to Time (milliseconds since epoch)
    • rMakeUniversalTime() - Convert struct tm to UTC Time
    • rParseHttpDate() - Parse HTTP date strings
    • rGetElapsedTime() - Calculate elapsed time since a mark
  • String Functions
    • sncat() - Safe string concatenation function
  • Command Execution
    • rRun() - Cross-platform command execution API now available on all platforms
    • rMakeArgs() - Public API for argument parsing
  • URL Client Authentication
    • urlSetAuth() - Set credentials for HTTP Basic/Digest authentication
    • urlParseAuthChallenge() - Parse authentication challenge responses

Security Hardening

  • Digest Authentication Security
    • HMAC-SHA256 nonce generation replacing SHA-256(secret:data) to prevent nonce forgery
    • Server-wide replay protection with nc (nonce count) validation per RFC 7616
    • Constant-time comparison using cryptMatchHmacSha256() for timing-attack resistance
    • Auto-login removed to eliminate security risk of automatic credential caching
    • MD5 deprecation warnings when MD5 algorithm is used
  • Fuzzing Results
    • Fixed invalid reference in JSON parser discovered through comprehensive fuzzing
    • Fixed potential null dereference in HTTP method parsing (fuzzing)
    • Enhanced error handling and boundary condition checks throughout the codebase
    • Resource leak prevention (file descriptors, memory)
  • URL Client Security
    • Quote escaping defense against header injection from malicious servers
    • RFC-compliant challenge parsing with proper quoted string handling
    • Parameter length limits (8KB max) to prevent memory exhaustion
    • HTTP warning when Basic auth used over unencrypted connections

Resource Management Improvements

  • Fixed file descriptor leak in putFile operation that could cause descriptor exhaustion
  • Fixed premature memory free in web getFile operation
  • Improved fiber exhaustion handling with better error reporting and diagnostics

Bug Fixes

Fixes

  • Fixed upload forms - Resolved issues with multipart form-data file upload handling
  • Fixed keep-alive timeout - Corrected HTTP keep-alive connection timeout behavior
  • Fixed hang in webSendFile - Resolved deadlock condition when sending large files
  • Fixed fiber exhaustion handling - Improved resource management when fiber pool is depleted
  • Fixed file descriptor leak - Prevents descriptor exhaustion in long-running applications
  • Fixed premature free in web getFile - Resolved use-after-free condition
  • Fixed ISO-8601 parsing - Correctly preserve milliseconds with 'Z' UTC indicator and numeric timezone offsets
  • Fixed ETag memory allocation - Parsing ETag memory allocation issue using new R_TEMPORAL_VALUE list flag
  • Fixed 304 response handling - 304 Not Modified responses now sent correctly
  • Fixed duplicate Connection header - Eliminated duplicate Connection header in responses
  • Fixed max body limit checking - Body size limit now correctly excludes header size
  • Fixed PUT file handling - Proper error handling for premature disconnect and insufficient body data
  • Fixed localhost socket binding - Prefer IPv4 for localhost compatibility (previously used AF_UNSPEC)
  • Fixed socket close with no-linger - Graceful shutdown now only performed when linger is enabled
  • Fixed WebSocket close errors - Suppress spurious error logs when closing WebSocket connections
  • Fixed URL show flags - Corrected interaction between show flags and URL_NO_LINGER option
  • Fixed socket connection strategy - Two-pass connection (IPv4 first, then IPv6) for better server compatibility
  • Fixed socket connection verification - Use getpeername() to verify connection truly established after SO_ERROR check
  • Fixed macOS socket connection - Workaround for macOS SO_ERROR returning 0 when connection not ready
  • Fixed macOS dual-stack localhost - Use IPv4 only for localhost due to dual-stack reliability issues
  • Fixed URL connection error handling - urlStart now properly returns error on connection failure
  • Fixed PUT file upload limits - PUT file size now checked during upload, not just at completion
  • Fixed upload boundary parsing - Improved multipart form boundary delimiter handling
  • Fixed Windows pollFds clearing - Resolved issue where pollFds were not being cleared properly in rFreeWait on Windows
  • Fixed rSetFiberStack - Corrected fiber stack configuration behavior

Security Fixes

  • Fixed invalid reference in JSON parser (discovered via fuzzing)
  • Fixed null dereference in parseMethod (discovered via fuzzing)
  • Enhanced input validation throughout web server
  • Improved boundary condition checks across all modules

Build System

  • Hardened configure phase to handle missing build/.app file gracefully
  • Refactored app selection mechanism for cleaner, more maintainable builds
  • Fixed conditional compilation when building without cloud support (SERVICES_CLOUD)
  • Fixed Windows build preparation with improved prep.nmake
  • Removed obsolete webserver.vcxproj from Visual Studio projects
  • Regenerated project files for all platforms
  • Updated dependencies across all packages

Compiler & Warnings

  • Fixed compiler warning in cryptSha1Process function

Testing & CI/CD

  • Fixed missing test files for upload testing
  • Enhanced test coverage for authentication flows
  • Enabled complete test suite for all platforms
  • Fixed test setup and isolation issues
  • Fixed web temporary file handling in tests
  • Enhanced CI/CD pipeline reliability with better timeout handling

Configuration

  • Fixed initialization to use default 'unit' app when build/.app doesn't exist
  • Improved cloud service conditional compilation with proper #if SERVICES_CLOUD guards

API Changes

New Command-Line Tools

  • password - Generate hashed passwords for web server authentication
    • password [-a sha256|md5] [-p password] username - Generate password entries
    • Supports SHA-256 (default) and MD5 hashing algorithms

New Runtime APIs

Socket and Fiber Control:

  • rSetSocketLinger(socket, enable, timeout) - Configure socket linger behavior for clean connection termination
  • urlSetLinger(url, enable) - Set URL client socket linger mode
  • rStartFiberBlock() - Manage fiber execution blocks for better control flow
  • rGetSocketLimit() - Query current runtime socket limit
  • rSetSocketLimit(limit) - Adjust socket limit dynamically at runtime

Buffer Management:

  • rGrowBufSize(buf, size) - Grow buffer to specific size

Wait Handler Control:

  • R_WAIT_MAIN_FIBER - Flag to run wait handlers on main fiber instead of spawned fiber

Time Functions:

  • rGetHttpDate() - Format time as RFC 7231 HTTP date string
  • rMakeTime() - Convert struct tm to Time (milliseconds since epoch)
  • rMakeUniversalTime() - Convert struct tm to UTC Time
  • rParseHttpDate() - Parse HTTP date strings
  • rGetElapsedTime() - Calculate elapsed time since a mark

String and Command Functions:

  • sncat() - Safe string concatenation function
  • rRun() - Cross-platform command execution API
  • rMakeArgs() - Public API for argument parsing

URL Client Authentication:

  • urlSetAuth(up, username, password, authType) - Set credentials for HTTP authentication
  • urlParseAuthChallenge(up) - Parse authentication challenge responses

Web Server:

  • webSetCacheControlHeaders(web) - Set cache control headers based on route configuration
  • webFreeRanges() - Free range request resources
  • webReadDirect(web, buf, size) - Zero-copy HTTP body reading
  • webWriteResponseString(web, str) - Efficient static string response writing

List Fixes:

  • R_TEMPORAL_VALUE - List flag that auto-clones string values when added to the list is implemented

New Constants:

  • URL_SHOW_FLAGS - Bitmask for URL show options (URL_SHOW_REQ_BODY | URL_SHOW_REQ_HEADERS | URL_SHOW_RESP_BODY | URL_SHOW_RESP_HEADERS)

Debug Enhancements:

  • Socket debug tracing - New debug output showing socket bind addresses and families (enabled via rEmitLog("socket", "debug"))

Enhanced Error Handling

  • Improved fiber exhaustion error messages
  • Better diagnostics when fiber pool is depleted
  • Enhanced error reporting in web file operations

Logging Changes

  • CSRF token mismatch messages changed from error to debug level (reduces log noise)
  • MD5 deprecation warnings now only emitted in debug builds

Web Server Authentication APIs

The web server now includes comprehensive authentication configuration through web.json5:

{
    auth: {
        realm: "example.com",          // Authentication realm
        type: "digest"                  // "basic" or "digest"
    }
}

Per-Route Authentication

Routes can specify authentication requirements:

{
    routes: [
        {
            match: "exact",
            path: "/admin/*",
            auth: {
                type: "digest",
                require: "valid-user"
            }
        }
    ]
}

Cache Control Configuration

Routes can specify cache control headers:

{
    routes: [
        {
            match: '/static/',
            handler: 'file',
            cache: {
                maxAge: '1week',
                directives: ['public'],
                extensions: ['css', 'js', 'png', 'jpg', 'svg', 'woff2']
            }
        }
    ]
}

Breaking Changes

Runtime

  • rParseIsoDate() now returns -1 on error (previously returned 0, which was ambiguous with Unix epoch). Update error checking code: 
    // Before:
if (rParseIsoDate(str) == 0) { /* error or epoch */ }

// After:
if (rParseIsoDate(str) == -1) { /* error */ }

URL Command

  • --count renamed to --iterations in the url command-line utility

Platform Support

Tier 1 Platforms:

  • Linux (x86, x64, ARM, ARM64, RISC-V)
  • macOS (Intel, Apple Silicon)
  • Windows 11 (Native with VS2022, WSL)

Tier 2 Platforms:

  • FreeRTOS
  • BSD (NetBSD, OpenBSD, FreeBSD)
  • ESP32 (ESP-IDF)
  • VxWorks

Security Considerations

Authentication Security

This release adds industry-standard HTTP authentication:

  • Basic Authentication: Use only with TLS/HTTPS in production to protect credentials
  • Digest Authentication: More secure challenge-response mechanism suitable for HTTP
  • Password Storage: Passwords are stored as SHA-256 or MD5 hashes
  • TLS Enforcement: Configurable TLS requirement for Basic authentication

Recommendation:

  • Use Digest authentication for HTTP deployments
  • Use Basic authentication only with TLS/HTTPS enabled
  • Store password files outside the web document root
  • Use strong passwords and the SHA-256 algorithm

Security Hardening (Fuzzing)

This release includes important security fixes discovered through comprehensive fuzzing:

  • JSON parser boundary condition fixes
  • HTTP method parsing null pointer protection
  • Resource leak prevention (file descriptors, memory)
  • Enhanced input validation in web server

Recommendation: Users running web services exposed to untrusted input should upgrade promptly to benefit from these security improvements.

Package Updates

Updated embedded packages (paks):

  • r/ v1.2.0 - Runtime library with socket accept optimization (R_WAIT_MAIN_FIBER), buffer management (rGrowBufSize), runtime socket limits (rGetSocketLimit, rSetSocketLimit), fiber control limits, new time APIs (rGetHttpDate, rMakeTime, rParseHttpDate, rGetElapsedTime), fiber and socket APIs (rStartFiberBlock, rSetSocketLinger), string functions (sncat), cross-platform command execution (rRun, rMakeArgs), ISO-8601 parsing fixes, improved fiber exhaustion handling, IPv4-first socket connection strategy, improved connection verification with getpeername(), macOS dual-stack fixes, Windows pollFds clearing fix, rSetFiberStack fix, dynamic poll table growth for Windows/WSAPoll
  • json/ - JSON parser with fuzzing-discovered security fixes and boundary condition improvements
  • crypt/ - Cryptographic library with HMAC support for Digest authentication, compiler warning fixes
  • url/ v1.3.0 - URL client with HTTP Basic/Digest authentication (urlSetAuth), automatic challenge-response handling, nonce reuse optimization, security hardening (quote escaping, parameter limits), --user command-line option, linger control (urlSetLinger), improved error handling for connection failures
  • web/ v1.3.0 - Web server with event-driven non-blocking I/O (10x scalability), optimized socket accept on main fiber, zero-copy body reading (webReadDirect), static response strings (webWriteResponseString), client-side cache control, pre-compressed content serving (.gz, .br), Basic/Digest authentication with HMAC-SHA256 nonces and replay protection, dual-stack networking fix, upload forms fix, keep-alive timeout fix, webSendFile hang fix, resource leak fixes, PUT file size limit checking, improved upload boundary parsing

Upgrade Instructions

  1. Backup: Backup your current configuration and state directory
  2. Review Security Fixes: Note the fuzzing-discovered security fixes and resource management improvements
  3. Review Authentication: If adding authentication, review the authentication configuration options
  4. Update Source: Update to v3.0.0 source code
  5. Rebuild: Rebuild with your application configuration
  6. Configure Authentication (if needed):
    • Create password file using the password command
    • Update web.json5 with authentication configuration
    • Test authentication with your routes
  7. Test: Verify build and run comprehensive tests, especially:
    • Web authentication flows (if enabled)
    • Dual-stack IPv4/IPv6 networking
    • File upload operations
    • Web file serving operations (verify webSendFile fix)
    • Long-running applications (to verify leak fixes)
    • Fiber-intensive workloads (to verify fiber exhaustion handling)
    • Windows applications (to verify pollFds and socket fixes)

Breaking Changes: See the Breaking Changes section above for:

  • rParseIsoDate() return value change (now returns -1 on error)
  • URL command --count renamed to --iterations

Known Issues

None identified in this release.

Download

Download from the Builder Site

Documentation

Full documentation available at: https://www.embedthis.com/doc/

Support

For support, please visit:

Note: This is a major release adding HTTP Basic and Digest authentication, client-side cache control, event-driven non-blocking I/O for 10x connection scalability, socket accept optimizations, zero-copy body reading, and pre-compressed content serving. It includes important security fixes discovered through fuzzing, comprehensive security hardening for digest authentication, and Windows platform stability improvements. Upgrade recommended for users requiring these capabilities and highly recommended for deployments exposed to untrusted input.

Assets 2
Loading