- Alpine docker image
- Override the http port and host correctly
- Exit immediately if HTTP server port is occupied
- When using debug, log exact error to why a plugin could not be loaded
- New protobuf protocol support (under the hood)
- Bulk actions instead of individual subscribes (under the hood)
- Official Plugin Support
- Monitoring Support
- Clustering Support (with small caveats)
- Listening Discovery Simplification
- V2 storage API
- V2 cache API
- Notify API
- Lazy data parsing
- Improved deepstream lifecycle
- Upgraded development tools
- New deepstream.io website
- All V3 SDKs no longer compatible due to protobuf binary protocol
You can see the upgrade guide for backwards compatibility here
You can see the in depth side explanation of the changes here
- a new standardised logging API with
debug,info,warnanderrormethods - the presence feature can now be used on a per user basis. The online status of individual users can be queried for as well as subscribed to. Check out the tutorial on our website here
perMessageDeflateoption can now be passed to uws, courtesy of @daviderenger #786- various fixes and performance improvements to the subscription registry #780, courtesy of @ronag.
- allow updating and writing to Lists via the HTTP API #788 courtesy of @rbarroetavena
- no data when sending HTTP requests is now considered undefined, rather than null #798.
- internal refactor to pull e2e client operations into framework and abstract from Cucumber steps.
- Added a
restartoption to deepstream service CLI - deepstream will now write to a pid file at
/var/run/deepstream/deepstream.pidwhile running as a service - Authentication and permission plugins can now be configured via config and will be resolved as normal plugins. Either a path or name will need to be provided at the top level, and any options specified will also be passed in.
Enabling clients to create, read, update and delete records, emit events, request RPCS and read presence using a JSON bulk request/response format via HTTP.
- The HTTP API is enabled by default on PORT 8080 and can be configured in the
connectionEndpoints -> http section of deepstream's
config.yml - To disable the HTTP API set the above config to null
The above HTTP API makes deepstream.io compatible with the deepstream PHP client
The deepstream 3.0 release lays the groundwork for multiple combinable endpoints/protocols, e.g. GraphQL or Binary to be used together. It also introduces a new endpoint type enabling developers to write their own. Please note - at the moment it is not possible to run multiple subscription based endpoints (e.g. websocket) simultaneously.
To address the scalability issues associated with the message connector interface's coarse topics deepstream will move to a build-in, high performance p2p/small world network based clustering approach, available as an enterprise plugin. The current message connector support is discontinued.
- Moved end-to-end tests into this repository from
deepstream.io-client-js. - Replaced
javascript-state-machinedependency with custom state machine.
- Improved handling of invalid record names.
-
Added new CLI command, including:
-
deepstream daemon This command forks deepstream and monitors it for crashes, allowing it to restart automatically to avoid downtime
-
deepstream service add This command allows you to create an init.d or systemd script automatically and add it to your system.
sudo deepstream service --help Usage: service [options] [add|remove|start|stop|status] Add, remove, start or stop deepstream as a service to your operating system
-
-
Added brew cask support
You can now install easily install deepstream on your mac using brew cask install deepstream driven by config files within /user/local/etc/deepstream/conf
- Fix issue where certain invalid paths would return 'Invalid Type' on the server.
- Fix issue in request/response where selecting a remote server as not done uniformly.
- Fix issue where using both
.0.and[0]within a json path resulted in inserting into an array. However, when using other SDKs such as Java they would be treated as an Object key or array index. - Fix issue where nested array access/manipulation didn't work via json paths.
Due to the nature of this fix, it may result in compatability issues with applications that used json paths incorrectly ( using .0. intead of [0] ). Please ensure you change those before upgrading.
- Fix for issue #703 where record deletions were not being propogated correctly within a cluster.
- Fixes config-loading issue present in the binary release of 2.3.5.
- Hardcode v3.0.0-rc1 dependency on javascript-state-machine, as v3.0.1 causes deepstream.io startup to fail
- Hot path needs to store values in the correct format
- Binary config files have the correct latest structure
- Fix an issue where heavy concurrent writes on the same record fail
- Fixing a connection data regression where it wasn't formatted the same as pre 2.3.0
- Correctly merging config options from
config.ymlfile with the default options
- Adds "storageHotPathPatterns" config option.
- Adds support for
setData(): upsert-style record updates without requiring that a client is subscribed to the record. This uses a new 'CU' (Create and Update) message. ThesetData()API is up to 10x faster than subscribing, setting, then discarding a record. - Support for connection endpoint plugins.
- Significant performance improvements stemming from message batching.
- Moved uws into a connection endpoint plugin.
- Explicit state-machine that initializes and closes dependencies in a well-defined order.
- Adds support for custom authentication and permissioning plugins.
- Adds support for generic plugins.
- Added check to ensure subscriptions are not removed from distributed state registry prematurely.
- Unsolicited RPCs now get a
INVALID_RPC_CORRELATION_IDmessage
- RPC lifecycles have been improved and don't throw exceptions on response after a timeout by ronag
- Correct options now being passed into the
RuleCache, courtesy of ralphtheninja
- Records now can be set with a version -1, which ignores version conflicts by datasage
- Delete events are now propagated in the correct order by datasage
- You can now request the HEAD of a record to retrieve just its version number by datasage
- Providers for listeners are now by default selected randomly instead of in order of subscription
- Ensure record updates are not scalar values before trying to save them in cache by datasage
- Long lived RPC requests now use dynamic lookups for providers rather than building the Set upfront by [ronag]{ronag}
- Huge optimization to subscription registry, where the time for registering a subscriber has been reduced from n^2 to O(n log n)
- Deleting grunt since everything is script based
- Due to uws releases being pulled from NPM, we're now using uws from a git repo
- Created a separate repo uws-dependency with binaries.
- Due to problems with build resulting from uws unpublishing, these two npm packages have been unpublished (noop)
- Unsolicited message in Listening when all clients unsubscribe [#531]
- Handle Non text based websocket frame [#538]
- Aligning binary config with node [#488]
- Event subscription data mishandled in Valve [#510]
- Logging after logger is destroyed [#527]
- Deepstream crash on empty users file [#512]
- Logging error object instead of name in connection error [#420]
- maxRuleIterations must be 1 or higher [#498]
- Ignore sender in subscriptionRegistry if messagebus [#473]
- Removing dead config options [#599]
- getAlternativeProvider in RPC Handler deals with more edge cases [#566]
- Update UWS build version to 0.12
- Packages built against node 6.10
- Fixing write error where only initial value is written to storage #517
- Valve cross referencing in both a create and read results in a ack timeout #514
- Record write acknowledgement. Records are now able to be set with an optional callback which will be called with any errors from storing the record in cache/storage #472
- Applying an ESLint rule set to the repo #482
- Stricter valve permissioning language checks #486 by @Iiridayn
- Update uWS version to v0.12.0
- Better handling/parsing of authentication messages #463
- Properly returning handshake data (headers) from SocketWrapper #450
- Fix case where CLIENT_DISCONNECTED is not sent from SocketWrapper #470
- Fixed issue where listen does not recover from server restart #476
- Handling presence events properly. Now when a user logs in, subscribed clients are only notified the first time the user logs in, and the last time they log out #499
- Fixed issue where connectionData was not available in auth requests #450
- Changelog of 2.0.0 mistakenly said that heartbeats were on port 80 instead of 6020
- User presence has been added, enabling querying and subscription to who is online within a cluster
- Introduces the configuration option
broadcastTimeouttoconfig.ymlto allow coalescing of broadcasts. This option can be used to improve broadcast message latency such as events, data-sync and presence For example, the performance of broadcasting 100 events to 1000 subscribers was improved by a factor of 20 - Adds client heartbeats, along with configuration option
heartbeatIntervalinconfig.yml. If a connected client fails to send a heartbeat within this timeout, it will be considered to have disconnected #419 - Adds healthchecks – deepstream now responds to http GET requests to path
/health-checkon port 6020 with code 200. This path can be configured with thehealthCheckPathoption inconfig.yml
- E2E tests refactored
- uWS is now compiled into the deepstream binary, eliminating reliability issues caused by dynamic linking
- Clients prior to v2.0.0 are no longer compatible
- Changed format of RPC request ACK messages to be more consistent with the rest of the specs #408
- We removed support for TCP and engine.io, providing huge performance gains by integrating tightly with native uWS
- Support for webRTC has been removed
- You can no longer set custom data transforms directly on deepstream
- Sending an invalid connection message is not caught by server #401
- Storage connector now logs errors with the correct namepspace @Iiridayn
- RPC now uses distributed state and no longer depends on custom rpc discovery logic
- Deepstream now uses connection challenges by default rather than automatically replying with an ack
- Upgraded to uWS 0.9.0
- Fix wrong validation of valve permissions when
datais used as a property #346
- Outgoing connections now have throttle options that allow you to configure maximum package sizes to find your personal sweet spot between latency and speed
# the time (in milliseconds) to wait for a buffer to fill before sending it out
timeBetweenSendingQueuedPackages: 1
# the amount of messages that should fit into a buffer before sending between the time to fill
maxMessagesPerPacket: 1000- Listening: Listeners have been drastically improved [#211], and now:
- works correctly across a cluster
- can inform the user whenever the last subscription has been removed even if the listener itself is subscribed
- only allows a single listener to provide a record
- has a concept of provided, allowing records on the client side to be aware if the data is being actively updated by a backend component
As part of this story, we now have multiple significant improvements in the server itself, such as:
- a
distributed state registrywhich allows all clusters to keep their state in sync - a
unique state providerallowing cluster wide locks - a
cluster-registrythat provides shares server presence and state across the cluster
Because of these we can now start working on some really cool features such as advanced failover, user presence and others!
-
Auth: File authentication sends server data to client on cleartext passwords #322
-
Auth: HTTP authentication missing logger during when attempting to log any errors occured on http server #320
- CLI: installer for connectors sometimes fail to download (and extract) the archive #305
- Auth: File authentication doesn't contain
serverDataandclientData#304
userA:
password: tsA+yfWGoEk9uEU/GX1JokkzteayLj6YFTwmraQrO7k=75KQ2Mzm
serverData:
role: admin
clientData:
nickname: DaveUsers can now provide a unauthenticatedClientTimeout config option that forces connections to close if they don't authenticate in time.
This helps reduce load on server by terminating idle connections.
null: Disable timeoutnumber: Time in milliseconds before connection is terminated
- Fixed issue regarding last subscription to a deleted record not being cleared up
-
Fix issue when try to pass options to the default logger #288 (update docs as well)
-
Fix issue deleting a record does not unsubscribe it and all other connections, not allowing resubscriptions to occur #293
You can start deepstream via a command line interface. You find it in the bin directory. It provides these subcommands:
startstopstatusinstallinfohashappend a--helpto see the usage.
You can now use a file based configuration instead of setting options via ds.set(key, value).
deepstream is shipped with a conf directory which contains three files:
- config.yml this is the main config file, you can specify most of the deepstream options in that file
- permissions.yml this file can be consumed by the PermissionHandler. It's not used by default, but you can enable it in the config.yml
- users.yml this file can be consumed by the AuthenticationHandler. It's not used by default, but you can enable it in the config.yml
For all config types support these file types: .yml, .json and .js
There are different options what you can pass:
- not passing any arguments ( consistent with 0.x )
- passing
nullwill result in loading the default configuration file in the directory conf/config.yml - passing a string which is a path to a configuration file, supported formats: .yml, .json and .js
- passing an object which defines several options, all other options will be merged from deepstream's default values
You can write your permission into a structured file. This file supports a special syntax, which allows you to do advanced permission checks. This syntax is called Valve.
deepstream now uses uws, a native C++ websocket server
deepstream will not longer stops your process via process.exit(). This happened before when a connector failed to initialise correctly #243 instead it will throw an error now.
Currently the API provides no event or callback to handle this error
other than subscribing to the global uncaughtException event.
process.once('uncaughtException', err => {
// err.code will equal to of these constant values:
// C.EVENT.PLUGIN_INITIALIZATION_TIMEOUT
// or C.EVENT.PLUGIN_INITIALIZATION_ERROR
})Keep in mind that deepstream will be in an unpredictable state and you should consider to create a new instance.
In 0.x you can set a permissionHandler which needs to implement two functions:
isValidUser(connectionData, authData, callback)canPerformAction(username, message, callback)
In deepstream 1.0 the isValidUser and onClientDisconnect methods are no longer part of the permissionHandler and are instead within the new authenticationHandler.
You can reuse the same 0.x permission handler except you will have to set it on both explicitly.
const permissionHandler = new CustomPermissionHandler()
ds.set( 'permissionHandler', permissionHandler )
ds.set( 'authenticationHandler', permissionHandler )All connectors including, the permissionHandler, authenticationHandler and logger all need to implement the plugin interface which means exporting an object that:
- has a constructor
- has an
isReadyproperty which is true once the connector has been initialized. For example in the case a database connector this would only betrueonce the connection has been established. If the connector is synchronous you can set this to true within the constructor. - extends the EventEmitter, and emits a
readyevent once initialized anderroron error.
The color flag can't be set in the root level of the configuration anymore. The default logger will print logs to the StdOut/StdErr in colors. You can use the deepstream.io-logger-winston which can be configured in the config.yml file with several options.
deepstream clients now have a handshake protocol which allows them to be redirected to the most efficient node and expect an initial connection ack before logging in. As such In order to connect a client to deepstream server you need also to have a client with version 1.0 or higher.
More details in the client changelog.