Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebSocket support #1449

Closed
usq opened this issue Apr 10, 2017 · 33 comments
Assignees
Labels
Milestone

Comments

@usq
Copy link

@usq usq commented Apr 10, 2017

Have you ever thought about adding WebSocket support?

I am looking for a way to push XML messages to clients to enable multi-client xml-applications and WebSockets would be an ideal fit.
I added a %rest:WEBSOCKET annotation to test the concept and got it to work, so I was wondering if something like this was ever on the roadmap for BaseX/RESTXQ?

Thanks!
Mike

@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Apr 11, 2017

Dear Mike, I remember there has been some discussion on web sockets on our mailing list (see e.g. https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg08836.html). Glad to hear it’s working for you. Fel free to share your experiences or your code with us.

@dirkk

This comment has been minimized.

Copy link
Contributor

@dirkk dirkk commented Apr 11, 2017

+1000 for WebSocket support and sharing your code. A simple %rest:WEBSOCKET annotation sounds perfect to me.

@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Apr 18, 2017

@usq: Did you get the WebSocket extension working? If yes, is you solution similar to the one from Marco (see my link to our mailing list)?

@usq

This comment has been minimized.

Copy link
Author

@usq usq commented Apr 18, 2017

Hi,
sorry for the late reply, I was swamped with work the last days.

Thanks for the mailing list link!

I just added https://github.com/TooTallNate/Java-WebSocket, added the %rest:WEBSOCKET annotation and send the serialized XML via the WebSocket to all connected clients when creating the RestXqResponse. But this was just for a prototype and is far from production code!

The websocket support is actually part of my master thesis on multi-client XML applications. I need to notify a group of connected clients about changes to the database (e.g. for an XML chat application, or multi-client word processing like Google Docs)

At the moment, I'm trying to figure out an API/annotation-parameter to specify that WebSocket messages should only be sent to either all clients, a subgroup or just one.
An example would be a chat server, which allows users to join specific chat rooms. As soon as a message is posted and stored in the database, only the clients inside the room should receive a WebSocket message containing the new database state.

As soon as I got that, I'll post a clean/non-hacky pull request (adding the WebSocket like mentioned on the mailing list).

@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Apr 18, 2017

Mike, thanks for your efforts. Looking forward to your pull request.

@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Jun 26, 2017

Hi @usq. I am curious. Any news so far?

@usq

This comment has been minimized.

Copy link
Author

@usq usq commented Jun 26, 2017

Hi,

I got everything I needed to work and pushed my code to
https://github.com/usq/basex/tree/Feature/WebSocketSupport

I got the Jetty 8 WebSocket module working, but I'm not really a Java developer and did not manage to get the WebSocket to listen to it's own port without rewriting half the Jetty setup method, so the WebSocket is listening on the same port as Jetty but on the staticoption WEBSOCKET path which defaults to "/ws".

When the rest:WEBSOCKET annotation is set, the result is sent via WebSocket as well as HTTP response.
However, my changes to the serialize method in RestXqResponse were not one of my proudest moments, this definitely needs work!

The WebSocket will send all responses wrapped in a JSON object of form

{
    "type":"xmlresponse", 
    "xml": "<XML here>"
}

as most WS clients should be using JavaScript and most WS clients require a JSON message container.

WebSocket subscriptions

When a clients sends the message

{
    "type":"subscribe", 
    "data": [<subscriptionkey>*]
}

via the WebSocket, the client will receive messages sent with one of the specified subscription keys (see major hack 1.).
The message

{
    "type":"unsubscribe", 
    "data": [<subscriptionkey>*]
}

unsubscribes the client.

There are two major hacks:

  1. The XQuery method can choose to send the result only to certain subscribed WebSockets.
    This is done by specifying the attribute websocketchannel="<subscriptionkey>" on the root element of the XML response. This is a horrible hack, but I didn't find any other "non-invasive" and user-friendly way to pass information from the XQuery function to the WebSocketFactory :(
    If the attribute is not present, the message is sent to all connected WebSockets.

  2. The WebSocket connections are created by the WebSocketFactory servlet, which is a Singleton.
    I have to register the servlet at the WebAppContext, which means Jetty will instantiate the Servlet when needed.
    When serialising the XML response, I needed a way to pass the response to the WebSocketFactory, so I got weak and created a Singleton.

I implemented everything with as few changes as possible, but due to the hacks I'm no longer sure if this was a good idea ;)

I can create a pull request if you are interested but didn't want to surprise you without giving fair warning :)

Regards,
Mike

@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Jun 26, 2017

Great feedback, Mike! I’ll check out your branch pretty soon.

@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Jun 27, 2017

I had a first (and superficial) look at your code enhancements, and I think it is definitely a great start to gather experiences with RESTXQ and WebSockets. I was positively surprised to see that (eventually, I guess, now that’s it’s working) your solution is so slick.

A general thought (which I guess goes beyond the scope of your master thesis): Maybe we would even have more freedom if we do not strictly bind server replies to client requests. Instead, we could introduce an additional XQuery module for sending WebSocket messages. This could also be a solution for the first hack you described (the function for sending messages could have a channel argument). Some background:

We also stumbled across use cases in the past in which WebSocket support would have been helpful. One common use case is that client requests are too time consuming in order to be evaluated just-in-time. In this case, we would possibly like to notify the client later, i. e., as soon as the result is available. This could be realized by the relatively new Jobs Module, which will ensure that the client receives a request before the actual work is done, and the yet-to-be-realized WebSocket Module. A virtual example:

module namespace code = 'code';

import module namespace websocket = "http://basex.org/modules/websocket";

declare %rest:path('expensive') function code:prepare() {
  jobs:eval("import module namespace code='code'; code:run()"),
  'your result will be generated; you’ll be notified.'
};

declare function code:run() {
  (: do the thing, return the result :)
  websocket:send(<expensive-result/>)
};

The example has been simplified a lot. For example, we would need to clarify if the result is really to be sent to all clients, or just the one that asked for it.

The WebSocket could have the following functions (and more):

(: Send result to all channels :)
websocket:send($result as item()*) as empty-sequence()

(: Send result to specified channels :)
websocket:send($result as item()*, $channels as xs:string()*) as empty-sequence()

(: Subscribe to a channel; requires an additional RESTXQ entry point that calls this function :)
websocket:subscribe($channel as xs:string()) as empty-sequence()

(: Unsubscribe :)
websocket:unsubscribe($channel as xs:string()) as empty-sequence()

Obviously, this approach opens up new questions. One question I immediately get in mind is: how can output options and response headers be specified? Maybe an additional argument could be passed on the websocket:send function:

websocket:send('the result', web:response-header(), 'channel-x')

Another question is: (When) does it make sense to repeatedly subscribe and unsubscribe channels?

And the next question: Is the module approach reasonable at all? ;) As we already have the Jetty 8 WebSocketServlet class, for example, it may not necessarily be the best idea to assign (un)registering of clients to the client.

Any feedback is welcome!

@usq

This comment has been minimized.

Copy link
Author

@usq usq commented Jun 28, 2017

I totally agree that WebSocket messages should not be tied to client requests!

An XQuery module for sending WebSocket messages would be a great idea and solve the first hack.

If I got you right, the WebSocket module would use Java bindings to call the WebSocket Servlet?

The interface

(: Send result to all channels :)
websocket:send($result as item()*) as empty-sequence()

(: Send result to specified channels :)
websocket:send($result as item()*, $channels as xs:string()*) as empty-sequence()

should cover most use cases in my opinion.

I did not really get the methods:

(: Subscribe to a channel; requires an additional RESTXQ entry point that calls this function :)
websocket:subscribe($channel as xs:string()) as empty-sequence()

(: Unsubscribe :)
websocket:unsubscribe($channel as xs:string()) as empty-sequence()

Clients could use RESTXQ methods to subscribe to channels/topics (similar to the MQTT protocol), but the WebSocket Servlet (which holds open WebSocket connections) does not know which WebSocket connection maps to which client.
I sent the subscription message via the established WebSocket connection so that the server knows which connection belongs to which client (or which connection is interested in which 'channel')

Another possibility would be for the client to send an identifier via the WebSocket and then send a HTTP request to a RESTXQ method to subscribe to this channel.
Something like this:

declare 
%POST 
%rest:path("subscribe/{$websocketId}")
%rest:form-param("channel","{$channel}", "")
function code:subscribe(
$websocketId as xs:string, 
$channel as xs:string
) {
    websocket:subscribe($websocketId, $channel)
};

After this call, the WebSocket Servlet knows that all messages to $channel should be sent via the connection which sent the $websocketId. Still not perfect though...

This approach would pass subscription-control back to the server, which can decide if a client is allowed to subscribe to a channel. I think allowing the client to subscribe to any channel without server supervision is a security risk, as a client could subscribe to everything and receive all messages.

But maybe I misunderstood the intention of your websocket:subscribe method?

Subscribe/Unsubscribe is a valid use case in my opinion.
A (probably a bit fabricated) warehouse example:
A client wants to receive all new open orders, so the client subscribes to the channel newOrders. After that, the clients switches to the 'processed orders' view, which should unsubscribe him from newOrders and subscribe him to processedOrders.

Passing output options to the websocket:send method is a great idea!
What exactly did you mean by "response header" though? As far as I know WebSocket messages do not contain a header (apart from the WebSocket message protocol header, which just contains a few bytes), but maybe I missed something?

@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Jun 28, 2017

Thanks for your feedback. I am glad to see that we seem to agree upon many aspects.

If I got you right, the WebSocket module would use Java bindings to call the WebSocket Servlet?

Some more background:

In BaseX, we have two types of built-in modules: Most of the modules can be used without explicitly importing the URL in the query prolog. There is a second group, listed at the end of the Module_Library documentation, which needs to be imported, because the implementations are contained in the optional basex-api sub project of BaseX (https://github.com/BaseXdb/basex/tree/master/basex-api/src/main/java/org).

If we move the Java implementation of the WebSocket Module to this project (java.org.basex.modules.Websocket.class ), we’ll be able to import it and address the Java functions via import module namespace websocket = "http://basex.org/modules/websocket";. So, as you say, we use Java bindings, but they pretty much look like XQuery function calls (and after we have realized #1413, they may even be XQuery function calls, who call the Java code in a second step). Within the module, we can access the HTTP context of the current servlet instance via queryContext.http (see Session.java#L114 as an example).

Regarding my idea with the (un)subscribe functions: If we implement websocket:subscribe as described above, we have access to the current HTTP context; for example, ((HTTPConnection) queryContext.http).req gives us access to the current javax.servlet.http.HttpServletRequest instance. Maybe we can utilize the available information to to the same what the WebSocketServlet class does.

But it could very well be that we either have too little information in our hands to do this by our own, or that we would need to reimplement things that are currently working out-of-the-box, so I’m not sure if this is the best way to proceed. As you say, one clear advantage would be that we could decide within RESTXQ and XQuery if we want to allow a client to subscribe to a channel or not. Maybe we could find out if the Jetty WebSocket solution provides other ways to subscribe clients (apart from using the WebSocketServlet), and which client information is required to make it work?

Passing output options to the websocket:send method is a great idea!
What exactly did you mean by "response header" though? As far as I know WebSocket messages do not contain a header

Good to know, thanks! So the output will always be sent as UTF8 string, I guess? So we could…

  1. simplify the argument to map with output options (identical to what is used by e.g. fn:serialize or file:write),
  2. only allow a string as argument (because fn:serialize can be called in advance), or
  3. define a fixed output format (based on your JSON example), if we know that the available WS clients will be able to handle it,
@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Jul 28, 2017

One month has passed… And I didn’t have time to pursure this further. Maybe I’ll get it done for BaseX 8.6.6.

@ChristianGruen ChristianGruen added this to the 9.0 milestone Dec 6, 2017
@ChristianGruen ChristianGruen changed the title Question: WebSocket support WebSocket support Dec 6, 2017
@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Feb 2, 2018

And I’m updating the status to »maybe we’ll get this done for BaseX 9.0«:

At least we had some time now to exchange some basic thoughts on possible server-side implementations:

With a BaseX-specific implementation, we could combine WebSockets and RESTXQ semantics.

An early example

declare
  %socket:bind('{$input}', '{$client-id}')
  %rest:path('/ws')
function local:socket($input, $client-id) {
  let $output := json:serialize(
    <json type='object'>
      <type>message</type>
      <text>{ $input }</text>
    </json>
  )
  for $client in socket:clients()
  where $client != $client-id
  return socket:send-text($output, $client)
};

Notes

  • If a client calls a RESTXQ function that is annotated with %socket:bind, (s)he will be registered, and the connection will stay open.
  • The function body itself will be called after successful handshaking, as soon as the client sends a message. The message string will be assigned to the first variable ($input).
  • Each subscribed client has a client id, which will be defined by the BaseX server. It will be assigned to the second variable.
  • In the given example, the client message will be broadcast to all other registered clients.
  • If the socket function creates a result, it will be sent back to the client who sent the message.

The sending of messages is not coupled to a incoming client message. If the following function is called, a good-bye will be sent to all registered clients:

declare function local:close() {
  socket:send-text(json:serialize(
    <json type='object'>
      <type>message</type>
      <text>good-bye!</text>
    </json>
  ))
};

Vice versa, it is possible to process client messages without notifying anyone, and returning a plain OK message:

declare
  %socket:bind('{$input}')
  %rest:path('/ws/{$channel}')
function local:log($input) {
  file:write-text('log.txt', $channel || ': ' || $input),
  'OK',
};

As the WebSocket URL follows the RESTXQ conventions, it may contain variables, regular expressions, etc. You can use any other RESTXQ annotations as well (but be aware that WebSockets require GET).

Annotation

  • Signature: %socket:bind('{$input}', '{$client-id}')
  • Description: Registers a client that calls this function. If the client sends a message after handshaking, it will be bound to $input, and the client id (that has been generated by the server) will be bound to $client-id.

Functions

socket:clients

  • Signature: socket:clients() as xs:string*
  • Description: Returns a list of all client IDs

socket:send-text

  • Signature: socket:send-text($data as xs:string, $client-ids as xs:string*) as empty-sequence()
  • Description: Sends a text message to the specified clients

socket:send-binary

  • Signature: socket:send-binary($data as xs:base64Binary, $client-ids as xs:string*) as empty-sequence()
  • Description: Sends binary data to the specified clients

Open issue (more to come)

With the current setup, it is not possible to map WebSocket connections of a client to other requests of the same client. This is e.g. required if a user triggers the creation of a PDF, and if the result is to be returned to that user via an asynchronous job. Maybe we could store the WebSocket ids in the client’s session data (with the URL path as key):

declare
  %socket:bind
  %rest:path('/ws/pdf')
function local:register-for-pdf-notification() {
};

declare
    %rest:path('/create-pdf')
function local:create-pdf() {
  jobs:eval("socket:send-binary( pdf:create(), '" || Session:get('/ws/pdf') || "' )")
};

In that case, we could get rid of socket:clients() (it would be equivalent to Sessions:ids() ! Session:get(., $url)), or provide some more socket functions as wrappers (e.g. socket:id($url) to retrieve the WebSocket if for the current user and the given URL).

Everyone’s feedback is welcome!

@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Feb 5, 2018

Some more thoughts:

(: Send text or binary data to specific clients :)
socket:send-text($message as xs:string, $channel as xs:string?, $client-id as xs:string*)
socket:send-binary($message as xs:base64Binary, $channel as xs:string?, $client-id as xs:string*)

(: Send text or binary data to all clients :)
socket:broadcast-text($message as xs:string, $channel as xs:string?)
socket:broadcast-binary($message as xs:base64Binary, $channel as xs:string?)

(: (Un)subscribe current client for a channel :)
socket:subscribe($channel as xs:string)
socket:unsubscribe($channel as xs:string)
  • It would completely be up to the RESTXQ application to decide how a user can subscribe for a channel.
  • If we use the basex serialization method, base64 would automatically sent as binary, so maybe we don’t need extra functions
  • The client-id could be identical to the session id (but this might prevent us from having multiple websocket connections per client). To couple it to the function, it could also be an incremental internal ID, prefixed by the URL path of the client request.
  • If we want to support multiple WebSockets per client, a »broadcast« function may not be specific enough.
@dirkk

This comment has been minimized.

Copy link
Contributor

@dirkk dirkk commented Feb 13, 2018

Finally I have a few thoughts about this:

  • Absolutely +1 for getting rid of the binary methods. basexshould be fine I assume and it makes the API much nicer
  • Why the rename to socket? Before, in the draft it was always websocket. Socket is a very generic term (and even BaseX uses different sockets itself), whereas websocket is as clear as it gets. I know you like short names, but I would even prefer ws to socket(and hey, it would be even shorter)
  • Would it be possible to include a convenience annotation similar to @SendTo in Spring (see e.g. https://spring.io/guides/gs/messaging-stomp-websocket/)? So when calling this functions the client is automatically subscribed to this channel and the result of the function is sent to this channel. I think this would be the most simple setup for the (I guess) most commonly used feature for WebSockets: Receiving a result from a long-running job. I am aware I as a user could do this as well, but it seems quite cumbersome to do so. But maybe I am missing something?
@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Feb 14, 2018

+1 for your thoughts, very appreciated!

We will check out some more existing frameworks before we’ll make a binding decision (and, possibly, move WebSockets to BaseX 9.1). The reference to Spring and @SendTo is an interesting one.

@ChristianGruen ChristianGruen modified the milestones: 9.0, 9.1 Feb 16, 2018
@dcore94

This comment has been minimized.

Copy link
Contributor

@dcore94 dcore94 commented Feb 23, 2018

A few comments on the implementation ideas hoping I correctly grasp the spirit of this initiative.

  1. I'd choose Jetty just because it's the stack you're relying upon already for the rest of HTTP stuff.
  2. As to the sentence "With a BaseX-specific implementation, we could combine WebSockets and RESTXQ semantics.": I think this could be achieved even if you use a library, isn't it?
  3. In general the idea to homogenize to the annotation interface used for RestXQ is excellent. I thought of accomodating in the bind clause also the possibility to filter on a specific sub protocol [1]. In, for example, JS based client apis, you can do something like: var ws = new WebSocket('wss://example.com/socket', ['appProtocol', 'appProtocol-v2']); which has the semantic of filtering messages that are transported in accordance to a specific sub protocol. Something like: %socket:subprotocol('appProtocol, appProtocol-v2')?
  1. Finally would it be possible to also implement a ws client library?

Eagerly looking forward to it!

[1] https://tools.ietf.org/html/rfc6455#section-1.9

@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Mar 11, 2018

A little update for all of you (and thanks for the continuous feedback, online and offline):

In analogy with Jetty Websockets and the JSR-356 WebSocket API, we currently tend to introduce 3 new annotations. The URL path will be specified as first argument:

  • ws:connect("/ws"): Will be called after handshaking.
  • ws:message("/ws"): Will be called with each message.
  • ws:close("/ws"): Will be called before closing/finalizing a connection.

This way, it will be very simple to inform other clients if a new client connect or says goodbye. Moreover, the annotations will already be familiar to Java users.

@dirkk: I think I can live with the ws prefix ;) And @SendTo could be an interesting extension once we add the STOMP messaging protocol.

@dcore94:

  1. As to the sentence "With a BaseX-specific implementation, we could combine WebSockets and RESTXQ semantics.": I think this could be achieved even if you use a library, isn't it?

I’m not sure how this could be done: The Jetty WebSocket would be a separate, independent servlet, and we would rather like to directly use XQuery annotations for defining WS entry points. But it would be handy if we could use an existing library for realizing the network traffic (incl. handshakes, frames, and extensions).

  1. I thought of accomodating in the bind clause also the possibility to filter on a specific sub protocol [1]. In, for example, JS based client apis, you can do something like: var ws = new WebSocket('wss://example.com/socket', ['appProtocol', 'appProtocol-v2']);

This reminds me of subscriptions in the STOMP protocol; but maybe it’s something else?

@jfinckh

This comment has been minimized.

Copy link
Member

@jfinckh jfinckh commented Mar 11, 2018

I thought of accomodating in the bind clause also the possibility to filter on a specific sub protocol [1]. In, for example, JS based client apis, you can do something like: var ws = new WebSocket('wss://example.com/socket', ['appProtocol', 'appProtocol-v2']);

This reminds me of subscriptions in the STOMP protocol; but maybe it’s something else?

I think STOMP is such a subprotocol as @dcore94 describes.
The Websocket-Protocol makes no assumptions about the format of a message.
With the Subprotocols there is a Possibility to define a Protocol for the Messages.
The Client-Code var ws = new WebSocket('wss://example.com/socket', ['appProtocol', 'appProtocol-v2']); will notify the server about the protocols it supports and the server can check wether it supports the protocol and use it or do sth. else. (Edit: The common way is that the WebSocket handshake is not completed and the onerror callback will be called). Another Protocol could be e.g. the SOAP-Protocol :)
A List of Subprotocols

@dcore94

This comment has been minimized.

Copy link
Contributor

@dcore94 dcore94 commented Mar 11, 2018

Yes, thank you @jfinckh. Subprotocols are a property of websockets at transportation level. STOMP subscription look to me rather as a semantic feature at presentation level.
For WebSockets subprotocols should rather be seen as a sort of inspection of the quality of service of the server.

@jfinckh

This comment has been minimized.

Copy link
Member

@jfinckh jfinckh commented Mar 11, 2018

@ChristianGruen

And @sendto could be an interesting extension once we add the STOMP messaging protocol.

Just a few thoughts that i will not forget until tomorrow:
If i got this right then STOMP is not neccessary for the @sendTo-Annotation. In the Websocket-Handshake it is possible to enable "Multiplexing Extensions for Websockets", meaning that the WebsocketFrames get an additional ChannelID. After doing this it is possible to have multiple WebsocketChannels on one single TCP-Connection. So what spring does (or what i believe spring does) is that it sets the channel-id in the header if the @sendto annotation is used. SockJS e.g. looks up those headers and call the appropriate callback.

@dcore94 +1 for the subprotocols-hint!
I do not understand why a new client libary is needed for the basex-websockets. What could be missing if the client use existing ones?

@dcore94

This comment has been minimized.

Copy link
Contributor

@dcore94 dcore94 commented Mar 12, 2018

@dcore94 +1 for the subprotocols-hint!
I do not understand why a new client libary is needed for the basex-websockets. What could be missing if the client use existing ones?

I'm not sure I understand this question. If it refers to my sentence: "Finally would it be possible to also implement a ws client library?" then, just for clarification, I meant a way to establish a websocket connection from an XQuery client script to an arbitrary ws server. Sort of what you do with sql:connect or db:connect....

ChristianGruen added a commit that referenced this issue Jul 27, 2018
@jfinckh

This comment has been minimized.

Copy link
Member

@jfinckh jfinckh commented Jul 27, 2018

The first implementation of websockets can be found in the masterbranch now.

Import in XQuery:

import module namespace ws = "http://basex.org/modules/Websocket";

Overview of the new Annotations:

  • ws:connect(path) -> Called when a Websocket connects
  • ws:message(path,message) -> Called when a Message arrives
  • ws:close(path) -> Called when a Websocket closes
  • ws:error:(path,message) -> Called when a Websocket throws an error
  • ws:param(name,variable[,default]) -> For getting additional Parameters

Possible ws:param's:

  • Http-Version -> f.e.: %ws:param("Http-Version", "{$version}")
  • Origin
  • Protocol-Version
  • QueryString
  • IsSecure
  • RequestURI
  • Host
  • Sec-WebSocket-Version
  • offset -> just for binary-Messages
  • len -> just for binary-Messages

Overview of new Functions

  • ws:emit(message) -> Emits a Message to all Connected Clients
  • ws:broadcast(message) -> Broadcasts a Message to all Connected Clients except the Caller
  • ws:send(message,id) -> Sends a Message to the Client with the id id
  • ws:id() -> Returns the Id of the Caller
  • ws:ids() -> Returns the IDs of all connected Clients
  • ws:get(key) -> Returns a WebsocketAttribute of the calling Client
  • ws:get(id,key) -> Returns a WebsocketAttribute of the Client with the id id
  • ws:set(key,value) -> Sets a WebsocketAttribute of the calling Client
  • ws:set(id, key, value) -> Sets a WebsocketAttribute of the Client with the id id
  • ws:delete(key) -> Deletes a WebsocketAttribute of the calling Client
  • ws:delete(id, key) -> Deletes a WebsocketAttribute of the Client with the id id
  • ws:path() -> Returns the Path of the calling Client
  • ws:path(id) -> Returns the Path of the Client with the id id

The Path is always the Path the Client connects to.

A quick-and-dirty-example for the most of the new Annotations/Functions can be found here: https://github.com/jfinckh/chat/tree/master

We would be pleased about feedback and further extensions.

@dcore94

This comment has been minimized.

Copy link
Contributor

@dcore94 dcore94 commented Jul 28, 2018

Great news! Excited to give it a try!

@dcore94

This comment has been minimized.

Copy link
Contributor

@dcore94 dcore94 commented Aug 6, 2018

Ok. Finally I've found couple of hours today to download 9.1 snapshot in order to test websockets support. Unfortunately I've not been able to get to the end of my experimentation.
First of all I would like to go with software I've already installed so no NodeJS or extra tools. Thus I open up a browser to set up a ws client and use the following code to connect:
sock = new WebSocket("ws://localhost:8984/w").
This drives me to the first question: is a pure xquery based websocket client library not planned? It would be very useful for us not only for avoiding switching back and forth from the browser while testing but also because we plan to make different xquery modules interact asynchronously through websockets.

  1. Starting from the example provided I had to remove the import statement:
    import module namespace websockets = "http://basex.org/modules/Websockets";
    because I get the error:
    [XQST0059] Module not found: http://basex.org/modules/websockets. 26.7 ms

  2. So now my serverside code is:

module namespace rw="remoteworker";
import module namespace websocket = "http://basex.org/modules/Websocket";

declare
%ws:connect("/w")
function rw:on-connect(){
"WS connected"
};
when trying to connect from browser I get
REQUEST [GET] http://localhost:8984/w
16:41:01.607 0:0:0:0:0:0:0:1:45026 admin 404 No function found that matches the request. 25.57 ms
so it looks as if RestXQ servlet is responding thus I just added the error handler to see if anything more meaningful is returned.
declare
%ws:error("/w", "{$error}")
function rw:on-error($error){
$error
};
and after trying to reconnect with the usual JS line from the browser console now the error is:
REQUEST [GET] http://localhost:8984/w
16:49:10.417 0:0:0:0:0:0:0:1:49088 admin 400 Stopped at /home/lettere/basex/webapp/remoteworker.xqm, 13/10: [basex:ws] Conflicting path annotations found. 29.31 ms.

I'm rather clueless now and I couldn't get any further. Any hints for a way out?

@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Aug 6, 2018

Hi Marco, thanks for performing some first tests. And sorry for the confusion, the current status quo is still in a state of flux. Maybe we should give you an new update once our Wiki documentation is finalized.

is a pure xquery based websocket client library not planned?

Currently no. However, it might be worth discussing how this XQuery library/module could look like. As WebSockets are persistent, I am wondering how this could work. Do you have some ideas that you would be willing to share?

As the chat example from Johannes demonstrates (even it’s still work in progress), we want to release a light-weight solution that does not require any additional libraries (at least in the very first version). Maybe we can include some usage examples in the DBA as well.

@apb2006

This comment has been minimized.

Copy link

@apb2006 apb2006 commented Aug 15, 2018

I wanted to give this a go. So I...

The result is shown below. Should this work? Tinkering with socket.xqm to add additional handlers did not seem to help.


[main] INFO org.eclipse.jetty.util.log - Logging initialized @383ms to org.eclipse.jetty.util.log.Slf4jLog
BaseX 9.1 beta [HTTP Server]
[main] INFO org.eclipse.jetty.server.Server - jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_171-b11
[main] INFO org.eclipse.jetty.webapp.StandardDescriptorProcessor - NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
[main] INFO org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0
[main] INFO org.eclipse.jetty.server.session - No SessionScavenger set, using defaults
[main] INFO org.eclipse.jetty.server.session - node0 Scavenging every 600000ms
Server was started (port: 1984).
[main] INFO org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.w.WebAppContext@10a035a0{BaseX: The XML Database and XQuery Processor,/,file:///C:/Users/andy/Desktop/basex-late/webapp/,AVAILABLE}{C:/Users/andy/Desktop/basex-late/webapp}
[main] INFO org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@6dde5c8c{HTTP/1.1,[http/1.1]}{0.0.0.0:8984}
[main] INFO org.eclipse.jetty.server.Server - Started @1017ms
HTTP Server was started (port: 8984).
HTTP Stop Server was started (port: 8985).
[qtp1358444045-17] WARN org.basex.http.ws.WebSocket - Unhandled Error (closing connection)
org.eclipse.jetty.websocket.api.WebSocketException: No WebSockets function found that matches the request: %ws:close(path).
        at org.basex.http.ws.WebSocket.findAndProcess(WebSocket.java:161)
        at org.basex.http.ws.WebSocket.onWebSocketClose(WebSocket.java:116)
        at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:98)
        at org.eclipse.jetty.websocket.common.WebSocketSession.notifyClose(WebSocketSession.java:514)
        at org.eclipse.jetty.websocket.common.WebSocketSession.onConnectionStateChange(WebSocketSession.java:557)
        at org.eclipse.jetty.websocket.common.io.IOState.notifyStateListeners(IOState.java:184)
        at org.eclipse.jetty.websocket.common.io.IOState.onCloseRemote(IOState.java:373)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:121)
        at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:476)
        at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:183)
        at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:105)
        at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:70)
        at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:220)
        at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
        at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:245)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:560)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:391)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Unknown Source)
[qtp1358444045-17] WARN org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection -
org.eclipse.jetty.websocket.api.WebSocketException: java.lang.NullPointerException
        at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:228)
        at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:245)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:560)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:391)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException
        at org.basex.http.ws.WebSocket.onWebSocketError(WebSocket.java:108)
        at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onError(JettyListenerEventDriver.java:112)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.unhandled(AbstractEventDriver.java:252)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:187)
        at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:476)
        at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:183)
        at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:105)
        at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:70)
        at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:220)
        at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
        ... 14 more
[qtp1358444045-12] WARN org.eclipse.jetty.websocket.common.WebSocketSession -
java.lang.NullPointerException
        at org.basex.http.ws.WebSocket.onWebSocketError(WebSocket.java:108)
        at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onError(JettyListenerEventDriver.java:112)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingError(AbstractEventDriver.java:96)
        at org.eclipse.jetty.websocket.common.WebSocketSession.incomingError(WebSocketSession.java:460)
        at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingError(AbstractExtension.java:177)
        at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.incomingError(AbstractExtension.java:133)
        at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingError(ExtensionStack.java:214)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.notifyError(AbstractWebSocketConnection.java:438)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.access$000(AbstractWebSocketConnection.java:60)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection$Flusher.onCompleteFailure(AbstractWebSocketConnection.java:73)
        at org.eclipse.jetty.util.IteratingCallback.failed(IteratingCallback.java:401)
        at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:245)
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
        at org.eclipse.jetty.websocket.common.io.FrameFlusher.enqueue(FrameFlusher.java:90)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:495)
        at org.eclipse.jetty.websocket.common.WebSocketSession.close(WebSocketSession.java:223)
        at org.eclipse.jetty.websocket.common.WebSocketSession.close(WebSocketSession.java:190)
        at org.basex.http.ws.WebSocket.onWebSocketError(WebSocket.java:108)
        at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onError(JettyListenerEventDriver.java:112)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingError(AbstractEventDriver.java:96)
        at org.eclipse.jetty.websocket.common.WebSocketSession.incomingError(WebSocketSession.java:460)
        at org.eclipse.jetty.websocket.common.WebSocketSession.notifyError(WebSocketSession.java:521)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.openSession(AbstractEventDriver.java:237)
        at org.eclipse.jetty.websocket.common.WebSocketSession.open(WebSocketSession.java:616)
        at org.eclipse.jetty.websocket.common.WebSocketSession.onOpened(WebSocketSession.java:544)
        at org.eclipse.jetty.io.AbstractConnection.onOpen(AbstractConnection.java:202)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onOpen(AbstractWebSocketConnection.java:446)
        at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:440)
        at org.eclipse.jetty.server.HttpConnection.onCompleted(HttpConnection.java:385)
        at org.eclipse.jetty.server.HttpChannel.onCompleted(HttpChannel.java:702)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:498)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Unknown Source)
@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Aug 15, 2018

Thanks for testing, Andy. We might have announced this too early, sorry – we’ll keep you updated!

@jfinckh

This comment has been minimized.

Copy link
Member

@jfinckh jfinckh commented Aug 17, 2018

Hi Andy,
in the current state, all web socket annotations mapped by jetty must be implemented. That means: ws: connect, ws: message, ws: close and ws: error. If these are not implemented, the first error is thrown here. That will change soon.
I suppose that the next errors describe the same problem here.

@apb2006

This comment has been minimized.

Copy link

@apb2006 apb2006 commented Aug 18, 2018

Hi Johannes,
Thanks for the info. I seem still get the issue when all annotations look defined to me. see
https://gist.github.com/apb2006/d82d0ca7aff6a9b2392f0c59f1f56ff3
I will wait for the next release.

@jfinckh

This comment has been minimized.

Copy link
Member

@jfinckh jfinckh commented Aug 19, 2018

Hi Andy,
thanks for testing again.
I have overlooked that you tried to get the path from the user request.
That is not possible right now, we've decided to use static paths here (i think we will have to specify that fact in the docs more precise). This helps to build different websockets in the same application (with other paths).

That means, if you change your "$path" to "/" your example will work. F.e.:

declare
%ws:error("/","{$message}")
function page:error($path, $message)
{
	let $_:=trace(" SOCKET error >>>",$path)
	return ()
};

We will inform you as soon as there are any updates

@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Aug 31, 2018

We’ve now included a little demo chat application in the latest stable snapshot of BaseX:

chat

You can test the chat as follows:

  1. Download a full distribution of BaseX,
  2. Start the basexhttp server,
  3. Visit http://localhost:8984/, and
  4. follow the instructions (create some database users, work with multiple browser sessions to simulate different users).

The relevant XQuery and JavaScript files are really light-weight. They can be found in the BaseX webapps directory (chat.xqm, chat-ws.xqm, chat.js).

Please note that there are still various issues that we need to look at:

  • Error handling is unsatisfying at the moment: Server-side RESTXQ errors won’t be propagated to the WebSocket client yet. Instead, a socket connection will simply be closed.
  • Timeouts need to be handled properly.
  • Chat-specific: In order to keep the connection alive, periodic pings need to be sent to the server to keep the connection alive.
  • Chat-specific: Users need to be properly unregistered if the browser tab is closed, if another page is visited, or if the log out link is pressed.

The documentation still needs to be revised, but you can find a first draft here:

I’d like to get BaseX 9.1 finalized in October. Cross your fingers…

Have fun! Christian (I’ll be on my vacations now, but @jfinckh may have time to answer some of your questions).

@ChristianGruen

This comment has been minimized.

Copy link
Member

@ChristianGruen ChristianGruen commented Nov 27, 2018

Closing this… Feedback is still welcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.