Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#2248 - uhttpd: implement ubus event listener via Server-Sent Event protocol #7133

Open
openwrt-bot opened this issue Apr 21, 2019 · 9 comments
Labels
flyspray packages

Comments

@openwrt-bot
Copy link

@openwrt-bot openwrt-bot commented Apr 21, 2019

nicopace:

Usecase:
As a web developer I would benefit from having the possibility to send events from the server to the browser so the browser can react to it dynamically.

The proposed implementation is the leanest possible.

A SSE ([[https://www.w3.org/TR/eventsource/|Server-Sent Event]]) is a simple protocol on top of basic http polling to send messages from the http server to the browser for the purpose of instant notifications.
It comes implemented in all major browsers, including [[https://streamdata.io/blog/server-sent-events/|mobile]].

If paired with ubus listen or ubus monitor, a seamless integration with the ubus pipeline could be implemented.

We tried to implement it by adding an option in [[https://git.openwrt.org/?p=project/uhttpd.git;a=blob;f=ubus.c;hb=HEAD#l530|this line of code]] but got stuck in our lack of corutine mechanisms and how uhttpd works.

The difference with the call and list options is that this one should be long-lived and progressive, preventing the dreaded polling from the web (also, better us of resources, more elegant implementation, etc).

As an example, a very basic implementation of a SSE server in php:

An example of a very basic implementation of a SSE client in HTML: <script> var source = new EventSource("demo_sse.php"); source.onmessage = function(event) { document.getElementById("result").innerHTML += event.data + "
"; }; </script>
@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 22, 2019

jow-:

Just implement a long living cgi process and adjust timeout and concurrency values accordingly. I do not understand why modifications to uhttpd itself are required.

Please elaborate.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 23, 2019

nicopace:

Hi Jo,
thanks for your response.
As uhttpd is single threaded (assumption here), i felt that was not possible to keep a long living thread subscribing to the ubus bus.

So, would something like this be possible?

#!/usr/bin/env lua

require "ubus"
require "uloop"

uloop.init()

local conn = ubus.connect()
if not conn then
error("Failed to connect to ubus")
end

local sub = {
notify = function( msg, name )
print("data:{name:"", name, "", count:", msg["count"], "\n\n")
end,
}

conn:subscribe( "test", sub )

uloop.run()

with a publisher like this?
https://gitlab.labs.nic.cz/turris/ubus/blob/master/lua/publisher.lua

if so, will document the usecase so others can benefit.

Still, it feels that it could be something that is integrated in the ubus api, as it would complete it by providing the listen and monitor apis that are not available right now.

Thanks!

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 23, 2019

jow-:

The problem I see with listen/monitor APIs is that they can easily use up the entire available connection / request pool in the server. Since there is no builtin timeout by design, a few clients keeping such listen connections open can essentially DoS the entire server.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 23, 2019

nicopace:

Having someone connected to the router shouldn't take up much ram/cpu. no buffers need to be kept (each event that comes has to be spitted to the client right away, so no need to buffer it).
How many connections would make the DoS kill the device? a few tenths would be enough for many implementations? even one would be enough for some uses.

And as ubus listen/monitor api uses uloop to wait for it, it wouldn't take to much space.

Still... if the functionality is available, would be possible to explore this.
In a use of this functionality, if I am worried about doing a DoS, I would put it behind authentication so only a few users could use it.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 23, 2019

jow-:

Well for starters you would need to untangle the max_request, max_connections and timeout variables because right now they apply to both ubus and http (cgi) requests.

Furthermore we need to treat ubus monitor and/or listen requests differently than list or call requests because right now ubus calls are strictly limited to the global timeout set through the -t option.

The SSE protocol looks simple enough so I might consider implementing it but I can't promise any time frame for this at all.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 23, 2019

nicopace:

Because the way SSE works, if the session times out, the browser will resume it.
If you give the messages ids, the browser will tell you which was the last id it received.

so, no timeout/max_connections untanglement needed (at least at the beginning), neither change of global timeout.

an mvp would be pretty simple, and then we can walk together once it starts being used.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 24, 2019

nicopace:

Hi Jo,

I did a small example, put these two files in /www/cgi-bin;

ping:

#!/usr/bin/lua

require "ubus"
require "uloop"

uloop.init()

local conn = ubus.connect()
if not conn then
error("Failed to connect to ubus")
end

ubus_objects = {test = {}}

conn:add(ubus_objects)

conn:notify(ubus_objects.test.__ubusobj, "ping", {})

io.stdout:write("\rContent-Type: application/text\n\n")
io.stdout:write("ping")

pong:

#!/usr/bin/lua

require "ubus"
require "uloop"

io.stdout:write("\rContent-Type: text/eventstream\n\n")
io.stdout:flush()

--uloop.init()

local conn = ubus.connect()
if not conn then
error("Failed to connect to ubus")
end

local my_event = {
ping = function(msg)
io.stdout:write("data:'long pong'\n\n")
io.stdout:flush()
end,
}

conn:listen(my_event)

uloop.run()

when i head to router/cgi-bin/ping, it certainly shows 'ping'.
when i head to router/cgi-bin/pong, it stays waiting, with no output.
even if i go again to /ping while leaving pong open, nothing happens on pong.

i think the uloop loop is not working properly there... any suggestions?

ps: no errors in logread either...

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 25, 2019

nicopace:

Ok, did another test with os.popen and it works.
though it is a hack, it is pretty consistent:

#!/usr/bin/lua

require "ubus"
require "uloop"

io.stdout:write("\rContent-Type: text/eventstream\n\n")
io.stdout:write("data:'ping'\n\n")
io.stdout:flush()

ubuslisten = io.popen("ubus listen")

for line in ubuslisten:lines() do
io.stdout:write("data:'ping'\n\n")
io.stdout:flush()
end

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 25, 2019

nicopace:

So, it worked!

This is what goes on /www/cgi-bin/ubuslisten

#!/usr/bin/lua

io.stdout:write("Content-Type: text/event-stream\n\n")
io.stdout:flush()

ubuslisten = io.popen("ubus listen")

for line in ubuslisten:lines() do
io.stdout:flush()
io.stdout:write("data: ")
io.stdout:write(line)
io.stdout:write("\n\n")
io.stdout:flush()
end

and this on /www/ubuslisten.html

<script> var source = new EventSource("/cgi-bin/ubuslisten"); source.addEventListener('message', function(e) { console.log(e.data); }, false); </script>

two drawbacks of this implementation:

  • you loose messages between reconnections: can be fixed by having a small daemon that has a 5second buffer of what has been happening, and each time a new connection happens, if it has a timestamp for the last message (done with id: parameter on SSE if we want), you can ask all the messages that came since that one
  • the resources usage: this one has a long session, and also uses an extra ubus listen process for each one.

@aparcar aparcar added the packages label Feb 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flyspray packages
Projects
None yet
Development

No branches or pull requests

2 participants