Example for http.server

[lost-RD]

Can you please post one? I'm unsure how to use it and the docs are vague.

[daurnimator]

unsure how to use it and the docs are vague.

That's mildly intentional: I have been modifying the api significantly and regularly.
I don't intend to document it until I've settled on a final API.

If you'd like, I can tell you how to use the current iteration (and/or you can help create the next iteration); but it will only be of short term value.

[lost-RD]

A solid starting point would be appreciated, yes. I'm happy to follow the changes after that.

[torhve]

There's a rudimentariy webserver in my IRC bot here: https://github.com/torhve/ivar2/blob/master/core/webserver.lua

[lost-RD]

@torhve, thanks for the reference. I'm attempting to compile the relevant code from your codebase into just a simple server that returns 404s but I haven't been able to get the server process to stay open yet though so I fear that your implementation may be beyond me.

I'm getting "unable to connect". I assume I've missed some necessary code.

lua-http.zip

[torhve]

My file that I linked to returns a object. I load the file and start the webserver like this:

local webserver = assert(loadfile('core/webserver.lua'))(self)
local cqueue = cqueues.new()                                                 
cqueue:wrap(function()                                                           
  pcall(function()                                                               
    local server = webserver.start(self.config.webserverhost, self.config.webserverport)                                                                         
    server:run(webserver.on_stream, cqueue)
  end)                                                                           
end)                                                                             

Hope that helps

[vavrusa]

I have some working code in here: https://github.com/CZ-NIC/knot-resolver/tree/master/modules/http

However, I had to fork a couple files - namely h2_connection, that creates new cqueue (so new pollfd per each connection) to use server pollfd instead, server to handle disconnects instead of throwing error and proper closing of streams to avoid leaking. I'm not sure what's @daurnimator plan for server, I could make PR for merging back to upstream if that's the way to go, but need to fixup tests first.

[daurnimator]

Sorry I haven't had a lot of time this week.

However, I had to fork a couple files - namely h2_connection, that creates new cqueue (so new pollfd per each connection) to use server pollfd instead, server to handle disconnects instead of throwing error and proper closing of streams to avoid leaking.

Hmm, what were the issues here?

Re: http2 fd leaks, the cause of the leak has been fixed in cqueues upstream (just need a release).

[vavrusa]

I didn't have much time to work in it last weeks either, but I'll submit a PR when I get to it, it'd be more helpful probably. Some of those leaks are indeed fixed by the bugfixes with ephemeron tables in cqueues upstream.

The second thing is that h2_connection still creates pollfd for each connection, this is acceptable for clients but not great for server multiplexing multiple connections, because now it needs 2x number of open fds and extra overhead for creating them and overhead from nesting.

[daurnimator]

The second thing is that h2_connection still creates pollfd for each connection, this is acceptable for clients but not great for server multiplexing multiple connections, because now it needs 2x number of open fds

Compared to currently widely accepted http1 behaviour it should still be a significant reduction in fds.

But I'm curious how you'd propose to do this instead. Every other approach I came up with was racey.

extra overhead for creating them and overhead from nesting.

Yeah I was concerned about this. It's part of the reason I haven't merged #21

---

If anyone would like to chat about the design of the http server API I'm happy to be contacted via IRC/xmpp/skype to discuss.

[vavrusa]

I propose to multiplex all client connections in parent cqueue. cqueue is just a wrapped kqueue/epollfd and while you can nest them, there is not much benefit in doing so (unless you want to i.e. serve only one socket from active socket at a time). I changed the API to either accept passed cqueue or reuse current cqueue.

[daurnimator]

@vavrusa, wouldn't they break self-stepping if used from a thread managed by the same cqueue?

[daurnimator]

I made an example over here: https://gist.github.com/daurnimator/192dc5b210718dd129cfc1e5986df97b
I'm still not happy with the lua-http server API, so I'm not going to commit that example to the example directory.

Sadly I haven't found a way to write a useful server that doesn't use pcall (which means they can't work in an unmodified lua 5.1; though compat-5.3 does give a shim, it has global side-effects)

[lost-RD]

Thanks for this! So are you saying that I will need compat-5.3 to use this or should I instead be using a modified 5.1?

[daurnimator]

Thanks for this! So are you saying that I will need compat-5.3 to use this or should I instead be using a modified 5.1?

I'm saying that in PUC lua 5.1 (luajit doesn't have this limitation) you cannot yield over a pcall. This makes it impossible to catch errors in a function that might yield.
Compat-5.3 can replace pcall with one emulated via coroutines. However loading that portion has global side effects, so I wouldn't want to use it in a library.
This fundamentally guides how the API for a server handing requests should look (if I want to support PUC lua 5.1): I guess the pcall call has to live outside of lua-http, so that people have the option to use compat-5.3.

Unless perhaps I can emulate pcall myself without having far reaching issues or global side effects?..... I might look into this option more.