Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

http server in the standard library #910

Open
andrewrk opened this issue Apr 11, 2018 · 31 comments
Open

http server in the standard library #910

andrewrk opened this issue Apr 11, 2018 · 31 comments

Comments

@andrewrk
Copy link
Member

@andrewrk andrewrk commented Apr 11, 2018

See #2007 for http client. This issue used to be for both.

http server in the standard library will be helpful for writing tests of the http client at the very least. We can re-evaluate what code does or does not live in std lib closer to 1.0.0; right now it's just helpful to have more test coverage for the language.

@ansarizafar
Copy link

@ansarizafar ansarizafar commented Apr 23, 2018

Check this
Lwan is an opensource high-performance & scalable web server https://lwan.ws/

@bheads
Copy link

@bheads bheads commented May 7, 2018

I can understand putting in an http client for the package manager but an http server? Building a secure http server is a big project. And if its in the std lib people will use it and it will have to be supported. But an officially support http server reference project used by zig to host zig would be awesome, but I would avoid including it in the std lib.

I would also avoid including C projects directly in the std lib. One mistake in D was including Curl in the std lib, now they have to deal with Curl versioning and distributing curl libs with the application. Would be better having an official package for these kitchen sink items, then when we want to end support they can be forked off into the community (or left abandoned in git) without it affecting the std lib.

@costincaraivan
Copy link

@costincaraivan costincaraivan commented Jun 7, 2018

Agreed. Include a HTTP client and then everything else should come from community packages. If the community converges on a set of very popular, high quality libraries, then you pull those in and even then, for a language such as Zig you'd probably want them just "outside" of the stdlib, in set of libraries maintained by the language developers yet separate (a sort of recommended and supported libraries, but not necessarily with the same commitment for backwards compatibility as the stdlib, which generally is super strict).

The main reason is that you want to avoid the Python or Java situation where you have a ton of libraries which are either bad or completely obsolete but can't be removed.

@andrewrk
Copy link
Member Author

@andrewrk andrewrk commented Jun 7, 2018

Right now the standard library exists for 2 reasons:

  • To test out the language and make sure language decisions are good
  • To eventually become the standard library we will ship with zig 1.0.0

So an HTTP server / client would certainly help for the first case. We can evaluate whether it belongs in std when we do the stdlib audit before 1.0.0.

@costincaraivan
Copy link

@costincaraivan costincaraivan commented Jun 7, 2018

Ah, ok. But don't forget that as more people start to use libraries, the backlash caused by removing something, even before 1.0, increases. I'd at least mark things clearly as "experimental" or such to clarify a bit, to minimize outcry.

@renatoathaydes
Copy link

@renatoathaydes renatoathaydes commented Jun 7, 2018

I would suggest adding to the standard library a low-level http library based on RFC-7230, which defined simply the HTTP message format, basically, not semantics (e.g. meaning of HTTP verbs, headers etc. which is defined in other RFCs). This is simple and can be the foundation of higher level libraries, including http clients and servers, while still being useful on its own to accomplish the goals mentioned by @andrewrk.

I wrote a low-level HTTP library in Java (which I am thinking of translating to zig for the async support and speed) and it's not a lot of work. Higher level APIs to work with HTTP are hard to get right and most people have different preferences on what they should look like (which probably explains most languages having several different options - which is probably a good thing), hence those are better left to third-party libraries, IMHO.

@ziglang ziglang deleted a comment Jun 7, 2018
@binary132
Copy link

@binary132 binary132 commented Jun 18, 2018

HTTP/2?


Separately,

This may be an unpopular position, but I have noticed that Go suffers from very weak a) websocket, b) terminal, and c) GUI packages. I realize there is a vast amount of work represented in that short sentence, but there was also an enormous amount of community interest in it, and a likewise enormous amount of community fragmentation around different half-baked solutions to these problems.

Rust has a similar fragmented set of community solutions. Everyone wants to write a parser, everyone wants to do their own async i/o, etc.

Perhaps the way to do this is to support a canonical set of packages, sort of a second-tier set of official repos that are still outside of the stdlib, but that will always be the go-to for Zig for that problem. Thinking out loud here.

@Wulfklaue
Copy link

@Wulfklaue Wulfklaue commented Jul 7, 2018

If it helps - here are some languages that have HTTP either in core or standard:

https://crystal-lang.org/

Has default HTTP server build in.

https://dlang.org/

Relies on external Vibe.D for HTTP support.

Rust has a similar fragmented set of community solutions. Everyone wants to write a parser, everyone wants to do their own async i/o, etc.

Its a issue that all Languages share. Any language that has a package manager tends to fragment the community because everybody wants "their" code to be popular or they disagree with the authors of other packages or ...

Look at Go ... Sometimes 20 packages that do the same thing, just in different ways, different supports ( not counting the clones with minimal changes ). Npm ... same.

Languages need to have a tighter controle over there eco system. No Raft Protocol package? Somebody can write one, that follows the language library standard. That becomes a official standard library package.

Somebody want to improve it? Commit to the official version.
Its not maintained anymore... Allow for a new version "2.0" that is backwards compatible.

Keep tight control over the resources. What makes PHP so easy? Not just the language but also the fact that all official extensions are fast, secure and well documented. Its in general the stuff outside the standard library that has issues over time. Hell, you can use 10 year old library code and it will mostly still work.

Unlike languages that rely a lot on external 3th party plugins, where your old code need to use unmaintained code or it does not work or ... and then requires rewrites or whatever issues.

@ansarizafar
Copy link

@ansarizafar ansarizafar commented Aug 8, 2018

Is there any progress on this issue?

@andrewrk
Copy link
Member Author

@andrewrk andrewrk commented Aug 8, 2018

Yep, lots of progress:

  • cancel semantics for coroutines are now sound and thread-safe
  • the stage2 compiler is now a successful proof-of-concept of zig's async/await syntax integrating perfectly into linux's epoll, macos's kqueue, and windows's IOCP.
  • we now have these building blocks:
    • std.event.Channel
    • std.event.Lock
    • std.event.RwLock
    • std.event.Future
    • std.event.Group
  • in implementing the file system watcher, I discovered a much better API for a tcp server, using a Channel for incoming sockets rather than an async fn handler. This solves the 1 catch @panic("TODO") that I had in the status quo tcp server implementation.
  • I'm working on #1294 which is a nearly complete PR for async/await file system API
  • @kristate is working on #1331 which implements some async/await networking API
@ansarizafar
Copy link

@ansarizafar ansarizafar commented Aug 8, 2018

Thanks for the update

@kristate
Copy link
Contributor

@kristate kristate commented Aug 8, 2018

Once #1271 lands, first it will be great to get some HTTP action. We could even be faster to market than the rust guys with coroutine async.

@u007
Copy link

@u007 u007 commented Sep 20, 2018

is context in place yet? i think context is important to allow cancellation or io interruption during call to make http cancellable. go context is a good example. i think it should be part of the std http package as well

@daurnimator
Copy link
Collaborator

@daurnimator daurnimator commented Dec 17, 2018

I'm the author of lua-http

I think the networking side of this issue needs to wait for #1778. However there is a small amount of work that can be done before then including e.g. adding a headers object.

I agree with the above assesment(s) that the standard library should only contain low level interfaces. High level HTTP interfaces are far too subjective and will be a source of churn.

I advocate for following the newer HTTP specs, which e.g. consider the method to simply be a header field with name :method. This manages to really simplify things so that a HTTP connection has many streams which has a number of header or body blocks.

Re: debate about including a http server. I should tell the history of lua-http: I originally wrote some code to play around with the http2 spec while it was being written to shake out bugs. Originally I attempted to just write a client, but I found existing servers were terrible for debugging and didn't have good error messages; considering that http2 clients and servers aren't that different, I wrote in support for both. Once I had http2 client and server, it wasn't a huge leap to add a basic http1 client. However to complete this then required writing a http1 server which is quite the large job. But then that was helpful in writing http1 tests...... So after a lot of effort, lua-http then supported http2 and http1/1.1, both client and server.

@ghost
Copy link

@ghost ghost commented Feb 25, 2019

@andrewrk would it be acceptable to break this into 2 issues?

personally i have more need for a client than a server

that way we can close one without the other if need be

if acceptable we should rename this to server and i can make a new one for client -
or something

@andrewrk
Copy link
Member Author

@andrewrk andrewrk commented Feb 25, 2019

@cup sure thing. I broke http client out into #2007

@andrewrk andrewrk changed the title http server and http client in the standard library http server in the standard library Feb 25, 2019
@daurnimator
Copy link
Collaborator

@daurnimator daurnimator commented Feb 26, 2019

@andrewrk as I mentioned above, I don't think the implementation for http client and http server should really be separated: you need one to sufficiently test the other. I would keep them as one issue.

@ghost
Copy link

@ghost ghost commented Feb 26, 2019

you need one to sufficiently test the other

@daurnimator no, no you dont. if a client is implemented first, it could be tested against anything here pretty much

https://rosettacode.org/wiki/Hello_world/Web_server

why would it need to be tested again a Zig server? If Zig server is implemented first, its even easier:

curl something-being-served-by-zig

finally, i dont think either of the issue should be venue for debate - andrew is very busy and unless you are offering to do some of the heavy lifting (IE Zig code writing) i dont think its fair to people who might want the client or server without the other to have to wait for both because you think its a good idea.

@renatoathaydes
Copy link

@renatoathaydes renatoathaydes commented Feb 27, 2019

@cup you're right, but I believe you guys might consider having a third issue in this case, which blocks both client and server: create a http message parser.
This is like 90% of a http server and client. HTTP messages can be requests or responses, but the two are nearly identical. The differences are the first line (the spec calls it start-line) and the logic to determine whether a body is expected (e.g. for a response, some status codes imply no body, like 204).

@andrewrk

This comment was marked as off-topic.

@rishavs
Copy link

@rishavs rishavs commented May 22, 2019

@andrewrk whats the plan on http/2?
Or even Quic? will this be part of stdlib?
what of websockets?

Also, I am not sure if these are related but is there a plan on unix sockets?

@ghost
Copy link

@ghost ghost commented May 22, 2019

@rishavs i think thats getting way ahead of the situation.

thats not even supported by certain versions of cURL

https://daniel.haxx.se/blog/2019/05/11/tiny-curl/

plus we dont even have a client yet - so i would assume that comes before a server

@rishavs
Copy link

@rishavs rishavs commented May 22, 2019

@cup ^_^ sorry about that.
But I just wanted to get a gist of the plan on these. Are we looking to have support for them in the stdlib?
or is the idea just to do a very basic http server and let the other protocols like http/2 be done in external libs?

@ghost
Copy link

@ghost ghost commented May 22, 2019

I would guess if it happens at all it would be at least a year

@rishavs
Copy link

@rishavs rishavs commented May 22, 2019

I am less worried about the timeline but more of "should it even be done?"

I feel http/2 and websockets are very important in today's world and any modern http server implementation that we start with should at least consider them as possible features.

@ghost
Copy link

@ghost ghost commented May 22, 2019

@rishavs yeah it get that - but you have to understand - this entire project has about 4 active developers - thats not a lot when the project is a programming language.

Its like youre asking about space travel when people are still riding horses. In my opinion its just not an appropriate time to be asking these type questions, unless you yourself are willing to help with the code and have the technical knowledge to do so.

@andrewrk andrewrk modified the milestones: 0.5.0, 0.6.0 Aug 27, 2019
@andrewrk andrewrk added this to To do in Package Manager Oct 17, 2019
@andrewrk andrewrk modified the milestones: 0.6.0, 0.7.0 Feb 10, 2020
@adontz
Copy link

@adontz adontz commented Feb 22, 2020

And if its in the std lib people will use it and it will have to be supported.

Yes they will. As a workaround, built is HTTP server can be explicitly marked as for tests-only (like put it in std.internal.not_public.for_tests_only.you_have_being_warned), and also make it so bad, nobody will ever be able to use it in production. For instance it may explicitly not support hosting of static files or returning replies larger than one kilobyte. It must be terrible for a greater good.

Also, as a side note, under Windows it's more convenient to use HTTP Server API, and not socket's base server. HTTP Server API already supports TLS, HTTP/2, WebSockets. It includes HTTP parser, allows multiple applications to listen on the same port, but different paths, integrates with NTLM and Kerberos authentication. Also it is implemented in kernel and is fast as hell. Any comparable custom solution will be slower, less feature rich and orders of magnitude more complex.

@leaanthony
Copy link

@leaanthony leaanthony commented Jun 20, 2020

Piping up here as a Zig lurker and Go enthusiast: There are 2 really great thing about having a standard http server in the standard library:

  1. It's your go-to for creating web based applications - be that servers, microservices, APIs, etc... In just a few lines you can get something up and running, in a way other people will understand.
  2. You have default structures to build upon. This is powerful as this means that as supplementary libraries grow around the standard server, they can inter-operate. Good examples here are middleware and routers. The vast majority of supplementary libraries in Go work on top of the standard interface of (request, response) error. This means you are much less likely to have to rework code should you change 'framework' and I believe this is a big win for everyone.

I'm really looking forward to seeing where Zig goes. It's got great potential.

@iamgreaser
Copy link

@iamgreaser iamgreaser commented Jun 20, 2020

TL;DR do an HTTP/1.1 server now, do TLS later, and anything outside of that can be deferred to libraries, although I'm not sure how one would handle websockets or WebRTC. Don't bother with HTTP/2 or HTTP/3, because you have dedicated web servers for that.

In The Real World™, you tend to proxy your middleware through a dedicated HTTP server such as nginx, and the more serious setups have cache servers and load balancers.

The server will be fine as an HTTP/1.1 server. If someone wants an HTTP/2 or HTTP/3 server, then that's the responsibility of e.g. nginx... but if someone insists on having that handled in something written in Zig then that can happen outside of the standard library.

TLS for the server is not strictly necessary but considering that it's likely that an HTTP client would need TLS support, and the bulk of the difficulties of certificate checking and whatnot are typically handled by the client rather than the server, I suspect that the server would end up getting TLS support because "we might as well have it and we've done the hard part anyway".

As for websockets and WebRTC... that's something I lack expertise in, so I'd suggest asking someone how it all works.

Also, as a side note, under Windows it's more convenient to use HTTP Server API, and not socket's base server. HTTP Server API already supports TLS, HTTP/2, WebSockets.

The problem with that is it's a Windows-only thing, and in general if someone's going to be running an HTTP server they're going to be running it on Linux. I don't see that ever being used in Zig purely because it's so Windows-specific.

Whereas if you target the Linux case, then Windows will gain whatever support Linux has barring any Win32 API jankiness that hasn't been ironed out at the time.

@watzon
Copy link
Contributor

@watzon watzon commented Jun 20, 2020

Especially in the beginning here I completely agree @iamgreaser. TLS support should eventually be added to the client and server, but it's not needed for either to begin with. The biggest need in the immediate future is for a client that Just Works so it can be used for the package manager, and that's unlikely to need TLS support.

Andrew also mentioned during his last stream that, while things like a HTTP server will be integrated into the standard library to begin with, anything that doesn't actually serve a purpose as a part of the compiler/package manager/etc will probably be relegated to its own community managed repository come v1.0.

So for anyone worried about "a HTTP server doesn't belong in the standard library", don't worry, it's not a forever thing.

@LaPingvino
Copy link

@LaPingvino LaPingvino commented Dec 10, 2020

I think a good first target is having a webserver that is fully acceptable behind a reverse proxy and/or having a fastcgi library. There are very good webservers out there that most people use anyway and you would use those with either of fastcgi or reverse proxy (e.g. I would use Caddy). If someone decided to write something like a Caddy/Apache/Nginx replacement, that would warrant something more capable, but that is not like anything 90% of web projects that would use the webserver would need... having a good http client is way more important as a first target.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet