Addressing HTTP servers over Unix domain sockets

[rkjnsn]

It is often desirable to run various HTTP servers that are only locally connectable. These could be local daemons that expose an HTTP API and/or web GUI, a local dev instance of a web server, et cetera.

For these use cases, using Unix domain sockets provides two major advantages over TCP on localhost:

1. Namespacing. If two users on a system are running the same service, TCP requires them both to pick, configure, and remember different port numbers. With Unix domain sockets, each socket can live in the respective user's runtime directory and be named after the service.
2. Access control. Even if the service is diligent only to bind to localhost, TCP still allows any (non-sandboxed) process or user on the machine to connect. Any access control has to be implemented by the service itself, which often involves implementing (hopefully with sufficient security) its own password authentication mechanism. Unix domain sockets, on the other hand, can take advantage of the access control functionality provided by the filesystem, and thus can easily be restricted to a single user or set of users. In the event that a service wants to allows multiple users to connect and discriminate between them, several operating systems provide a means of querying the UID of the connecting process, again without requiring it's own authentication scheme.

Indeed, due to these advantages, many servers/services already provide options for listening via a Unix domain socket rather a local TCP port. Unfortunately, there is not currently an agreed-upon way to address such a service in a URL. As a result, clients who choose to support it end up creating there own bespoke approach (e.g., a special command-line flag, or a custom URL format), while others choose not to support it so as not to bring their URL parsing out-of-spec (among other potential concerns).

Here are some of the various URL formats I've seen used or suggested:

• Transport only: unix:/path/to/socket.sock. This lacks both the protocol and resource path, so it can only be used for clients that already know they'll be speaking to a specific HTTP API, and is not generally usable.
• HTTP with socket path as the port: http://localhost:[/path/to/socket.sock]/resource. Only allowed when host is localhost. Paths containing ] could either be disallowed or URL encoded.
• Composite scheme with socket path as URL-encoded host: http+unix://%2Fpath%2Fto%2Fsocket.sock/resource. Distinct scheme allows existing http URL parsing to stay the same. URL encoding reduces read- and type-ability.
• Combining ideas from the previous two: http+unix://[/path/to/socket.sock]/resource or just http://[/path/to/socket.sock]/resource. (The latter would require using the leading / of the socket path to disambiguate from an IPv6 address.)

References:
Archived Google+ post suggesting the socket-as-port approach:
https://web.archive.org/web/20190321081447/https://plus.google.com/110699958808389605834/posts/DyoJ6W6ufET
My request for this functionality if Firefox, which sent me here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1688774
Some previous discussion that was linked in the Firefox bug:
https://daniel.haxx.se/blog/2008/04/14/http-over-unix-domain-sockets/
https://bugs.chromium.org/p/chromium/issues/detail?id=451721

[annevk]

It seems you don't need just addressing for this, but some kind of protocol as well. I recommend using https://wicg.io/ to see if there's interest to turn this into something more concrete.

[rkjnsn]

I'm not sure I understand why any additional protocol would be necessary. It's just HTTP over a stream socket. The server accepts connections and speaks HTTP just like it would for a TCP socket. Indeed, I can set up such a server today, and it works fine provided that the client provides a way to specify the socket, e.g., curl --unix-socket /path/to/socket.sock http://localhost/resource.

[avakar]

I don't even understand how this is not a thing yet. Especially now that Windows started supporting AF_UNIX sockets natively, it seems to be the best, cross-platform way to connect web and native apps without consuming a TCP port.

[annevk]

Let me take a step back, what exactly is the ask from the URL Standard here?

[rkjnsn]

The ask is for the URL standard to specify a syntax for referring to a page served via HTTP over a UNIX domain socket. Currently, applications that want to support connecting to an HTTP service have to pick from one of the following three:

1. Provide a bespoke mechanism for specifying the server's socket outside of the URL, such as curl's --unix-socket command-line argument.
2. Accept a custom URL format outside of the URL standard for addressing resources served via HTTP over UNIX domain socket.
3. Forgo the functionality altogether if 1 is impractical and 2 is undesired.

None of these are ideal. Deciding on a standardized URL syntax allows different implementations to implement the functionality in a common, standards-compliant way.

[annevk]

I see, https://wicg.io/ is the place for that. The URL standard defines the generic syntax. If you want to define the syntax for a particular URL scheme as well as behavior, you would do that in something that builds upon the URL standard. E.g., https://fetch.spec.whatwg.org/#data-urls for data: URLs.

[rkjnsn]

Let me rephrase: the specific ask for the URL standard is to provide an allowance in the URL syntax for specifying a UNIX domain socket, either in lieu of the port (e.g., http://localhost:[/path/to/socket.sock]/resource) or in lieu of the hostname (e.g., http://[/path/to/socket.sock]/resource), both of which are currently invalid according to the URL standard.

[annevk]

I recommend using something like unix:/path/to/socket.sock?url=http://localhost/resource. We can't change the URL syntax for each new protocol that comes along.

[cyanogilvie]

It's the same protocol over a stream socket, just a different address (ie. authority part). Ok, so it's a different protocol in the sense of IP, but so are IPPROTO_IP and IPPROTO_IPV6, and the URL standard doesn't treat those as different. The relevant comparison I think are address families for stream sockets, like AF_INET, AF_INET6 and AF_UNIX. Once the stream socket has been established (as specified by the authority part of the URL), HTTP software shouldn't care or even know how the stream is transported.

Most invented, non-standard approaches for HTTP-over-unix-sockets seem to gravitate to something like a different scheme (since the authority part can't really be disambiguated from a hostname if relative socket paths are allowed from what I can see), like http+unix or https+unix, and then percent-encoding the socket into the authority part, and then everything works naturally from there from what I can see.

I've also seen (and used) enclosing the socket path in [] in the authority part and keeping the scheme as http or https, but I think that namespace clashes with IPv6 style numeric addresses like [::1]:80. RFC 3986 (in section 3.2.2) kind of leaves space for this by anticipating future formats within the [], and providing a version prefix to disambiguate them. Overall I like this approach the best (it extends into the error space so it doesn't change the interpretation of any valid existing URL, lives in an extension space envisioned by the standard, minimally extends just the appropriate part of the standard (authority part), keeps the schemes http and https to mean "this is a resource we talk to this authority using the http(s) protocol for", and so preserves compatibility for software that uses the scheme to know what protocol to speak with the authority over the socket.

[annevk]

Changing the syntax of URLs is not really something we're willing to do. That has a substantive cost on the overall ecosystem. The benefits would have to be tremendous.

[michael-o]

Syntax in mod_proxy:

In 2.4.7 and later, support for using a Unix Domain Socket is available by using a target which prepends unix:/path/lis.sock|. For example, to proxy HTTP and target the UDS at /home/www.socket, you would use unix:/home/www.socket|http://localhost/whatever/.

[karwa]

The strongest argument I can think of for this is: http(s) URLs have special parsing quirks which don't apply if the scheme is http+unix. So for a perfect 1:1 behaviour match, UDSs would need to use an actual http URL, not a custom scheme (similar to IP addresses).

That said, I'm also not a fan of adding yet another kind of host (file paths). My preference would be to use a combination of:

• Fake hostname (localhost, example, test and invalid are all reserved and will never be allowed as a TLD, so something like uds.localhost should work), and
• Socket address in the fragment (HTTP clients should strip it before sending the request anyway)

http://uds.localhost/some/path?some=query#/path/to/socket.sock

This is a perfectly valid HTTP URL, and should be capable of representing any HTTP request target.

Alternatively, you could try to get uds or socket as reserved TLDs, but I'm not sure how you'd go about doing that.

(Note: this would also mean that all UDS URLs have the same origin, although that could be remedied by adding a discriminator to the fake hostname to make your own zones of trust, e.g. 123.uds.localhost)

[rkjnsn]

I'm not sure using the fragment is really tenable for these use cases (and local web dev, especially). Many web applications use the fragment for their own purposes in JavaScript, whereas the host (at least it my experience) tends to be handled more opaquely.

What would be the main drawback for allowing additional characters within [] for the host portion of an HTTP URL?

[karwa]

Ah yes, you're right, it wouldn't work for local web development. I was thinking more about generic HTTP servers.

The main drawbacks IMO are:

• Complexity. This standard has a hard-enough time trying to document existing browser behaviour without inventing new things. Then again, there is a reasonable counter-argument that URLs shouldn't have to stay frozen while other aspects of technology and the web evolve to meet new use-cases. There is a counter-counter argument that URLs are in a particularly sorry state compared to most other web technologies. Perhaps this is something for the future, once all browsers conform to this standard and things have stabilised a bit?

• Possible loss of validation for IPv6 addresses. Unless we want to get in to the business of validating local system paths (and I'm quite sure nobody is thrilled by that idea) we would basically have to accept any non-empty string within [] in the host portion of HTTP URLs. How do we know http://[::::foo]/some/path doesn't refer to a valid path on some system somewhere?

[cyanogilvie]

Yes, I think the place for the UDS socket is in the authority portion - that's the bit that has the responsibility for describing the endpoint of the stream socket to talk to for this resource. Putting it elsewhere feels like an abuse and likely to cause unforeseen problems (HTTP client software will certainly have the host portion of the URL available in the portion of the code that establishes the stream socket, but may not have the fragment).

I think the namespace collision with IPv6 literals and syntax validation for UDS paths can be solved by:

• Reusing the syntax for the path portion of the URI: "/" is a separator, path elements must be percent encoded.
• Socket paths must be absolute (start with "/" or "~"). This distinguishes them from IPv6 literals, and should be the case anyway (what would a relative path be relative to? No similar relative resolution for hostnames exists in the standard).
• Possibly using a version prefix as envisioned by RFC 3986, putting it within the syntax anticipated in that standard, something like: http://[v1.uds:/tmp/mysock]/foo/bar.

It's up to the host to decode and translate the path into whatever native scheme that OS uses (just as it is for the path portion of the URI).

For me the motivation for supporting HTTP over UDS goes way beyond web browsers (and I would see that as a minor use case for this) - for better or worse HTTP has become a lingua franca protocol for anything that wants to communicate on the Internet (consider websockets for some of the forces that drive this), and that is increasingly machine to machine. For example: we run an online marketplace that serves about 10 million requests a day over HTTP (excluding static resources offloaded to a CDN), but each of those involve several HTTP interactions with other services to construct the response: Elasticsearch queries, S3 to fetch image sources that are resized, etc, a whole host of REST services for shipping estimates, geocoding, ratings and reviews, federated authentication providers etc. So, by volume, the overwhelming majority of HTTP requests our webservers are party to are between them and other servers, and aren't transporting web pages.

As the trend toward microservices and containerization continues this will only increase, and it's particularly there that I see HTTP-over-UDS being useful:

• Communication over UDS is materially faster and lower latency than over the loopback interface because a lot of the complexities in the network stack can be skipped - packet filtering and transformation, TCP, etc. The loopback interface doesn't have network latency but it still has all these other things. Local sockets (UDS) are more or less just buffers managed by the kernel. This starts to really matter to page response times when generating the page involves many interactions with microservices.
• The namespace for sockets is hierarchical for UDS rather than flat for ports on localhost, so there is a natural way to scope the namespace for each microservice, and which is self-describing. Compare http://localhost:1234/ with http://[/sockets/session/addrs]/ for the address of a microservice providing the address management service for the current session user.

The other trend is for UIs to be implemented in HTML rather than some OS-native widget set (Android, iOS, GTK, QT, MacOS native controls, Windows native controls, etc), even when the application is entirely local on the user's device. There are very good reasons for this:

• HTML+Javascript is portable, greatly reducing the cost to develop the application if it has to run across platforms.
• HTML+Javascript is much richer and more capable than those native widget sets in the types of UIs they can implement.
• Essentially every developer these days already knows HTML and Javascript.
• Gone are the days when users expect native OS controls. These days they expect web application style interfaces, since that's the majority of what they're exposed to (gmail, various cloud based office applications, twitter, etc.)

In this use case the hierarchical namespace issue is important and addresses a major downside to this pattern - choosing a port from the flat, system-wide shared namespace (ok, so the listening socket can specify 0 and have the OS pick a random unused port on some systems, but that's a bit ugly). Much nicer to use ~/.sockets/<app>/<pid>, and more discoverable. Another reason to use UDS in this case is that the user for the client side of the socket can be obtained from the OS in a way that only trusts the OS, solving the other issue with this pattern - knowing which user we're interacting with. If these issues were solved by HTTP-over-UDS, do you think something like Prusaslicer would use that (HTML, Javascript, webGL) rather than wxWidgets for its UI portability requirements? That would make porting to mobile devices like tablets much easier too.

Finally, consider things like headless Chrome in an automated CI/CD pipeline - the software managing the tests being run on the deployment candidate version could start a number of headless chrome instances and run tests in parallel, easily addressing the websocket each provides with a UDS path like /tmp/chrome/<pid> rather than somehow managing port assignments.

The tech already exists to make these obvious next steps in application provisioning and inter-service communication happen (even Windows supports Local sockets aka UDS), and the scope of the change for existing HTTP client software should be small and of limited scope (URL parsing, name resolution and stream socket establishment steps) but it can't happen unless there is a standardised way to address these sockets.

[annevk]

What exactly is wrong with #577 (comment)? @karwa uds.localhost can resolve locally.

[mnot]

Alternatively, you could try to get uds or socket as reserved TLDs, but I'm not sure how you'd go about doing that.

You ask the IETF, just like .onion did. Admittedly, there are some politics involved, but it's possible, and this is a pretty clearly technical use case. The backstop would be to use a subdomain of .arpa.

Personally, I'd go with something like:

http://%2Ftmp%2Fmysock.uds/foo/bar

Yes, the escaping is ugly, but it's much cleaner than overloading IPV6 in URLs. Alternatively, you might be able to get away with:

http://tmp.mysock.uds/foo/bar

[agowa]

@mnot any update on this? Was it implemented? Should this ticket be reopened? I'm also interested in this.

[mnot]

I just left a comment with some context; I don't know that anything else has happened.

[thx1111]

I haven't read anything here that seems to justify breaking with the familiar pattern, "<protocol>://<domain>/<filepath>" or injecting a lot of special characters into the URL, or mimicking an IPv6 address. The protocol is simply "http". The domain is right there in the name, "Unix Domain Socket". Like any other top level domain - net, com, org - the domain is simply "unix". I don't know any reason that a web browser application cannot parse the domain from a URL, recognize a nonstandard domain name, and invoke a special handler for a non-network socket. The difficulty seems to be in distinguishing the path to the socket from the path to the resource file.

The "HTTP with socket path as the port" option, above, makes the most sense. And since a special handler must already be invoked for this "unix domain", I expect that colons - ":" - can continue to be used as the "port" separator for the socket path.

Altogether, that suggests a straightforward URL, as in: "http://unix:/var/run/server/ht.socket:/path/to/resource.html".

Is there any reason that those repeating ":/" character sequences would pose a problem in a URL?

This approach would not impose any limitation on the use of ":" in the resource path name, since a "unix domain" must be followed by a socket path, and that path will always be delimited by ":/". Any subsequent colons must then be part of the resource path name.

And, of course, this URL format still supports specifying any arbitrary protocol, served through a unix domain socket. And there is nothing redundant or misleading in the URL, as would be the case with any format requiring the name "localhost" or involving special parameter passing.

[michael-o]

http+uds:///path/to/socket?

[rkjnsn]

@michael-o, that doesn't provide any means to specify the resource path, as it is putting the path to the socket where the resource path should go.

[peterjin-org]

But one thing I sort of need to point out in this context is still the fact that the URL standard is still an "interface". Which means that we need to differentiate between attempts to modify the "interface" of a URL by changing the URL standard, and attempts to modify the "implementation" of URLs by changing individual implementations, the former of which is merely a means of abstractly expressing a Unix domain socket or similar string in a URL with few constraints on the actual implementation, and the latter of which is the actual means of connecting to a Unix domain socket i.e. x = socket(AF_UNIX, SOCK_STREAM, 0); x.connect({AF_UNIX, "/some_unix_path"});.

All of the above syntax proposals would effectively be modifying the "interface" of a URL to support an extra method of connecting to a Unix domain socket. In many cases, interface implementations are not required to implement every possible method, if it is known that users of the interface will not use that method, and that is certainly true in other contexts (such as the Java Collections API with immutable or unmodifiable collections used with functions that only attempt to read from the collection). Sorry, but the Liskov substitution principle is not very applicable, otherwise every client that implements URLs would have to support every single URL scheme, and that is simply infeasible. Which means that even if we do have a standard for encoding a Unix domain socket path or similar string in a URL string, we cannot guarantee that every implementation of URLs (such as in browsers) will honor it.

This effectively means that many of the linked issues regarding non-support of Unix domain sockets in various clients that take in URLs might be considered to be wishful thinking, that is, even if a scheme for encoding a Unix domain socket path is devised, there is no guarantee that every app will end up supporting it.

On the other hand, my and @randomstuff's proposals of proxy servers or LD_PRELOAD libraries to support the connection of clients to Unix domain sockets are means of changing the implementation of URLs. It is similar to adding support for a new filesystem in an operating system kernel: the new filesystem can be used by applications transparently, by referencing paths on that filesystem in file access APIs, without having to change the application, because all the different filesystems all share the same interface. This is generally much more feasible to accomplish.

LD_PRELOAD might not be possible for Go binaries at this moment, but this is being worked on.

Ultimately, this means that the mere act of connecting to a Unix domain socket is not necessarily something that requires changing the URL standard, if it is possible to shoehorn it into some existing interface. It may seem very hacky or unsightly, but the major advantage is that client applications do not need to be changed, considering how many HTTP clients or web browsers there exists out in the wild.

A similar issue exists in issue #392 where there is discussion on encoding an IPv6 link local zone identifier in a URL. The mere act of connecting to an ipv6 link local address is something that can simply be done by changing the implementation. For example, interpreting subdomains of the ipv6-literal.example domain as "resolving" a string encoding an IPv6 link local with scope into a sockaddr_in6 which fills sin6_scope_id. It is not necessary to change the interface of a URL in doing so, because it reuses the existing "domain" interface.

A more relatable example is the fact that the URL syntax did not need to be changed in order for connections to domain names to go over IPv6. If the URL standard did not have the square bracket notation, then it would have still been possible to connect to IPv6 websites on the network layer, the only limitation would have been that it would have required the use of a domain name to do so. The main reason why the URL standard ultimately did need to be changed in that case is because of the legitimate interest in connecting to IPv6 literal websites in the same way we could have done it with IPv4.

[robin-aws]

Why was this closed? I found this issue based on searching for a solution to this exact problem, and the discussion mirrored and built on my own thoughts wonderfully.

In fact, HOW was it closed? The GitHub UI isn't even showing a Closed event of any kind in the timeline.

[pauldraper]

[robin-aws]

Ah thank you, I completely missed that. I've never in my life seen this many comments on a closed issue AFTER it was closed. :)

It does seem like there's still a lot of discussion here about whether a different URL scheme is the best solution to the problem - #577 (comment) points out there are at least four different alternatives. Given that I'd argue this issue should be reopened, unless there is another active issue or RFC out there tackling the same problem.

[robin-aws]

I should also mention that like several other commenters here, I'm willing to invest some of my own time into a standard solution, as I'd rather contribute there than build my own local standard solution.

[thx1111]

Hmm - reading back through this all:

@mnot

What's different here is that unix domain sockets have a completely different authority ...

As I discussed above, reading in RFC 3986, I have argued that the Unix Domain Socket, UDS, must be "the 'port' subcomponent of authority of an Address Family AF_UNIX socket" within the meaning of RFC 3986. In reference to that, I have no idea what you mean by "unix domain sockets have a completely different authority". Please explain. Are you arguing that the UDS is not an "authority"? If so, how so? And, what do you mean by "have" an authority, rather than being an authority - and, "different" from what?

The simplest revision to RFC 3986, as I have suggested here, is to generalize the definition of "port" to include a UDS filesystem path, rather than be restricted to being exclusively an AF_INET number, and to also extend the use of the port ":" delimiter, as a kind of "toggle" delimiter using also a trailing ":", optional after an AF_INET number and required after an AF_UNIX socketpath, as in ":some/path:" or ":/some/path:", with the trailing ":" explicitly preceding the "path" component of the "authority" component of the "hier-part" of the URI in RFC 3986. Thus revising the original:

 URI         = scheme ":" hier-part [ "?" query ] [ "#" fragment ]

 hier-part   = "//" authority path-abempty
             / path-absolute
             / path-rootless
             / path-empty

authority   = [ userinfo "@" ] host [ ":" port ]

port        = *DIGIT

to express the URI in more plain and explicit format:

 URI         = scheme ":" authority path [ "?" query ] [ "#" fragment ]

where "path" is already defined explicitly and separately in Section "3.3. Path":

      path          = path-abempty    ; begins with "/" or is empty
                    / path-absolute   ; begins with "/" but not "//"
                    / path-noscheme   ; begins with a non-colon segment
                    / path-rootless   ; begins with a segment
                    / path-empty      ; zero characters
...

and extend the definition of the authority, only changing the description, and not the actual definition, to become instead:

The authority component is preceded by a double slash ("//") and is
terminated by the next question mark ("?") or number sign ("#")
character,  or by the next slash ("/") character after any socketpath,
or by the end of the URI.

authority   = [ userinfo "@" ] host [ ":" port ]

and extend the definition of "port", keeping in mind that "path" is already defined in the RFC, to become instead:

port        = *DIGIT [ ":" ] / socketpath
socketpath  =  path ":"

Note that this approach still allows the use of a ":" in a "hier-part path", distinct from the "hier-part authority", though it would prohibit the use of a ":" in the socketpath itself: "https://example.com:socket/path:/some:path/with/a:colon". The second ":" in the authority terminates the socketpath. Of course, that prohibition would be true for any delimiter that is used for the socketpath. Still, a literal "::" would terminate any "hier-part authority", without actually specifying a port.

Incidentally, from Section "3.3. Path":

   If a URI contains an authority component, then the path component
   must either be empty or begin with a slash ("/") character.  If a URI
   does not contain an authority component, then the path cannot begin
   with two slash characters ("//").  In addition, a URI reference
   (Section 4.1) may be a relative-path reference, in which case the
   first path segment cannot contain a colon (":") character.

I also note that, following my rant about the "://", reference to 'a double slash ("//")' in RFC 3986 Section "3.2. Authority" must not be confused with Section "4.2. Relative Reference", referring to a URI "hier-part path", following a URI "hier-part authority":

   A relative reference takes advantage of the hierarchical syntax
   (Section 1.2.3) to express a URI reference relative to the name space
   of another hierarchical URI.
...
   A relative reference that begins with two slash characters is termed
   a network-path reference; such references are rarely used.  A
   relative reference that begins with a single slash character is
   termed an absolute-path reference.  A relative reference that does
   not begin with a slash character is termed a relative-path reference.

This "relative reference" idea just seems to me to be a needless complication of the URI. I don't know that I've ever seen anyone actually use a "relative reference". And, while most browsers seem to accept a URL with a missing "scheme" and elided "authority", which is to say, the "https://" prefix deleted, that is not an example of a "relative reference" as defined in the RFC.

It would be useful for people to please argue whether this "socketpath as port" interpretation of the UDS, as outlined here, is, or is not: effective, "minimally invasive" to existing standards, and/or not counterintuitive.

[thx1111]

@annevk
Will you please explain how you believe that this issue is "completed"? Seems to be that the issue is very much open. And, I have argued, that the issue arises from a glaring shortcoming of RFC 3986, apparently only considering AF_INET sockets, without accounting for AF_UNIX. And, the "fix" also seems to require nothing more than a minor extension - more of a clarification - to the RFC.

[robin-aws]

I do think there's a good case here for making authority specification more flexible.

I find @thx1111's proposal roughly appealing, as I agree that the current numeric port section is overly biased for specific protocols. At the same time, extending it to only support socket paths as well (or at least a path-like value) feels like it fails to prevent future instances of the same problem for other host schemes. My preference would be to ignore the port for protocols where it isn't useful and instead focus on scheme-specific authority syntax refinement.

As a data point, though, I thought I would go with the http://[v1.uds:/tmp/mysock]/foo/bar option in my current POC, since it looked like it would work based on RFC 3986. However, I found that Java's java.net.URI parser still rejected it, because it was only customized to support IPv6 addresses in square brackets and not the IPvFuture option: https://bugs.openjdk.org/browse/JDK-8019345. Given that, I'm pessimistic that support for any URI format change will percolate through the various existing URI parsers in my lifetime, given RFC 3986 is already 20 years old!

I suspect a supplementary RFC proposing standard ways of embedding alternative kinds of authorities in the existing URI host syntax will be more useful in practice. This could be referenced by extensions to existing schemes like http:, without necessarily splitting off explicit sub-schemes like http+unix:.

FWIW, in my POC I'm finding something building on the ...uds.localhost idea, with an abstraction that maps symbols like foo.bar.blarg.uds.localhost to <UDS root by blarg organization and OS conventions>/bar/foo is working out fairly nicely.

[randomstuff]

FWIW, in my POC I'm finding something building on the ...uds.localhost idea, with an abstraction that maps symbols like foo.bar.blarg.uds.localhost to /bar/foo is working out fairly nicely.

Yes. Something like name.system.uds.localhost and name.username.users.uds.localhost would work nicely (with the major caveat of important security considerations). This is the kind of things I am intending to do with soxidizer.

[annevk]

@thx1111 it's indeed not completed, but I don't think rejection was an option back in 2021. We can't change the URL parser for each new use case that comes along. As Robin suggests above (and mnot explained earlier) there's plenty of room to innovate within the constraints of the existing syntax.

[yrro]

For comparison, OpenLDAP uses a different scheme for ldap-over-UNIX-sockets: ldapi://%2Fusr%2Flocal%2Fvar%2Fldapi connects to the socket at /usr/local/var/ldapi. There is no (documented) syntax for sockets in the 'abstract' namespace (i.e., those where sun_path[0] == 0).

[thx1111]

@robin-aws

At the same time, extending it to only support socket paths as well (or at least a path-like value) feels like it fails to prevent future instances of the same problem for other host schemes.

Please provide or contrive an example.

Also, please clarify - the words "host" and "scheme" are specific and distinct terms-of-art as used in the RFC 3986 definition of a "URI", where "host" is a component of the "authority". By "host scheme", did you mean a "host", or a "scheme" - or something else?

You can find a list of registered "schemes" under "Uniform Resource Identifier (URI) Schemes" at: https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml

We may note that many of the "schemes" registered there, as seen in their description templates, define no "authority" component, and consist of only a "scheme" and "path". Obviously, without an "authority", there would be no "port". Even a "localhost" AF_UNIX URI would have to specify a "host" component, which is not optional if there is an "authority" in the URI, as for instance, literally "localhost". But then, either a "scheme" needs an "authority" - or it doesn't.

Nonetheless, any RFC 3986 defined URI is going to use the exact same definition of "authority", as defined in the RFC. The "authority" does not change with - is not a function of - the type of "scheme" selected. These are two distinct things, the "scheme" and the "authority", within the RFC.

I have not even proposed changing the definition of the "authority". I have only proposed extending the definition of the optional "port" component of the "authority", to allow for an AF_UNIX "path", as distinct from the "url-path" or "urlpath", as it was termed originally in RFC 1738:

url-path
        The rest of the locator consists of data specific to the
        scheme, and is known as the "url-path". It supplies the
        details of how the specified resource can be accessed. Note
        that the "/" between the host (or port) and the url-path is
        NOT part of the url-path.

   The url-path syntax depends on the scheme being used, as does the
   manner in which it is interpreted.

@randomstuff

Something like name.system.uds.localhost and name.username.users.uds.localhost would work nicely...

Hmm - rhetorically, how does a hostname of the form which you suggest, having the top-level domain "localhost", get routed around the Internet? I do not understand what you are describing there. Can you be more specific?

You could use the "userinfo" component of the "authority", preceding the "@" delimiter, to customize the effect of the URI. From the RFC, Section "3.2.1. User Information":

   The userinfo subcomponent may consist of a user name and, optionally,
   scheme-specific information about how to gain authorization to access
   the resource.  The user information, if present, is followed by a
   commercial at-sign ("@") that delimits it from the host.

Did you mean something like that?

@yrro

For comparison, OpenLDAP uses a different scheme for ldap-over-UNIX-sockets: ldapi://%2Fusr%2Flocal%2Fvar%2Fldapi connects to the socket at /usr/local/var/ldapi.

In terms of RFC 3986, that format also places the UDS socketpath into the URI "authority". But, where is the "host" component? The URI "host" is required by the RFC when any "authority" is given. So this OpenLDAP ldapi URI scheme does not comply with RFC 3986, and neither is it registered with the IANA, as is.

But, same as above, "@ host" could be appended to that socketpath, to make it RFC compliant. Still, then URI "userinfo" will have been lost. Still, the socketpath could be added as a "password" for the "user", which is being deprecated anyway. But, strictly speaking, the RFC says:

   ... Applications should not render as clear text any data
   after the first colon (":") character found within a userinfo
   subcomponent unless the data after the colon is the empty string
   (indicating no password).

So, the user could not visually discover or verify the actual UDS socketpath in a RFC compliant application using "socketpath as password".

@annevk

We can't change the URL parser for each new use case that comes along. As Robin suggests above (and mnot explained earlier) there's plenty of room to innovate within the constraints of the existing syntax.

"Plenty of room to innovate" here seems like "code words" for "stuffing a square peg into a round hole." And, please explain how "room to innovate" is something the does not also mean "change the URL parser"?

The classic method used to make an easy problem seem difficult and complicated is to have more parameters than abstract variables. This then becomes a game of "musical chairs", and something always gets left-out, no matter how the parts are rearranged. That's a "fool's errand".

[robin-aws]

@thx1111 - mea cupla, "host schemes" was a poor choice of words. Perhaps "kinds of authority" is a better phrase: the current URI syntax seems to be biased towards TCP, and your proposal tries to at least extend it to more naturally support UDS.

I do have another concrete example in mind: for my use case I also want to be able to do HTTP over VSOCKs. The address for a AF_VSOCK socket is a 32-bit port number together with a 32-bit context ID. I'm not sure how I'd encode a VSOCK address into an authority with your port proposal: something like localhost:vsock/<port>/<cid> isn't terribly natural and could be interpreted as a UDS path instead.

My point being that making the port either a number or a path is also not as future-proof as it could be: IF we want to make the authority syntax less of a round role for other kinds of authority, I'd be inclined to make the authority more opaque at the generix URI syntax level, so specific URI schemes can encode the kinds of authority they support more naturally and with less need for percent-encoding.

[thx1111]

@robin-aws

man 7 vsock:

       The VSOCK address family facilitates communication between
       virtual machines and the host they are running on.
...
       A socket address is defined as a combination of a 32-bit Context
       Identifier (CID) and a 32-bit port number.  The CID identifies
       the source or destination, which is either a virtual machine or
       the host.  The port number differentiates between multiple
       services running on a single machine.

Casual inspection of this description from the man page reveals the terms "port number" and "host". Then, referring to the current version of RFC 3968, I see:

 authority   = [ userinfo "@" ] host [ ":" port ]
 port        = *DIGIT

Rhetorically, what would happen if the VSOCK "host" were a URI "host" and a VSOCK "port number" were a URI "port" number?

Clearly, I'm not appreciating the actual problem there, since "host:port" is already defined in the RFC. What is the actual problem?

[robin-aws]

You're correct that it's also possible to encode the VSOCK information in the URI as http://1234567:123/.... But you don't have a clear indication that the authority uses the VSOCK address family, and it could easily be misinterpreted as an INET authority with 1234567 as a registered name instead.

By the same argument, a UDS authority at /path/to/socket could be encoded as http://%2Fpath%2Fto%2Fsocket/... without your proposed extension. But that also leaves the address family implicit, and definitely has the readability of a square peg in a round hole. :)

I'm trying to say that your proposal makes progress by allowing a path as a port as well as a number, but I think IF we're going to propose a URI syntax change, it should have deeper impact and provide enough flexibility to support other address families nicely as well.

(And to be clear I'm personally not enthusiastic at this point about a URI syntax change, given the IPvFuture extension in 3986, which already does something similar, doesn't seem to be widely implemented AFAICT. I do think it may be productive to propose standard ways to encode authorities from more address families in the existing syntax, so multiple URI schemes such as http/https/ftp/smtp/... that operate on arbitrary stream interfaces could agree on standard approaches.)

[thx1111]

@robin-aws

You're correct that it's also possible to encode the VSOCK information in the URI as http://1234567:123/.... But you don't have a clear indication that the authority uses the VSOCK address family, and it could easily be misinterpreted as an INET authority with 1234567 as a registered name instead.

No, it could not be misinterpreted as an INET authority. And, "1234567" is not a valid top-level domain name.

From RFC 3986:

   The host subcomponent of authority is identified by an IP literal
   encapsulated within square brackets, an IPv4 address in dotted-
   decimal form, or a registered name.  The host subcomponent is case-
   insensitive.  The presence of a host subcomponent within a URI does
   not imply that the scheme requires access to the given host on the
   Internet.  In many cases, the host syntax is used only for the sake
   of reusing the existing registration process created and deployed for
   DNS, thus obtaining a globally unique name without the cost of
   deploying another registry.  However, such use comes with its own
   costs: ...
...
   The syntax rule for host is ambiguous because it does not completely
   distinguish between an IPv4address and a reg-name.  In order to
   disambiguate the syntax, we apply the "first-match-wins" algorithm:
   If host matches the rule for IPv4address, then it should be
   considered an IPv4 address literal and not a reg-name. ...

   A host identified by an Internet Protocol literal address, version 6
   [RFC3513] or later, is distinguished by enclosing the IP literal
   within square brackets ("[" and "]").  This is the only place where
   square bracket characters are allowed in the URI syntax.  In
   anticipation of future, as-yet-undefined IP literal address formats,
   an implementation may use an optional version flag to indicate such a
   format explicitly rather than rely on heuristic determination.
...

man 3 inet:

inet_aton() converts the Internet host address cp from the IPv4 numbers-and-dots notation
into binary form (in network byte order) and stores it in the structure that inp points to.
inet_aton() returns nonzero if the address is valid, zero if not.  The address supplied in cp can
have one of the following forms:

       a.b.c.d   Each of the four numeric parts specifies a byte of the address; the bytes are assigned
       in left-to-right order to produce the binary address.
...

man 7 ipv6

The  address  notation  for IPv6 is a group of 8 4-digit hexadecimal numbers, separated with a ':'.

This reference to a "first-match-wins" algorithm is just another tacit assumption of RFC 3986 which presumes the crafting of an appropriate heuristic. The application author is on-their-own for crafting that algorithm.

We could pretty much apply this same reasoning in the RFC - just presuming some heuristic provided by an application to recognize the URI authority "host" - to the interpretation of the URI authority "port", to distinguish a "*DIGIT" from a socketpath, as in ' path ":" '. There is nothing "novel" in that. It would simply be polite to articulate such a presumption in the RFC itself, if it were to be adopted as a de facto standard in applications.

[peterjin-org]

@randomstuff

Something like name.system.uds.localhost and name.username.users.uds.localhost would work nicely...

Hmm - rhetorically, how does a hostname of the form which you suggest, having the top-level domain "localhost", get routed around the Internet? I do not understand what you are describing there. Can you be more specific?

You could use the "userinfo" component of the "authority", preceding the "@" delimiter, to customize the effect of the URI. From the RFC, Section "3.2.1. User Information":

   The userinfo subcomponent may consist of a user name and, optionally,
   scheme-specific information about how to gain authorization to access
   the resource.  The user information, if present, is followed by a
   commercial at-sign ("@") that delimits it from the host.

Did you mean something like that?

@randomstuff's "soxidier" tool maps domain names to unix domain sockets, such that if the user went to http://name.username.users.uds.localhost/foo/bar then it would be effectively connecting to e.g. /home/username/name, and making the request GET /foo/bar HTTP/1.1, with the possibility of using symlinks to redirect connections to other sockets outside of this pattern.

In this way, the syntax of the URL is not changed, as it looks like a normal URL with a domain name. Only the interpretation of the URL in terms of the network stack is changed.

Mapping domain names or IP addresses to unix domain sockets may be a bit hacky, but the mappings can be automatically generated and managed with scripts, SQL tables, and/or similar.

This basically shows that it is not necessary to modify the URL syntax to merely connect to a Unix domain socket, because as I've said, there are many URL parsers which would have to be updated, and it's not really practical to modify every one of them.

But modifying the URL syntax still has a few advantages. For example, you might have an app that may print out something like this on the console:

===================================

To view a preview of your website, go to http://unix:/var/run/whatever.sock:/foo/bar?query=string in your browser.

===================================

With both my and @randomstuff's solutions, one would have to register the Unix domain socket in the respective tools. But embedding the unix path in this manner does not have this restriction.

[thx1111]

@phjarea217

@randomstuff's "soxidier" tool maps domain names to unix domain sockets, such that if the user went to http://name.username.users.uds.localhost/foo/bar then it would be effectively connecting to e.g. /home/username/name, and making the request GET /foo/bar HTTP/1.1, with the possibility of using symlinks to redirect connections to other sockets outside of this pattern.

Well, fine. But, as I said, that is not an RFC 3986 compliant URI.

A list of IANA recognized top-level Domain Names can be found at: https://data.iana.org/TLD/tlds-alpha-by-domain.txt
"localhost" is not on that list.

Of course, you could configure your own local dns server to resolve "localhost" to some server on your local network, but that is not a general solution. It would be useless for accessing a host on the global Internet.

To be clear, with respect to the solution which I have proposed, the pros and cons come down to a question of a single ":" character. Seriously - the placement of a single ":" character in the context of the entire RFC 3986.

[peterjin-org]

@PHJArea217

@randomstuff's "soxidier" tool maps domain names to unix domain sockets, such that if the user went to http://name.username.users.uds.localhost/foo/bar then it would be effectively connecting to e.g. /home/username/name, and making the request GET /foo/bar HTTP/1.1, with the possibility of using symlinks to redirect connections to other sockets outside of this pattern.

Well, fine. But, as I said, that is not an RFC 3986 compliant URI.

A list of IANA recognized top-level Domain Names can be found at: https://data.iana.org/TLD/tlds-alpha-by-domain.txt "localhost" is not on that list.

Of course, you could configure your own local dns server to resolve "localhost" to some server on your local network, but that is not a general solution. It would be useless for accessing a host on the global Internet.

To be clear, with respect to the solution which I have proposed, the pros and cons come down to a question of a single ":" character. Seriously - the placement of a single ":" character in the context of the entire RFC 3986.

Section 3.2.2:

This specification does not mandate a particular registered name
lookup technology and therefore does not restrict the syntax of reg-
name beyond what is necessary for interoperability. Instead, it
delegates the issue of registered name syntax conformance to the
operating system of each application performing URI resolution, and
that operating system decides what it will allow for the purpose of
host identification. A URI resolution implementation might use DNS,
host tables, yellow pages, NetInfo, WINS, or any other system for
lookup of registered names. However, a globally scoped naming
system, such as DNS fully qualified domain names, is necessary for
URIs intended to have global scope. URI producers should use names
that conform to the DNS syntax, even when use of DNS is not
immediately apparent, and should limit these names to no more than
255 characters in length.

Unix domain socket URIs are not intended to have global scope, due to the local nature of Unix domain sockets. Sure, you might not be able to connect to sockets whose name contains a colon, but one could place a symlink at a file path which does not contain a colon, and point it to a file path that does contain a colon.

[jdimpson]

Allowing use of DNS to resolve a URI to a Unix Domain Socket path sounds like a wonderful gift to hand to malevolent actors. Convincing a user to click a link that resolves to a well-known UDS-based service would become commonplace. Regardless of what any RFC says, a URI referring to local resources should look significantly different from one referring to external resources, so that no person and no legacy or naive code could be confused about what transport mechanism is involved in accessing the resource in question.

[peterjin-org]

Allowing use of DNS to resolve a URI to a Unix Domain Socket path sounds like a wonderful gift to hand to malevolent actors. Convincing a user to click a link that resolves to a well-known UDS-based service would become commonplace. Regardless of what any RFC says, a URI referring to local resources should look significantly different from one referring to external resources, so that no person and no legacy or naive code could be confused about what transport mechanism is involved in accessing the resource in question.

And this is why my solution involves the use of IPv6 link locals without scope id, as opposed to domain names (but domain names may still point to the IPv6 link local addresses), because at least on Linux, they fail to resolve in the network stack by default, however they can still be intercepted and made useful/usable by LD_PRELOAD libraries or eBPF programs.

And just to be clear with my last example with "/var/run/whatever.sock", that is not at all suggestive of the intended URI syntax. What I was suggesting was that if there were a URI syntax then an app would have some means of statelessly encoding the Unix domain socket path, whether it's directly embedding the path string, or substituting, escaping, or percent encoding special characters.

[yrro]

Could the .alt special-use TLD be used? "This top-level label can be used as the final (rightmost) label to signify that the name is not rooted in the global DNS and that it should not be resolved using the DNS protocol."

e.g.:

• http://1234.vsock.alt:5678/whatever
• http://%2frun%2fuser%2f1000%2fproject.sock.unix.alt/whatever
• http://%00mysocket.unix.alt/whatever (abstract namespace)

[peterjin-org]

Could the .alt special-use TLD be used? "This top-level label can be used as the final (rightmost) label to signify that the name is not rooted in the global DNS and that it should not be resolved using the DNS protocol."

e.g.:

* http://1234.vsock.alt:5678/whatever

* http://%2frun%2fuser%2f1000%2fproject.sock.unix.alt/whatever

* http://%00mysocket.unix.alt/whatever (abstract namespace)

Sure, you could have a different TLD for the unix domain socket namespace, but it's still not intrinsically part of the url standard.

If you go to https://jsdom.github.io/whatwg-url/#url=aHR0cHM6Ly9uYW1lLnVzZXJuYW1lLnVzZXJzLnVkcy5sb2NhbGhvc3QvZm9vL2Jhcg==&base=YWJvdXQ6Ymxhbms= you see that the hostname portion of the URL http://name.username.users.uds.localhost/foo/bar is name.username.users.uds.localhost.

The ability to connect to a unix socket in this manner is realized because name.username.users.uds.localhost or 1234.vsock.alt is interpreted at the DNS level to be meaningful (i.e. related to unix sockets), rather than at the URI level. At the URI level, these domain names just look like any other ordinary domain name.

As I've said before, the interpretation of DNS names at the DNS level is operating system dependent and is modifiable by various subsystems. And, it is still necessary to parse the hostname as a domain name to determine that it has a tld of .alt in the first place.

So it's not really the issue of which TLD the unix domain socket namespace is "mounted" on.

[thx1111]

@jdimpson

Allowing use of DNS to resolve a URI to a Unix Domain Socket path sounds like a wonderful gift to hand to malevolent actors.

To review, the topic of this discussion is "Addressing HTTP servers over Unix domain sockets". And now, addressing that goal, I have proposed interpreting an RFC 3986 "URI" - which is what is implied by the phrase "Addressing HTTP servers" in the title - more generally, so as to successfully resolve an AF_UNIX socket "filesystem pathname", as defined in man 7 unix. A "filesystem pathname" is more specifically defined in "POSIX.1-2024", under "3.254 Pathname", here:
https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap03.html#tag_03_254

It is important to note that an AF_UNIX "address" has a format different from other socket address families, where RFC 3986 is especially biased in support of AF_INET and AF_INET6 sockets.

Also, man 7 unix tells us:

       The  AF_UNIX  (also  known  as  AF_LOCAL)  socket  family  is  used to communicate between
       processes on the same machine efficiently.  Traditionally,  UNIX  domain  sockets  can  be
       either unnamed, or bound to a filesystem pathname (marked as being of type socket).

where "efficiently" generally implies side-stepping use otherwise of the local machine's IP stack.

And, while use of an AF_UNIX socket is "to communicate between processes on the same machine", at the same time, the meaning of "Addressing HTTP servers", again, implies use of an RFC 3986 "URI", which itself is subject to additional standards recommendations.

For instance, RFC 4007 extends the Textual Representation of an IPv6 address to include a "zone_id", using the "%" character as a delimiter, and RFC 6874 extends the definition of the RFC 3986 "URI" to address the use of this "%" delimiter, otherwise always treated as an escape character in a URI, such that the "%" character is replaced by its equivalent "escaped" representation, "%25":

      IPv6addrz = IPv6address "%25" ZoneID

Of course any delimiter, for whatever purpose, might be represented using an escaped character sequence, but RFC 6874 only apples to an IPv6 "ZoneID", and is of no use in IPv4 addresses, to otherwise describe an AF_UNIX "Pathname" address as a kind of "ZoneID".

More recently, 2020 June, RFC 8820, "URI Design and Ownership" contains some rather pointed comments warning about overly rigid, difficult, or presumptuous interpretations of the RFC 3986 "URI". For example, RFC 8615, "Well-Known Uniform Resource Identifiers (URIs)", extends the interpretation of the URI "path":

   A well-known URI is a URI [RFC3986] whose path component begins with
   the characters "/.well-known/", provided that the scheme is
   explicitly defined to support well-known URIs.

But, then it goes on to say:

   Applications that wish to mint new well-known URIs MUST register
   them, following the procedures in Section 5.1, subject to the
   following requirements.

   Registered names MUST conform to the "segment-nz" production in
   [RFC3986].  This means they cannot contain the "/" character.

So, a UDS pathname could not, itself, be used as a "/.well-known/" registered name in a URI. Still, a UDS pathname could be modified in other ways, such as using the escaped "/" character, "%2F". But "/.well-known/" registered names seem an unnecessarily complicated approach to accommodating every different address style in every different address family to be found in man 7 address_families.

More generally, the goal in addressing the question raised in this discussion is to provide a simple URI mechanism to accommodate the varied "address" formats used in the many man 2 socket "address families". I have suggested that a single ":" delimiter is a pretty simple and effective solution.

Now, to your point, you seem to be suggesting that some security conscious System Administrator, for example, such as yourself, would configure a publicly accessible server to provide open access to a UDS, and thereby, leave the system vulnerable to some kind of security breach. Is that correct?

Please explain - Why would you do that? Can you provide an example?

There is no standards specification which requires any computer system, networked or not, to provide open access to a UDS. In fact, there are quite a number of mechanisms, already in use, to allow a System Administrator to restrict access to system resources, especially including any networked computer system.

What prevents a System Administrator from making use of those security measures with respect to a UDS, as compared to any other socket family, and especially, as compared to AF_INET and AF_INET6 sockets?

[peterjin-org]

Another example where changing the interpretation of domain names allows connections to Unix domain sockets is the Nginx "upstream" module: https://nginx.org/en/docs/http/ngx_http_upstream_module.html

Here, we use the URL http://backend or http://dynamic. Again, this is a totally normal URL, with the host name of "backend" or "dynamic". The upstream blocks effectively change how domain names and hostnames are interpreted, such that it is possible for these domain names to resolve to Unix domain sockets.