enhance http.ServeMux routing

[jba]

This is a discussion that we hope will lead to a proposal.

We would like to expand the standard HTTP mux's capabilities by adding two features: distinguishing requests based on HTTP method (GET, POST, ...) and support for wildcards in the matched paths. Both features are particularly important to REST API servers.

Background

The current mux has a few important properties that we want to preserve:

1. The semantics of the mux do not depend on the order of the Handle or HandleFunc calls. Being order-independent makes it not matter what order packages are initialized (for init-time registrations) and allows easier refactoring of code. This is why the tie breakers are not based on registration order and why duplicate registrations panic (only order could possibly distinguish them). It remains a key design goal to avoid any semantics that depend on registration order.

2. The mux is fairly simple and straightforward to understand. It is not a goal to add every last possible bell and whistle. Other custom or more full-featured muxes should remain easy to write and a well-supported part of the Go web ecosystem.

As a refresher, today's patterns used in Handle and HandleFunc take the form [host]/path[/]. The rules are:

• A pattern ending in a trailing slash matches any URL with that prefix.
• A pattern not ending in a trailing slash matches only that exact path.
• A pattern starting with a host name only matches requests for that host.
• The same pattern cannot be registered multiple times (Handle/HandleFunc panics).

In the (likely) event that multiple registered patterns match a request, ties are broken as follows (in order):

1. Patterns with a host win over patterns without a host.
2. Longer patterns win over shorter patterns.

Potential Changes

First, a pattern can start with an optional method followed by a space, as in GET /codesearch or GET codesearch.google.com/. A pattern with a method is only used to match requests with that method. So it is possible to have the same same path pattern registered with different methods:

GET /foo
POST /foo

Second, a pattern can include a wildcard path element of the form {name} or {name...}. For example, /b/{bucket}/o/{objectname...}. The name must be a valid Go identifier; that is, it must fully match the regular expression [_\pL][_\pL\p{Nd}]*.

These wildcards must be path elements, meaning they must be preceded by a slash and then be followed by either a slash or the end of the string. For example, /b_{bucket} is not a valid pattern. (It is not a goal to support every possible URL schema with these patterns; for special cases, using other routers will continue to be a good choice.)

Normally a wildcard matches only a single path element, ending at the next literal slash (not %2F) in the request URL. If the ... is present, then the wildcard matches the remainder of the URL, including slashes. (Therefore it is invalid for a ... wildcard to appear anywhere but at the end of a pattern.)

The tie-breaking rules change to:

1. Patterns with a host win over patterns without a host.
2. Patterns with a method win over patterns without a method.
3. Patterns with a longer literal (non-wildcard) prefix win over patterns with shorter ones.
4. Longer patterns win over shorter ones.

Two patterns with the same literal prefix must not have any possible overlap. For example, it is OK to register both of these:

/b/{bucket}/o/{objectname...}
/b/{bucket}/a/{acl}

because the third path element keeps them from ever matching the same URL.

But it is not OK to register both of these:

/b/{bucket}/o/{objectname...}
/b/{bucket}/{verb}/{noun}

Both of these match, for example, /b/_/o/_ and they have the same length literal prefix (/b/), so there is no clear winner. Instead, the second registration will panic during mux.Handle / mux.HandleFunc.

In contrast, these two are OK, because the longer literal prefix breaks any ties:

/b/{bucket}/o/{noun}
/b/o/{bucket}/{objectname...}

There is one last, special wildcard: {$} matches only the end of the URL, allowing writing a pattern that ends in slash but does not match all extensions of that path. For example, the pattern /{$} matches the root page / but (unlike the pattern / today) does not match a request for /anythingelse.

Examples

Say the following patterns are registered:

/item/
POST /item/{user}
/item/{user}
/item/{user}/{id}
/item/{$}
POST alt.com/item/{user}

In the examples that follow, the host in the request is example.com and the method is GET unless otherwise specified.

1. “/item/jba” matches “/item/{user}”. It is the longest matching pattern (tie-breaking rule 4).
2. A POST to “/item/jba” matches “POST /item/{user}” because that pattern has an explicit method (rule 2).
3. A POST to “/item/jba/17” matches “/item/{user}/{id}” because that is the longest matching pattern (rule 4) and there is no method-specific pattern to override it.
4. “/item/” matches “/item/{$}” because it is longer than "/item/" (rule 4).
5. “/item/jba/17/line2” matches “/item/”. Patterns that end in a slash match entire subtrees, and no other more specific pattern matches.
6. A POST request with host “alt.com” and path “/item/jba” matches “POST alt.com/item/{user}”. That pattern beats “POST /item/{user}” because it has a host (rule 1).
7. A GET request with host “alt.com” and path “/item/jba” matches “/item/{user}”. Although matching patterns with a host beat patterns without a host, in this case the pattern with a host doesn’t match, because it specifies a different method.

API

To support this API, the net/http package adds a new field to Request:

package http

type Request struct {
	...
	Vars map[string]any
	...
}

The ServeMux fills out Vars before calling a registered handler's ServeHTTP method. Of course, other routers may wish to use Vars for their own variables, and they can easily do so. All the values of Vars will be strings if we adopt the design above. But the value type of Vars is any to support those other routers, and to allow for future extensions.

[jub0bs]

I welcome a proposal meant to beef up the capabilities of http.ServeMux. However, I'm worried about the optional-method bit:

[...] a pattern can start with an optional method followed by a space, as in GET /codesearch or GET codesearch.google.com/. A pattern with a method is only used to match requests with that method.

This change would introduce a new way of restricting methods: within the pattern (in addition to within the handler itself). That could cause some confusion.

Besides, imagine I want to configure /foo for Cross-Origin Request Sharing (CORS), possibly via some middleware. CORS typically requires support for the OPTIONS method, because preflight requests use that method. Would the following patterns match a preflight request to /foo?

GET /foo
POST /foo

Or would I also have to register the OPTIONS /foo pattern?

I think you could argue that the concern of restricting methods is best left out of the pattern.

[eliben]

Why do you think this would cause confusion? Our review found that pretty much all the popular Go router packages have this capability (*) in one way or another. It makes the simple cases simple, and doesn't interfere with implementing more complex cases.

(*) For example, gorilla/mux's README mentions explicitly that an OPTIONS matcher has to be set for CORS middleware to work.

[danp]

I like it!

If GET /foo is defined and ServeMux handles a POST /foo request, does it return a 405 (method not allowed)? Or does it continue to return 404 as it does today?

[jba]

I assume you mean only GET /foo is defined.
I don't think we'd want to change existing behavior, so it would return the same status.

[aofei]

I assume you mean only GET /foo is defined.
I don't think we'd want to change existing behavior, so it would return the same status.

Sorry, but it just feels wrong to return a 404 (Not Found) for a request that matches the path part but not the method part. Isn't this exactly the scenario 405 (Method Not Allowed) should be involved in?

The 404 (Not Found) status code indicates that the origin server did not find a current representation for the target resource or is not willing to disclose that one exists.

-- https://www.rfc-editor.org/rfc/rfc9110.html#name-404-not-found

The 405 (Method Not Allowed) status code indicates that the method received in the request-line is known by the origin server but not supported by the target resource.

-- https://www.rfc-editor.org/rfc/rfc9110.html#name-405-method-not-allowed

[danp]

It may be possible to maintain current behavior and support 405 for new method-based patterns, if that is desirable. For example, given these registered patterns:

/foo
GET /bar

Since /foo is registered without a method, current behavior dictates requests for any method go to the registered handler.

For /bar, it only has a method-based pattern registered so a POST /bar request could return 405.

For a GET /baz request, 404 still makes sense as no pattern matches it.

In other words, returning 405 would only become possible if a resource is only accessible via method-based patterns. Otherwise it returns 404. Would that be reasonable?

[jba]

@danp, that does seem reasonable, and backwards-compatible.

[Dmitry-White]

Great idea!

I'm a bit confused with the Changes Example 4
“/item/” matches “/item/{$}” because it is longer than "/item/" (rule 4 - Longer patterns win over shorter ones.)

Since {$} matches only the end of the URL, allowing writing a pattern that ends in slash but does not match all extensions of that path, doesn't it mean that in case of the above example we can default to Current Rule 2 (A pattern not ending in a trailing slash matches only that exact path.)?
I.e. register /item instead of /item/{$}?

[jba]

The pattern /item matches only the path /item. It would not match the path /item/.

The pattern /item/ does match the path /item/, and so does /item/{$}. We use rule 4, longer wins, to distinguish between them.

[Kruemelmann]

Good Idea

My question why not add an additional parameter to the handle function to implement the first part "distinguishing requests based on HTTP method (GET, POST, ...)"
e.g:

How its at the moment:

mux.Handle("/api/", apiHandler{})

How it would look with an additional parameter

mux.Handle("GET","/api/", apiHandler{})

[danp]

@Kruemelmann that would change the signature of Handle(Func) which would break existing programs. Since ServeMux is in the standard library it's covered by the compatibility promise that tries to avoid such changes.

Another option could be to add new methods, like HandleMethod("GET", ...). But having the pattern used with Handle(Func) support an optional method makes sense, similar to it supporting the optional host.

[Kruemelmann]

You are right many thanks for your explanation

[rogpeppe]

In general this looks really nice, thanks, particularly the order-independence invariant.

One thing though:

But it is not OK to register both of these:

/b/{bucket}/o/{objectname...}
/b/{bucket}/{verb}/{noun}

I might well be missing a fundamental issue with this, but this seems overly restrictive to me.
By the "longest prefix wins" rule, this would be OK AIUI:

/o/{objectname...}
/{verb}/{noun}

So I don't really understand why something similar to the "longest literal prefix wins" rule couldn't apply
even in inside a (matching) non-literal prefix, which would allow the first example.

[rogpeppe]

Yes, it means that you have that same flexibility in defining routes whether there's a wildcard parent or not.

It's pretty common to have a top level name (a wildcard) followed by a bunch of routes pertaining to that name. Restricting the "literal wins over wildcard" rule to just the top level seems needlessly restrictive to me and would make the new enhancements less useful for real world cases IMHO.

[jba]

I don't think we should do this, at least not initially, for two reasons.

First, it's hard to describe the rule in clear, simple English. Certainly not as simple as "longest literal prefix wins." That means it will be harder for people to grasp, and harder for them to understand what the problem is when ServeMux complains about conflicts.

Second, I think the rules we already have are complicated enough. For instance, rule 4 is broken (see my recent comment) but no one noticed. The fault is entirely my own, of course, since I made it up, but its ability to slip under the radar of many smart people makes my point.

Evidence for this more general resolution rule would have to come from actual code that benefited from it. If it could be shown that few servers would use the longest-literal-prefix rule, but many would the generalized rule, then we should consider it in spite of its additional complexity. It's on my to-do list to grep a lot of open-source routing code to get a better sense of what features are actually used.

[josharian]

For instance, rule 4 is broken but no one noticed.

I believe that the core problem here is that there is no implementation to try out. It is hard to think through all the consequences of something this in the abstract; nothing forces you to grapple with an API like trying to actually use it.

[deefdragon]

Certainly not as simple as "longest literal prefix wins."

To my understanding, what is being suggested here is basically "Most specific prefix wins". Specific requires a bit more knowledge to know what it means to fully understand the rules, but I would argue so does longest.

As for the benefits, I know a number of sites (twitch and twitch for quick examples) that have, for example, /settings as one page, with /{username} as another page. While these are front-end routers, not back-end, I still feel it shows the need for wildcard and specific matches not being exclusive.

[joncalhoun]

I agree with @rogpeppe on supporting both. I also believe most third party libraries support both, which might suggest it is something people want from a router. Or maybe it is just a coincidence.

For instance, chi supports both patterns and uses the more generalized rule to match. gorilla/mux also supports both, but uses the order the patterns are defined to determine which is used (first pattern defined that matches gets used iirc).

First, it's hard to describe the rule in clear, simple English. Certainly not as simple as "longest literal prefix wins." That means it will be harder for people to grasp, and harder for them to understand what the problem is when ServeMux complains about conflicts.

While true, the docs will also need to document the possibility of a panic if this isn't supported, so it has to explain this situation one way or another. At the moment, the best solution I can think of is a fifth rule that also documents the possibility of a panic:

5. Patterns with equivalent prefixes have these rules applied using their differing suffixes. If no difference is found in the suffixes, registering the second pattern will cause a panic.

[aofei]

Thank you so much @jba! ❤️ Finally someone cares about http.ServeMux, which definitely needs to be improved, both its capabilities and performance.

I just finished reading the thread and have some ideas/questions that I hope will be helpful in finalizing the details before they become a proposal.

1. What data structure will the new http.ServeMux use for routing?

Let's face it, the current performance of http.ServeMux.match isn't great (it's terrible, actually). I mostly blame it on the data structures currently in use.

I highly recommend utilizing a radix tree as the primary data structure for the new http.ServeMux since it's basically the de facto choice for routing. Well-known routers or web frameworks such as julienschmidt/httprouter, gin-gonic/gin, and labstack/echo have adopted it since day one, which speaks volumes.

2. How about support registering two or more methods at once?

There is a scenario where both GET and HEAD methods are usually required to be registered, such as for serving static files.

Instead of

GET /robots.txt
HEAD /robots.txt

I prefer

GET,HEAD /robots.txt

3. How to safely get a string path variable in one-liner way?

I'm delighted that http.Request.Vars was introduced, and I'm glad to see that it's map[string]any instead of map[string]string. I believe it will bring great convenience to existing web frameworks that wrap net/http. People have had enough of using http.Request.WithContext with context.WithValue to store arbitrary data.

But to be honest, for practicality, map[string]string is actually better than map[string]any because, with the former, we don't need to do typecasting.

Instead of

id, _ := req.Vars["id"].(string)
user := GetUserByID(id)

I prefer

user := GetUserByID(req.Vars["id"])

But still, I'm personally glad it's map[string]any, because it offers more extensibility. So I was thinking, maybe we could also introduce http.Request.PathVar(name string) string to make life easier.

4. Why doesn't the pattern matching order take path element type into account?

I don't think that simply prioritizing longer patterns over shorter ones is a good idea. For example, I don't want github.com/{username} to win over github.com/sponsors, since I want to do special treatment for the later one. Maybe the example isn't great, but I believe it makes some points.

Instead of relying solely on pattern length, I think it would be better to establish a matching order based on the type of path element, such as: static > variable > wildcard-variable.

5. Why not use the :name style for registering path variables like most routers do?

I believe /b/:bucket/o/* is more concise and convenient for string matching than /b/{bucket}/o/{objectname...} in practice. Any concerns about choosing the colon-style?

6. How about we make http.Request.Vars non-exported?

Consider the following API design:

package http

type requestVarKey struct {
	name string
}

var PathVarsRequestVarKey = &requestVarKey{"path-vars"}

type Request struct {
	...
	vars map[any]any
	...
}

func (r *Request) Var(key any) (value any, ok bool) { ... }

func (r *Request) SetVar(key, value any) { ... }

func (r *Request) PathVars() map[string]string { ... }

func (r *Request) PathVar(name string) string { ... }

func (r *Request) setPathVar(name, value string) { ... }

This design is intended to avoid collisions between packages using http.Request.vars. For example, any middleware is free to utilize http.Request.SetVar to store their arbitrary data within request-scope without concerns about tampering by others.

For http.Request.PathVars, the implementation is:

func (r *Request) PathVars() map[string]string {
	pathVars, ok := r.Var(PathVarsRequestVarKey)
	if !ok {
		pathVars = map[string]string{}
		r.SetVar(PathVarsRequestVarKey, pathVars)
	}
	return pathVars.(map[string]string)
}

7. How about we make the http.Handler used internally by http.ServeMux.Handler customizable?

Currently, http.ServeMux.Handler returns a "page not found" handler for requests that do not have a match. It also provides an internally-generated handler that redirects to the canonical path for a matched pattern ending in a trailing slash. It would be nice if they were all customizable.

Consider the following API design:

package http

type ServeMux struct {
	...
	NotFoundHandler Handler
	MethodNotAllowedHandler Handler
	TSRHandler Handler
	...
}

The MethodNotAllowedHandler is used when a request path is matched, but its request method is not.

The name TSRHandler is an abbreviation for "Trailing Slash Redirect Handler".

[jba]

Thanks for your detailed comments. I'll reply by number.

1. Implementation is out of scope for this discussion/proposal. I think we'd be happy to have a more complex implementation if it could be demonstrated that the current one actually affects latency or CPU usage. For typical servers, that usually access some storage backend over the network, I'd guess the matching time is negligible. Happy to be proven wrong.

2. Adding a multiple-method syntax would be worth it if that happens a lot. We'd need more data.

3. @willfaught already addressed this.

4. I'm not really sure what your proposed matching algorithm is. In the example you gave, github.com/sponsors does win over github.com/{username}, contrary to what you said, so maybe the current algorithm does what you want? It is sensitive to whether something is a variable or not, by the "longest literal prefix" rule.

5. Curly braces have some precedent too (URL templates). And if we used :name then we could never extend the pattern language to partial matches in a path segment, like ab{c}d. We don't want to do that now, but at least the braces make it possible.

6. This seems less about whether Request.Vars is exported and more about making it part of a general mechanism for storing data in Requests. That mechanism is out of scope for this discussion, but I'll point out that since names in braces must be Go identifiers, there are still many strings available for keys into Request.Vars that won't conflict with mux patterns.

7. Interesting thought, but also out of scope for this discussion. Worthy of its own proposal.

[zjzjzjzj1874]

For the fourth point, I think supporting it would make it too complicated.

4. github.com/{username} should already include the matching rule for github.com/sponsors. If we need to validate "static" first, then validate the variable, and finally validate the regular expression matching rule, it will require several additional layers of switch and case statements. I still think that matching based on length is easier to understand.

[aofei]

Thanks for your reply! @jba

---

1. Implementation is out of scope for this discussion/proposal. I think we'd be happy to have a more complex implementation if it could be demonstrated that the current one actually affects latency or CPU usage. For typical servers, that usually access some storage backend over the network, I'd guess the matching time is negligible. Happy to be proven wrong.

Well, I can agree with this, indeed we should first determine the matching algorithm, and then talk about the implementation details.

I have to say though, that having a better matching performance is good for everyone, even if the matching time is negligible compared to the execution time of the matched http.Handler. There's a reason people keep benchmarking routers.

---

2. Adding a multiple-method syntax would be worth it if that happens a lot. We'd need more data.

Yes, I agree. Since we need more use cases and adding support for it in the future won't break compatibility, I believe it can wait.

---

3. @willfaught already addressed this.

If you mean user := GetUserByID(req.Vars["id"].(string)), then no, this should definitely not be a recommended usage, it's not safe at all. As @deltamualpha pointed out, it will cause a panic if "id" does not exist or is not a string. I believe we all agree that mistyping a literal string name is always possible.

---

4. I'm not really sure what your proposed matching algorithm is. In the example you gave, github.com/sponsors does win over github.com/{username}, contrary to what you said, so maybe the current algorithm does what you want? It is sensitive to whether something is a variable or not, by the "longest literal prefix" rule.

Ok I'll try to provide a detailed explanation of the matching algorithm I mentioned. Given that this algorithm differs significantly from what you're proposing, I thought it would be best to write about it in a separate comment. Please allow me some time.

---

5. Curly braces have some precedent too (URL templates). And if we used :name then we could never extend the pattern language to partial matches in a path segment, like ab{c}d. We don't want to do that now, but at least the braces make it possible.

Well, I'm convinced. I agree that using the curly braces style is more appropriate. I just realized that this style also allows for adding support for things like /users/{id:/^\d+$/} (regular expressions), for example. Anyway, it's more extensible.

---

6. This seems less about whether Request.Vars is exported and more about making it part of a general mechanism for storing data in Requests. That mechanism is out of scope for this discussion, but I'll point out that since names in braces must be Go identifiers, there are still many strings available for keys into Request.Vars that won't conflict with mux patterns.

Well, after thinking about it, actually yes, I do wish there was a general mechanism for storing arbitrary data within request-scope, apart from the http.Request.WithContext approach. Unfortunately, such a mechanism doesn't exist here. And I kind of believe that if I open a proposal to add support for the http.Request.{vars,Var,SetVar} I mentioned, it will probably be rejected, given that the proposal you're about to open covers some of it. So I guess even if you agree that http.Request.{vars,Var,SetVar} is a good idea, it should only be part of your proposal.

My proposed API design is mainly to solve these two problems:

• What I pointed out in 3, a safe and straightforward way to get string path variables. I believe http.Request.PathVar(name string) string is a good way to go.
• What I mentioned in 6, a way to store arbitrary data within request-scope without concerns about tampering by other packages (accidentally or not). That is to use vars map[any]any instead of Vars map[string]any, which kind of aligns with the design of context.{Value,WithValue}. In this way, there will be no collision between the names of our path variables and other types of data, because our path variables are just a map[string]string stored in vars map[any]any with http.PathVarsRequestVarKey as its key.

However, if you still insist that the general data storage mechanism we're discussing shouldn't be included in the proposal you're about to open, then I think we can consider opening a separate dedicated proposal for it.

---

7. Interesting thought, but also out of scope for this discussion. Worthy of its own proposal.

Considering that the topic of this discussion is http.ServeMux routing, and the three fields I proposed are closely tied to the behavior of http.ServeMux.Handler (as they will be returned by http.ServeMux.Handler), I believe they are within the scope.

Furthermore, if you consider adopting the design proposed by @danp in #60227 (reply in thread), it would necessitate an additional internally-generated MethodNotAllowedHandler regardless. This also presents an opportunity to introduce these fields.

[jba]

a safe and straightforward way to get string path variables

Presumably PathVar should return the empty string if the name is not in the map, or if it is not a string?

With generics, why stop at strings? We could have a general function that returns a value of type T from Request.Vars, or the zero value if the name isn't in the map or is the wrong type.

But why make that just for Request.Vars? Here is a function that will do that for any map[string]any: https://go.dev/play/p/VkimAPWVZyc

[aofei]

With generics, why stop at strings? We could have a general function that returns a value of type T from Request.Vars, or the zero value if the name isn't in the map or is the wrong type.

Are we talking about supporting things like {id:~int64}, {latitude:~float64}? If that's the case, then map[string]any is indeed better than map[string]string for storing path variables. But honestly I don't have much hope for that, given that it took all these years for the Go team to finally care about http.ServeMux, adding such fancy features in the foreseeable future looks unrealistic to me. The foreseeable future, who knows, perhaps by then #49085 will be resolved (if only we could be that lucky), allowing func (r *Request) PathVar(name string) string to evolve into func (r *Request) PathVar[T any](name string) T.

Personally, I'm not a big fan of the "typed path variables" feature. Since path variables are always strings, I prefer to parse them myself, so that I can provide more detailed error responses to the client.

It's all around path variables. But if you mean only doing ValueOrZero on arbitrary data other than path variables, then I think there's nothing wrong with my proposed API, it can be done with http.Request.{vars,Var,SetVar}.

[mateusz834]

“/item/jba” matches “/item/{user}”. It is the longest matching pattern (tie-breaking rule 4).

Isn't that a compatibility breaking change?
It this change going to change the behaviour of this example?

func main() {
	files := []struct {
		name string
		data string
	}{
		{"{myfile}.txt", "myfile data1"},
		{"myfile.txt", "myfile data2"},
		{"{myfile2}.txt", "myfile data3"},
		{"{myfile2}", "myfile data4"},
	}

	for _, v := range files {
                v := v
		http.Handle("/sth/"+v.name, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			w.WriteHeader(http.StatusOK)
			w.Write([]byte(v.data))
		}))
	}

	http.ListenAndServe("localhost:8888", nil)
}

[jba]

Curly braces are unsafe in URLs. Of course that doesn't mean that some Go server somewhere doesn't depend on them. If there is a lot of potential breakage we'd add a GODEBUG setting to preserve backwards compatibility (see #56986). We're hoping that's not the case.

[josharian]

If there were an initial draft of this API, even one that was mostly stubbed out in term of actual implementation, I could try porting our existing routing code to it to discover what does and does not work. (It is important that it actually be something that I can send to the compiler, though, otherwise subtle issues tend not to get noticed.)

Are there any plans to post such a rough or draft implementation for people to try out?

[jba]

Not sure what "mostly stubbed out in term of actual implementation" means. Do you want just the matching part, where you could register patterns and then feed it an http.Request and see which pattern matched?

[josharian]

Do you want just the matching part, where you could register patterns and then feed it an http.Request and see which pattern matched?

That'd be great, at least to my mind. I'm sure opinions will vary wildly.

[AndrewHarrisSPU]

It seems like the questions about matching rules should turn out to be isolated from questions about parsing values from input, but it might be hard to really get a feel for things without some way of experimenting with the parsed values. If the discussion can stay on track while also implementing something noncommittally for parsing values, I think it'd be clarifying.

[MicahParks]

I am in favor of this proposal. I'd like to ask to rename the proposed Vars field on the Request data structure to PathVars.

Generally speaking, I'm in favor of short names. However, I feel adding the Path prefix to Vars makes the field's name more intuitive.

This would be an 8 character field name. But for comparison, the mean and median length of existing field names on the http.Request data structure is 7, so it's not far off.

[seankhliao]

While this proposal will only place values parsed from the path there, having it named Vars lets middleware place parsed values from other places (eg headers) there without being out of place.

[flibustenet]

With any inside, we'll have the same issues that we have with context.Value ? Or maybe it's a good idea to replace usage of context in middleware ?

[nfisher]

Should this be introduced as a golang.org/x package and then moved into Go proper once stabilized?

[carlmjohnson]

I don't understand the problem this is meant to solve.

By way of contrast, I understand that the problem to be solved for log/slog was that there were a lot of third party structured loggers, but if you made a stand alone library, you couldn't count on which one of the many loggers your consumers would want you to interface with. The new slog package provides a common denominator, so that whatever consumer facing log API you use or whatever log sink backend you use, it can all interface with log/slog and everyone is happy. There was a clear problem statement.

What's the problem statement for http.ServeMuxt? So, obviously, there are lots of other HTTP routers for Go. Is the problem to be solved that they're incompatible? In that case, then I guess Request.Vars is good to have, although it's unfortunate that it has to go on HTTP client requests as well. In any event, existing third party routers have mostly settled on either wrapping the standard library with their own (e.g. gin.Context) or passing things through context.Context (which is not ideal in terms of type safety, but Request.Vars doesn't help here either). I don't see how this makes them significantly more compatible. In the short run, things will be worse because a new crop of third party routers will use Vars, while the old ones will all continue to exist and not use Vars.

Maybe the problem to be solved is that the standard ServeMux is just slightly underpowered? That might be true, but it's not clear why that problem needs to be solved and where to draw the line in solving it. I guess the urgency is because gorilla/mux is archived? It's hard to understand why the line is being drawn where it is and what the advantage of doing so is. It just feels a bit arbitrary to include methods and path parameters but not something else.

[josharian]

Great question.

Speaking as someone who is excited about this proposal, it is because gorilla/mux got archived, and I would prefer to use some thing standardized and out of the box that I know will be around approximately forever, instead of researching and hopping back and forth between third party muxes as they come and go.

[josharian]

On the standardized interface front, the main thing that I would like to see is have http.Handler grow an error return value. I’ve seen and written so many shims to do that. A std shim would be nice. But that is not this proposal. :)

[carlmjohnson]

I've started designing my APIs to not have path variables any more, because I think they're mostly unnecessary noise, so I'm less excited. 😄 But I do think it would be interesting to think about officially blessing the concept of middleware (func(http.Handler) http.Handler) with simple functions for composing stacks and whatnot.

[neild]

It's unfortunate that there's currently no easy way to register a ServeMux path for only a single method. #59470 is an example of this causing problems--http.Handle("/", http.FileServer(http.Dir("/tmp"))) will surprisingly respond to DELETE requests with the contents of a file. One way to address this would be to make http.FileServer method-aware, but a simple and clear alternative would be to make it easy to register a handler for GET and HEAD requests only.

I suspect there are many, many Go HTTP handlers out there that naively respond to all methods. The inability to scope down a registration to a single method or methods in ServeMux is a fairly large gap.

More broadly speaking, the existence of so many third-party HTTP routers seems to point at the standard library ServeMux being underpowered. It doesn't need to be all things for all people, but it seems like it could satisfy many more users than it does today without too much gain in complexity.

[carlmjohnson]

I looked at my own usage of go-chi, and I could basically rip it out and replace it with a few middleware helpers if http.ServeMux supported method restrictions. It's probably a good idea to do something. I just think it's good to be clear about what you're trying to do before you do it if you want it to be done well. :-)

[carlmjohnson]

Focusing on Request.Vars, existing routers have mostly been using context.Context to convey path variables. Why not an API like this, so they could be backwards compatibly adapted to using path vars also:

func PathVars(context.Context) map[string]any

// WithPathVar adds one path var
func WithPathVar(context.Context, string, any) context.Context

// WithPathVars adds many path vars
func WithPathVar(context.Context, map[string]any) context.Context

// Convenience method:
func (r *Request) PathVars() map[string]any { return PathVars(r.Context()) }

[carlmjohnson]

Wouldn’t any third party router that modifies .Vars also have to make a copy of the request to avoid the race issues? ISTM, you would still have the problem in #58809 in that case.

[jba]

I don't understand the concern about races. http.Request has a lot of exported fields, so there are many potential race conditions. But I believe that the handler that gets the request from the server has exclusive access to it (that is, no other goroutine can see it), and middleware handlers, including third-party routers, typically run sequentially. What am I missing?

[carlmjohnson]

IIUC before context existed, Gorilla and other routers did request specific contexts by having a map of request to data. Because of that, any changes to the request were potentially racey and had to be done by copying the request instead. I’m not sure if it still applies or I’m misunderstanding the history. When I wrote MaxBytesHandler, I put in a defensive copy of the request and no one else told to me to take it out!

go/src/net/http/server.go

Line 3639 in d75cc4b

r2 := *r

[carlmjohnson]

The docs for Handler say, “Except for reading the body, handlers should not modify the provided Request.” I assumed that was because of some race something.

[carlmjohnson]

StripPrefix also makes a copy of its request and the request URL fwiw.

I’m not sure how that advice fits with ParseForm, which modifies r.Form and r.PostForm, although I guess it fits under the concept of “reading the body”.

[nicolasparada]

Here is another idea for a method handler:

// MethodHandler maps each handler to the corresponding method.
// Responds with "method not allowed" if none match.
type MethodHandler map[string]http.HandlerFunc

// ServeHTTP dispatches the request to the handler whose method matches,
// otherwise it responds with "method not allowed".
func (mh MethodHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	allow := make([]string, 0, len(mh))
	for method, handler := range mh {
		if method == r.Method {
			handler(w, r)
			return
		}

		allow = append(allow, method)
	}

	w.Header().Set("Allow", strings.Join(allow, ", "))
	http.Error(w, http.StatusText(http.StatusMethodNotAllowed), http.StatusMethodNotAllowed)
}

mux := http.NewServeMux()
mux.Handle("/foo", MethodHandler{
	http.MethodGet:  getHandler,
	http.MethodPort: postHandler,
})

What do you think?

[jba]

Can you describe in words what the idea is, and how it relates to the topic of the discussion?

[nicolasparada]

Sure.
It's an alternative way to add support for routing based on the request method.
This doesn't take into consideration URL path variables and works perfectly fine with the current ServeMux, so the mux can focus on routing URLs to handlers.

[FarmerChillax]

this looks great

[flibustenet]

Very welcome proposal !
map[string]any + .(string) look a little bit weird and boilerplate when we just parse and match a string. Maybe it could be intuitive to have request.PathValue(string) string when we already have request.FormValue(string) string.
@jba Could you say more about "any to support those other routers, and to allow for future extensions" ?

[hherman1]

I also feel concerned about the use of any here. It would be helpful if there were specific examples you had in mind @jba which take advantage of the any type and justify the cost.

While it’s tolerable to be required to look up the docs for my router to figure out what types I should be casting to, it would ease my cognitive load if I didn’t have to do that look up.

[jba]

One could imagine a router that did type conversions for you. For instance, it would accept wildcard patterns like {size:int}, attempt to convert the string to an int, and store the int in the map.

In fact, we had something like that in an early design, but weren't convinced it was worth it. Still, other routers may want to do something like that, and we might consider it as a future extension.

[hherman1]

But then, why not add such an extension when there is actual demand for this feature, and build the thing we know we want now. Would it be so bad if we added a map[string]any down the line? I guess I just don’t find it super compelling that the basis is an imaginable use case but one y’all deemed not worthwhile.

[flibustenet]

I can see the benefit to improve matching pattern like we have in gorilla /articles/{category}/{id:[0-9]+}, because it's the job of a router to match and route.
But it's difficult to see the benefit of conversion in a router when we'll still have to convert the type of the result in the handler, specially that we'll have to do this even if it's a string.
I mean doing id,_ := r.Vars["id"].(int) instead of id,_ := strconv.Atoi(r.Vars["id]) doesn't help a lot (we should test the error right). And s, err:= r.Vars["s"].(string) would be difficult to explain to a beginner.
But it's also true that most of the time path vars are int, i'm not surprise that you was searching a design for that ! strconv or a custom func GetById is fine and more explicit for me.

[igoracmelo]

For not affecting who already uses http.ServeMux with the current behavior, it could be a good idea do create a new mux type with these new capabilities.

[jba]

That would involve a lot of code changes, perhaps well downstream of the ServeMux use. The nice thing about this proposal is that it is (in theory at least) backwards-compatible.

[jba]

I think rule 4 should be changed. It says that if there are two patterns that match the same request and aren't distinguished by the first three rules, then the longer pattern wins over the shorter one. This leads to surprising results. For instance, consider the patterns /a/{x} and /a/{xx}. These two patterns have no method or host and the same length literal prefix, so their precedence isn't resolved by the first three rules. Both patterns match paths like /a/b. According to rule 4, the second pattern would win. But it seems odd that a pattern that just happens to use a longer wildcard name has an advantage over one with a shorter name. Wildcard names shouldn't matter, and these two patterns should conflict with each other.

I added rule 4 for one reason: to select /{$} over /. I now believe that is the only purpose it is good for. So I suggest we change it to

4. A pattern that ends in {$} wins over one that doesn't.

This rule would allow both / and /{$} to coexist in a mux, as well as pairs like /a/{x}/{y...} and /a/{x}/{$}, and would choose the latter in each case. That is in keeping with the spirit of these rules—to choose the more specific pattern—without being overly general.

[jba]

The modified rule 4 is still wrong. It doesn’t account for cases like

/       vs.  /{x}
/{x}/   vs.  /{x}/a
/{x}/   vs.  /{x}/{y}/{z...}

In all these cases, the left pattern, the one with the trailing slash, should lose. We would like a pattern that ends in a slash or a "..." wildcard to act as a fallback, matching all paths that aren’t matched by other, similar patterns.

While there may be a way to describe that syntactically, it would be complicated. I think it is time for a new approach. I propose replacing both rules 3 and 4 with a new rule:

3': p1 wins over p2 if p1's path pattern is more specific than p2's.

A path pattern p1 is more specific than p2 if p2 matches all the paths of p1 and more. (To put it another way: the set of paths matched by p1 is a strict subset of that matched by p2.)

This rule has some nice properties. It is easy to state and easy to understand. It says what we really mean by “more specific,” instead of providing a syntactic ersatz. It gives us what we want for a trailing-slash or "..." wildcard pattern: that pattern is always more general than another pattern with the same prefix, so it will always lose.

It also provides additional power for simpler patterns. The longest-literal-prefix rule allowed both /a/{x}/b/{y} and /a/default/b/{y}, because the second has a longer literal prefix. But if you wanted to default the second wildcard instead, by registering the two patterns /a/{x}/b/{y} and /a/{x}/b/default, you would get a conflict. The new rule allows either of those:

/a/{x}/b/{y} and /a/default/b/{y}

or

/a/{x}/b/{y} and /a/{x}/b/default

because in both cases, the second pattern is more specific than the first. (It would not allow both default patterns together, however, because they overlap without either being more specific than the other.)

The new rule disallows some patterns that other rules allow. The patterns /a/{x} and {x}/b both match /a/b but neither is more specific than the other; they conflict according to rule 3'. These two patterns are allowed by rules that work left to right, like the longest-leftmost-prefix rule and the lexicographic rule of github.com/cespare/hmux.

But maybe it’s a good idea to reject those patterns. Consider a real-world example I found: /exports/{id} and /{domain}/events. If there is a domain called "exports", then the first pattern will match /exports/events when the second should. Patterns that overlap like this are probably a bad idea.

By the way, I looked at a lot of routing patterns over the last couple of days, and it seems that the vast majority of routing schemes use disjoint paths, either by having different numbers of segments or different literal segments in the same position. There were a few pairs of patterns like /item/{id} and /item/new, which all the proposed rules can handle. Patterns where the differences in the rules mattered were rare.

The remaining problem with the new rule is how to turn the definition of “more specific,” with its talk of possibly infinite sets of paths, into something computable. Happily, our patterns are simple enough that we can determine in a single pass whether one is more specific than the other, whether they overlap with neither being more specific (and thus conflict), or whether they are disjoint. The algorithm can be modified to also return a string that demonstrates overlap, which might be useful to display when conflicts are reported.

An implementation is available at github.com/jba/muxpatterns.

[AndrewHarrisSPU]

Thanks for posting this! 'More specific', defined by subset relationships, is a concise way to say a lot.

Intuitively, rule 3' and rule 1 seem nice together, but my intuition is really fighting the ordering of of rule 2 and rule 3'. If GET / and /foo are registered, GETting /foo matches GET /. Am I backwards on this? There's no strict subset relationship either way, yet I'm finding it hard to see how /foo isn't more specific than GET / here.

follow-up: looking at some of the big existing packages, they all have APIs that put http methods on muxer/router methods (i.e. router.Get(...)), but also have some option that isn't specific to http methods like router.Method(...) or router.Any(...).

• chi has a Mount for methodless registration, and matches paths with higher priority than methods.
• echo has an Any registration
• gin has an Any registration
• httprouter allows registration of a methodless /foo with Handle alongside GET /, but GETting /foo serves a 405. The documentation for httprouter is sort of frank about being strict here.
• gorilla/mux depends on registration order

I think my (intuition/opinion) fits the semantics of chi's Mount - like cespare’s hmux Handle, no method given in registration means all methods match. echo and gin are more explicit (a pseudo-method like ALL /foo would be analogous, I think). httprouter's approach can't fit with existing usage of the standard library, and gorilla/mux's approach can't be considered because of registration order.

[seankhliao]

If there was a wildcard, * is probably a better option that aligns with existing web standards: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Methods