url delims. worthwhile? #3270

isaacs opened this Issue May 15, 2012 · 20 comments


None yet

8 participants


It would be good to maybe discuss this, as it's a common complaint.

On the one hand, the various RFCs are quite clear that whitespace, <>, and quotes are delimiters, and thus cannot ever appear in URLs. In url.js, there's this:

// RFC 2396: characters reserved for delimiting URLs.
delims = ['<', '>', '"', '`', ' ', '\r', '\n', '\t'],

Any delims that are found are thus assumed to be not part of the URL.

However, in practice, this is a bit of a deviation from the way that URLs are handled by browsers in the address bar, the window.location object, and the anchor tag, which serve as models for node's URL parser.

Would anyone object if delims were just auto-escaped?

The change in behavior would be:


Currently this is parsed as

{ protocol: 'http:',
  slashes: true,
  host: 'foo.com',
  hostname: 'foo.com',
  href: 'http://foo.com/',
  pathname: '/',
  path: '/' }

With the proposed change, it'd be:

{ pathname: '%3Chttp://foo.com%3E',
  path: '%3Chttp://foo.com%3E',
  href: '%3Chttp://foo.com%3E' }

Which, incidentally, is exactly what you get if you type <http://foo.com> into a web browser address bar, or click on an <a href="<http://foo.com>">foo</a>. Yes, it is arguably not the user intent, but it's how it's interpreted everywhere else.

While this could potentially reduce the usefulness of the url module as a validator, it'd reduce a common gotcha for newcomers, and would make it a closer model to the browser, which was always the original intent of the module.

If we're going to do it, we should do it completely. We've had these little debates about escaping or not escaping spaces, and quotes, and various things one by one. Maybe we ought to just get rid of the delims altogether.


/cc @bnoordhuis @3rdEden


What is this common gotcha? How does it go?

Node.js Foundation member

-1 for this, though I don't mind if it'll be hidden under some flag or option.


Could both behaviors be triggered by a flag or something? Then the question boils down to what should be the default...


@polotek The common gotcha is "Why doesn't url.parse("http://foo.com/bar baz") work?"

@indutny @n2liquid No, let's just pick one and go with it. url.parse already takes 2 more flags than I'd prefer.

Node.js Foundation member

@isaacs -1 anyway.


Actually, the string passed in the example is not a valid URL per http://tools.ietf.org/html/rfc3986 and treatment of delims should ideally be done in context as in "parse URIs from this blob of text". But AFAIU the semantics of url.parse, no such context is expected to be assumed.


@indutny Do you ever actually have delims in your urls that you need to truncate the url rather than be escaped? Without looking, what do you think this should do? url.resolve("http://foo.com/bar", "baz<baf")?


@s3u I couldn't have put better; I thought about that too but couldn't find the words. However I do believe url.parse() should assume a single-URL string context... which means this stuff should be escaped, otherwise it's not a single URL, and what does it even mean to put two URL's in there?


@s3u RFC 3986 is about URIs, and well outside the scope of node's url parser, which is only concerned with Uniform Resource Locators, of the sort used to address resources on the web.

Also, literally every single place where you can feed a URL to a thing that takes URLs, which is anywhere related to HTTP or JavaScript (ie, web browser address bar, <a> tags, curl, and the location objects in the dom) auto-escape delims rather than truncating.

Delims are only relevant when parsing urls in a blob of text. If you know that the entire string represents a URL, then it seems rather common to assume that they'll be escaped.

The question is really: will this break any existing programs? What gotchas will it cause?

If the answer to both questions is "none that exist, but some that I can imagine, if I try really hard", then that's a +1 for the feature.


@isaacs minor nitpick - RFC 3986 obsoleted RFC1738 - the latter specified URLs.

Node.js Foundation member

@isaacs you convinced me


Since the API already assumes a context, escaping makes sense. But docs need to be clear about the context and how delims are escaped.


@isaacs none that exist, but some that I can imagine, if I try really hard +1

Node.js Foundation member

A downside is that whitespace now becomes significant. The strings 'http://foo.com' and ' http://foo.com ' currently parse to the same URL object. That will no longer be the case.


@bnoordhuis & all: in the particular case of empty characters, I believe the browsers just trim the string instead of escaping.


Non-blank characters are still escaped, obviously.


I do see the issue with the "trimmable" whitespace as @bnoordhuis mentions. This might be where most (if any) see issues.


Alright, seems like we have a rough consensus:

  1. Trim before parse.
  2. Escape delims rather than breaking on them.

I'm fine with this. Good talk.



@isaacs isaacs added a commit to isaacs/node that referenced this issue May 16, 2012
@isaacs isaacs Fix #3270 Escape url.parse delims
Rather than omitting them.
@isaacs isaacs added a commit that closed this issue May 16, 2012
@isaacs isaacs Fix #3270 Escape url.parse delims
Rather than omitting them.
@isaacs isaacs closed this in 9fc7283 May 16, 2012
@AndreasMadsen AndreasMadsen pushed a commit to AndreasMadsen/node-v0.x-archive that referenced this issue May 30, 2012
@isaacs isaacs 2012.05.28, Version 0.7.9 (unstable)
* Upgrade V8 to 3.11.1

* Upgrade npm to 1.1.23

* uv: rework reference counting scheme (Ben Noordhuis)

* uv: add interface for joining external event loops (Bert Belder)

* repl, readline: Handle Ctrl+Z and SIGCONT better (Nathan Rajlich)

* fs: 64bit offsets for fs calls (Igor Zinkovsky)

* fs: add sync open flags 'rs' and 'rs+' (Kevin Bowman)

* windows: enable creating directory junctions with fs.symlink (Igor Zinkovsky, Bert Belder)

* windows: fix fs.lstat to properly detect symlinks. (Igor Zinkovsky)

* Fix #3270 Escape url.parse delims (isaacs)

* http: make http.get() accept a URL (Adam Malcontenti-Wilson)

* Cleanup vm module memory leakage (Marcel Laverdet)

* Optimize writing strings with Socket.write (Bert Belder)

* add support for CESU-8 and UTF-16LE encodings (koichik)

* path: add path.sep to get the path separator. (Yi, EungJun)

* net, http: add backlog parameter to .listen() (Erik Dubbelboer)

* debugger: support mirroring Date objects (Fedor Indutny)

* addon: add AtExit() function (Ben Noordhuis)

* net: signal localAddress bind failure in connect (Brian Schroeder)

* util: handle non-string return value in .inspect() (Alex Kocharin)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment