Support custom protocols

[zcbenz]

• API to register custom protocol.
• Allow sending file in custom protocols.
• Allow sending string in custom protocols.
• API to control the cache of each request.
• API to intercept existing protocol.
• Provide information of from where the request is made (not sure if it's possible).

[zcbenz]

The WebKit only allows setting referrer for urls with http or https protocols, so the handler registered with protocol.registerProtocol will not be able to receive the referrer, which is critical for implementing custom browser resource loader. The only solution I can thought of is to patch WebKit.

[nathansobo]

Forgive my ignorance here, but I don't understand why the referrer header is required. Is it required to resolve relative URLs or something?

[zcbenz]

it's required to resolve file paths in URLs with custom protocols, when writing something like src="atom://image.png" in HTML, browser would not translate it to src="atom:///path/to/current/page/image.png", because the browser thought it's an absolute URL. To correctly provide the requested file, we have to know from which page the request was sent, and then resolve the paths.

[nathansobo]

Yeah, that makes sense. We could use the custom protocol only to resolve absolute references on the load path, but I think it would probably be better to traverse the ancestor directories looking for node_modules similar to how node resolves a path when require is used. What do you think about patching Chromium then? I know we wanted to minimize changes but this feature seems really important to me. Perhaps we could get the patch upstreamed? Maybe @aroben could offer an opinion or assistance?

[aroben]

I'm not sure I understand the requirements. Can you give some examples of URLs and the file paths they should map to?

[nathansobo]

We're looking for a coherent mechanism for referring to images and other static resources provided by third-party packages. I'm imagining that index.html sits in Atom's root directory, and we resolve any static asset references against the same load path we use for requiring scripts. That means I could refer to atom://core/images/logo.png and it would resolve to Atom.app/Contents/Resources/node_modules/core/images/logo.png. Or the a custom tree view package could write an image tag to the DOM referencing atom://tree-view/images/oak.png that would resolve to ~/.atom/packages/tree-view/images/oak.png. Right now everything is relative to index.html which is sitting in some directory inside the app bundle on disk, which is pretty useless. Maybe all relative paths should be resolved against the load path by default? Because a relative path is otherwise pretty useless.

[zcbenz]

This is how I think the API would be like:

var protocol = require('protocol');
protocol.registerProtocol('atom', function(url, referrer) {
  // The url is 'atom://avatar.png',
  // and the referrer is 'file:///Applications/Atom.app/.../index.html',
  // Then we calculate the actual path to requested resource by traversing ancestor directories:
  // '/Applications/Atom.app/.../avatar.png',
  return new protocol.RequestFileJob(calculatePathFromUrlAndReferrer(url, referrer));
});

As for referrer, I just found that adding <meta name="referrer" content="always"> in HTML would force WebKit to send referrer to all requests, so it's not a problem any more.

But another problem is, to resolve resource path like node's require, we have to know from which .js file the request is sent. In the case of ArchiveView, we could add an image when creating the view in /Applications/Atom.app/Contents/Resources/node_modules/archive-view/lib/archive-view.js, or in ~/.atom/packages/archive-view/lib/archive-view.js, in order to know whether atom://image/zip.png points to /Applications/Atom.app/Contents/Resources/node_modules/archive-view/images/zip.png or ~/.atom/packages/archive-view/images/zip.png, we have to know which .js file requests the image. The referrer just always points to the index.html.

[nathansobo]

It seems really difficult (if not impossible?) to tie URL references in the DOM back to the .js file that inserted them. Maybe everything can just always be relative to index.html, which will be a sibling of the built-in node_modules, but we can also add ~/.atom/packages to the load path. Then you will have to explicitly mention the package name in the path atom://archive-view/images/zip.png instead of atom://images/zip.png. I guess alternatively we should just use fsUtils.resolveOnLoadPath('../images/zip.png') to make paths absolute before inserting them into the DOM. We could potentially add a convenience function in our view layer to help make this easy:

class SomeView extends View
  @content: -> @img src: @resolve('../images/flower.png')

Could we add some kind of mechanism like require.resolve that would resolve relative paths to any kind of file, not just scripts? Then we could just handle it explicitly in scripts. Maybe that makes the custom protocol less necessary? Either way, it seems like once the URL is in the DOM it will be very hard to determine where it came from.

[nathansobo]

@zcbenz After talking with Kevin the API you propose for registering a custom protocol from the JavaScript layer looks really good. Is it possible to provide a similar API to allow us to intercept relative file:// URLs? We would like to make it possible to refer to a less stylesheet in markup for example and automatically compile it before handing it back to the browser.

[nathansobo]

Another thought: I wonder if it would improve performance to register callbacks that are associated with regular expressions? Then we can run the regular expression against the URL in the native layer and only call into JS if it matches a certain pattern.

[zcbenz]

I found that the Network panel of devtools has a column of initiators, which is the exact thing we want for implementing the require.resolve style of resource loading, but the magic hides in the private WebKit code, I'm seeing how to expose it.

[zcbenz]

So the devtools' initiators feature is implemented by dumping the current JavaScript stack trace just before the request is sent, it's a clever idea. If we want to do the same in atom-shell, we still need to patch Chromium, since devtools is making use of private WebKit API.

[zcbenz]

After some explores on implementing the initiator feature, I think there are 4 types of initiators we need to get:

1. requests from static pages, can be got via referrers.
2. ajax requests, can be got via devtools's initiator column of Network panel.
3. requests from dynamic DOM inserted by JavaScript scripts.
4. requests sent from CSS.

The 1 is simple, the 2 can be implemented but needs small patches on Chromium, the 3 seems impossible, and the 4 needs big patches on Chromium.

Now I give up on this feature and stick to current API.

[taoqf]

Hi, guys! Do we plan to support require.resolve('jsfile', {paths:'xxxx'});?