Support for UNIX sockets (#7392)

[vlasky]

@abernix et al, here's my enhancement to implement Support for UNIX sockets (#7392) in meteor.

If the PORT environment variable contains a UNIX file/path, this will be treated as the UNIX domain socket file to listen for incoming connections.

To summarise what I've done:

1. I added an additional regex check to the existing code in WebAppInternals.parsePort() to test if the PORT variable contains a UNIX file/path and if so, return it as a string.

2. In function main, we now pass an object called listenOptions to httpServer.listen() and this object contains the required options to configure httpServer to either listen on a TCP port or UNIX socket.

3. There is a sanity check that handles the situation where there is an existing an existing socket file. If there is, it will try to connect to it to confirm that it does not belong to a running server instance. If it does not, it presumes that the socket file is stale and deletes it.

I note that this support only works in production mode and not when using the meteor tool. I would have liked to add support to the meteor tool, however I see that the package node-http-proxy that it relies on does not appear to support forwarding connections to UNIX sockets, so that would have required a lot more work.

[hwillson]

Thanks for submitting this PR @vlasky! I've looked things over and added a few comments, but this will be reviewed more thoroughly very shortly. Thanks again!

[hwillson]

Could you update the existing parsePort tests to verify this additional parsing works properly?

[vlasky]

Done :-)

[hwillson]

Bit of a nit pick, but can you follow the existing opening/closing brace style that's used in the rest of the file (open brace on same line, etc.)?

[hwillson]

It looks like this httpServer.listen block of code is repeated a few lines down, and is the exact same. Could this be extracted into a common function to clean the code up a bit?

[abernix]

Thanks for submitting this PR! It would certainly be great to support Unix sockets.

[abernix]

I'm not convinced this new RegExp is much better than just checking to see if the path has a / in it at all (i.e. /\//, which is still not ideal. As it stands, this wouldn't allow the socket file to be in the current directory, something we should probably try to support via path.resolve, for example.

In addition to that, can we also could we consider another option for detecting if it's a TCP socket file? This should also probably check process.platform (to ensure it's win32) for the first part.

[abernix]

The addition of all this additional logic suggests that these should probably be come more modular functions instead of this rather-large else block.

[abernix]

What is this EADDRINUSE ➡️ ECONNREFUSED logic based on? I'm hesitant to have all this extra logic in here which would normally produce valid error messages anyhow. Can we really safely assume this is stale?

[abernix]

Why does a new socket'sECONNREFUSED error make it safe to cleanup the previous socket file?

[abernix]

Are we completely okay with the startHttpServer getting called again here as it will have already been called on line 859 below?

[vlasky]

@abernix In commit d9e7f86 I have rewritten the code to address the majority of your review comments.

1. The second regexp has been removed. The code now checks to see if the PORT value is a number or a string containing a windows named pipe (via the original regex), otherwise the value is presumed to be of the desired UNIX domain socketPath (and socket file).

2. The code no longer calls startHttpServer() twice. It now performs fs.existsSync(socketPath) to determine if there is an existing file with the same name as the desired socket file specified in PORT and then perform fs.statSync(socketPath).isSocket() to determine if that existing file is in fact a socket file (which is a special file) as opposed to an ordinary file (or any other type).

If that file is a socket file, it will then try to perform a dummy connection to it to determine if it's live - i.e. belonging to another running process which is listening for connections on it.

If the dummy connection attempt is successful, we know the socket file is live and being used by a running process, so we will not delete the file. Instead, we will log an error and exit.

If the connection attempt results in the error ECONNREFUSED, we know that the socket file is stale and we can safely delete it using fs.unlinkSync() and call startHttpServer() which will create a fresh socket file.

3. There is now cleanup code that deletes the socket file created by this process when the node process exits or is ended using SIGINT (Control-C). For those unfamiliar with UNIX socket files, it is the duty of the programmer to delete them up once the program no longer needs them - the OS does not remove them itself.

4. I removed parsePort() as it is no longer needed by the new logic. Also, it no will no longer convert invalid port values like "8080abc" to the numeric port value 8080 - they will instead be treated as a string.

I really think that old behaviour was wrong and served only to conceal mistakes made by those writing launcher scripts.

[abernix]

@vlasky Sorry I didn't get to review this today but it's first on my plate tomorrow! (And rest assured, Meteor 1.5.1 is not coming out tomorrow. 😉)

[abernix]

Keeping in mind that this is a major entry point into every application, and in consideration of my comments below, I think this still needs work before it's ready to merge.

Keep in mind that .listen creates the socket if it doesn't exist, and it also fails if the file does exist already with an error that can be captured. Also, calling .close on a socket does remove it if the sock file was created.

Aside from the deep nesting (of if > if > if > if > if), which could be broken out into more maintainable functions, I think there is still a safer approach here of just trying to startHttpServer and reacting to the error appropriately, possibly by fixing or checking something and calling back to startHttpServer again (with a couple retries max).

Thanks for taking a look at this. It won't be ready for 1.5.1, unfortunately. I do still think it's a very valuable change if you're willing to work through tidying it up.

[abernix]

I realize it's called WebAppInternals, and I think the impact is very low given the general uselessness of this parsePort function, but this is currently a public API and we should consider the repercussions of removing it or if it's worth keeping in place. (I suspect it was originally only to facilitate testing, but it could have been adopted.)

[abernix]

While I realize this isn't an exhaustive test by itself, it was testing something and I'm not comfortable removing the only test for this, changing this very core logic and not adding new tests (probably in tools/tests/). Preferably a few, especially since file-system operations and sockets can behave differently on different operating systems.

[vlasky]

@abernix, in my previous commit 693ac8a, I preserved the legacy parsePort() behaviour and tests, but you weren't happy because socket path would require an absolute path starting with a forward slash, i.e. you could not create a socket file in the current directory.

I note that I have seen other frameworks that use the presence of a forward slash to distinguish between listening on a UNIX socket and listening on a TCP port.

In order to satisfy your concern, I changed this behaviour in commit d9e7f86 so that a number would be treated as a TCP port and anything else would be treated as a UNIX socket, and this required removing the existing parsePort() functionality.

[abernix]

While it still works, fs.existsSync is technically deprecated (even in Node.js 4) so we should avoid introducing this.

[abernix]

Also, running fs.statSync on the next line is running the same check again but with a race-condition in between. Reacting to the error from statSync would be better than having two separate calls.

[sebakerckhof]

@abernix existsSync is not deprecated, exists is however:
https://nodejs.org/api/fs.html#fs_fs_existssync_path

Afaik, exists was deprecated because it didn't follow the (err, result) callback signature.

[abernix]

Ah, bizarre, but true! But still not necessary to use it here. :)

[vlasky]

That's right, existsSync() is not deprecated, however I take @abernix's point about replacing the separate calls to existsSync() and statSync() with a single call to statSync() to avoid a race condition.

[abernix]

It's typically discouraged to use the net.Socket.prototype.connect method directly, but rather to rely on net.createConnection. Something to consider?

[vlasky]

net.createConnection() is not suitable here because it is necessary to set the error handler in advance for this very short-lived client socket object whose sole purpose is to distinguish between a live UNIX socket and a stale UNIX socket file.

The documentation does not discourage its use, but says "use this only when implementing a custom socket" or in my view "use it when you have to".

[abernix]

I don't think we should be calling process.exit anywhere in this code. That will be handled by the caller to main() here in boot.js. I think you can just return ERROR or something (though you should leave the console.error.

[abernix]

This doesn't seem relevant to this change at all, what's the reason for this?

[vlasky]

Without this cleanup code, a stale UNIX socket file (.sock) file is left on disk if you terminate the node process with Control-C (SIGINT).

[hwillson]

Hi @vlasky - have you had a chance to look over the recent review comments?

[hwillson]

Hi @vlasky - are you still interested in working on this PR? We'd love to get these changes in place, after the recent review questions have been addressed. If you're not available to work on this let me know, and I'll jump in to help out. Thanks!

[vlasky]

@hwillson I have been away dealing with other business commitments recently.

My company has been using commit d9e7f86 in production without any issues.

The only change I plan to make to this PR for time being is the one mentioned in #8702 (comment) to resolve a race condition.

After that you (or anyone else) can jump in if you see a need.

[hwillson]

No problem at all @vlasky - thanks again for putting this together. After you add in changes to help with the race condition, I'll jump in to help to get this wrapped up / merged.

[vlasky]

@hwillson - the race condition fix has been implemented in 76808e8. Feel free to jump in.

[skirunman]

Just wondering if this is moving forward in 1.5.2 or 1.6 releases as currently tagged as 1.5.1.

[hwillson]

Good question @skirunman - I'm holding this one up unfortunately; hopefully I'll have the discussed changes wrapped up this week. We'll then be able to see which release it makes sense to add this to. Thanks!

[hwillson]

Hi all - I have a new version of this just about ready to go, but I noticed a rather large issue while testing.

First off, regarding net.Socket.prototype.connect versus net.createConnection, as long as the error handler callback is set immediately after the createConnection call, we should be okay. createConnection handles this by deferring the error event to the nextTick. That being said, unfortunately I don't think either approach will work as attempted.

Let's assume we're using createConnection (but this issue stands with net.Socket.prototype.connect). Using createConnection to check to see if we can connect to an existing socket file, then if we connect flag that socket file as already being used by another server, isn't a perfect check. If you SIGKILL the running node process, the socket file is left in place (not much we can do about this). Then if you use createConnection to connect to that file again, it will work. When that happens, we're flagging the socket file as being used by another server, when in reality it isn't.

Unfortunately, there isn't a perfectly reliable way to tell if a socket file is already being used by another process. That means we need to handle this check in another way. A few options:

1. (Least safe but automatic) Use net.createConnection as outlined, but if we can connect to the socket file, throw an exception, remove the existing socket file, and continue on. With this approach we don't care if the existing socket file is being used by another process or if it was left over from a SIGKILL. In either case we remove it, and continue to start the http server. This approach will work, but if another process really is using the socket file, and that socket file is removed while it's running, well, needless to say that wouldn't be good ...

2. (Most safe but manual) We don't try to connect to an existing socket file, ever. Instead, when firing up the webapp we first check to see if an existing socket file exists (like we're doing now), but if found we throw an exception describing the problem and the location of the existing socket file, and exit. This approach is nice and safe (as we're not killing a socket file that could be used by another process), but we're leaving the responsibility of cleaning up a potentially stale socket file to the developer deploying/starting the app. Requiring manual intervention like this could be risky, especially when the application is deployed to an environment that tries to automatically re-start crashed apps. It could be waiting for the stale socket file to be removed manually, and it could take some time before people notice.

As you can see, neither approach is perfect. I'm leaning towards the first approach and making sure people know that they need to define their socket paths very carefully, to make sure they aren't clobbering another process. This approach falls more in-line with other areas of Meteor, that make certain assumptions to keep the development/deployment process more automatic. I'd love to hear what others think, and if anyone has other suggestions? Thanks!

[hwillson]

Just to add - the existing code in this PR is already achieving #1 above, since the socketPath file is being removed during the specified process.on('exit', ... callback. Firing up a second server, getting the socket file already being used message, then watching my first server crash really threw me off during testing however. If we agree this is the way to go though, I'll adjust the error handling / messaging a bit to be more explicit about what's happening.

[zimme]

=== ?

[zimme]

@hwillson, dod you read this? https://stackoverflow.com/questions/16178239/gracefully-shutdown-unix-socket-server-on-nodejs-running-under-forever?answertab=active#tab-top

[hwillson]

Thanks @zimme - yes, I have seen that approach and have tried several variations. None of them are perfect unfortunately. If I SIGKILL the running node instance, the sock file is left in place, and I can still connect to it on retry. That means we never actually get a ECONNREFUSED when retrying a connection with a stale socket file, or when connecting using a socket file that's already in use - in both cases I can connect using that socket file, so there isn't an easy way to differentiate between the two scenarios.

[zimme]

I guess using ref/unref won't help either when SIGKILL is used?

[hwillson]

Hi all - based on my comments in #8702 (comment), I've adjusted this PR a bit. If an existing socket file is found when the application starts up, it is first removed before a new socket file is created when the HTTP server starts. As mentioned previously, this could be a bit dangerous if another application is configured to use the same socket file, on the same server. As long as people are configuring their applications properly, and making sure to use unique socket paths for different applications, this really shouldn't be an issue. The good news is by making this assumption up-front, we're able to simplify the codebase (and possible error paths).

With this new approach, some of the previous (outstanding) review comments are no longer valid. A quick synopsis:

• I've re-arranged things to help reduce the nested if levels.

• Regarding WebAppInternals.parsePort, since it was available as part of the previous public API, I've restored that functionality. It will now return a parsed numeric port if it can, otherwise it will return the full port string. This should provide the same functionality that was in place (I've restored the removed tests to verify this), and allow unix socket path's to be returned as well. That being said, since this function is no longer used internally, I've added a DEPRECATED label, and mentioned that using this function directly is not recommended.

• net.Socket.prototype.connect vs. net.createConnection no longer matters (we're not attempting to connect to an existing socket file - we're just removing it if it exists).

• process.on('exit', ... and process.on('SIGINT', ... are no longer being used. Since we're removing any previous socket file on startup no matter what, we don't need to worry about cleaning the socket file at the various possible program exit states.

• I've removed the use of process.exit and am instead throwing exceptions where necessary (since as mentioned process.exit is already being handled upstream in https://github.com/meteor/meteor/blob/devel/tools/static-assets/server/boot.js#L353).

I believe this addresses all outstanding comments. After these new changes have been reviewed, I'll put a History.md entry together. I'll also (shortly) post a link to the test application I've been using to verify these changes (Nginx + Meteor), along with testing steps.

Thanks!

[hwillson]

Adding a test app: https://github.com/hwillson/meteor-pr-8702

[zimme]

process.on('exit', ... and process.on('SIGINT', ... are no longer being used. Since we're removing any previous socket file on startup no matter what, we don't need to worry about cleaning the socket file at the various possible program exit states.

Shouldn't we try and do cleanup anyway? As we're creating the socket file we should be removing it when we do a graceful shutdown, even though we always remove it before starting again. But what if you don't start it again or change the name of the socket, and all of a sudden you have an unnecessary socket file.

[hwillson]

Good question @zimme - I had exit cleanup code in place just before I committed, but decided to take it out in the end to simplify things. My logic was that we're removing it again at startup, and if a socket file using version of the app is never started again, we've ended up leaving one 0 byte file on the filesystem. I also wasn't crazy about the idea of having additional exit handlers defined in the webapp package (when the process is managed via tools/static-assets/server/boot.js). Again though, I was on the fence on this one as well, so maybe I'll add it back in. Thanks for the feedback!

[zimme]

we've ended up leaving one 0 byte file on the filesystem

Still littering the filesystem, what if someone setup meteor to always generate a new socket file on boot with a script? Then you might end up with a lot of unused socket files.

[hwillson]

Haha - that would be horrible, but valid point. I'm re-visiting the cleanup code now. Thanks!

[hwillson]

Oh Circle - 4 failures when everything passes locally 🙁. Restarting, but the reported failures shouldn't be related to these changes. Also re-moving the WIP label from the title; this should be ready for a final review. I still have to add a History.md entry, but I'd like to make sure we're all on board with this new direction before finalizing. Thanks!

[abernix]

Just a few comments!

I think the logic chosen here is great and there doesn't seem to be any de-facto standard on it anyhow.

If anything, this new code (and the code that was there before) is more clean now than it was before, easy to iterate on and looks like a clean execution path in this (relatively critical) entry point. This was a relatively popular feature request, so hopefully we'll get helpful feedback quickly from those who use it and can iterate further if necessary.

[abernix]

Should we actually print a deprecation warning to the console, once per session? Or just add this to the History.md? Or do nothing? (It is called Internals, after all.)

[abernix]

This does appear to still be used from webapp_tests.js. Should we export this function with another internal name for that purpose?

[hwillson]

It's only used in webapp_tests.js to test itself 🙂. Regarding how to deprecate it, I think just mentioning it in the History.md should be enough (because as you said, Internals).

[abernix]

Should we explicitly state the .js extension? At least for consistency with other code?

[abernix]

Nit-⛏: Braces.

[hwillson]

Sure thing - I didn't really review that part of the code; I'll update that entire function a bit.

[abernix]

Not that we test on Windows, but this will fail on Windows. Could we use files.mkdtemp from tools/fs/files.js to create a directory for this file to live in?

[hwillson]

Ha - oops. Good point - changing, thanks!

[hwillson]

Right - I can't reference tools/fs/files.js directly from webapp; I'll use Node's os.tmpdir instead.

[hwillson]

Well, I was just prepping a History.md entry for this and was about to say we're good to go here, but then remembered we needed to add a mention that this was production only. The thing is, this doesn't need to be production only if we don't use the PORT environment variable to set the UNIX socket file path. If we use another unique environment variable like UNIX_SOCKET_PATH, then socket files can be used with the Meteor Tool and in production Node apps. Since hijacking PORT to store a pipe or socket path was a hack anyways, I've gone ahead and changed this around. UNIX domain sockets can now be used by setting something like UNIX_SOCKET_PATH=/tmp/somesock.sock. Windows named pipes are still set using PORT, but that's a change for another day/PR. When starting the Meteor Tool with the UNIX_SOCKET_PATH set, it will now show that a socket file is being used (instead of displaying the host:port combo). I've also finalized the History.md entry. Standing by for feedback - thanks!

[zimme]

Windows named pipes are still set using PORT, but that's a change for another day/PR.

Are we sure we don't want to add WINDOWS_NAMED_PIPE which would work the exact same way as UNIX_SOCKET_FILE?

[hwillson]

We could - I left those changes out since this PR should be good to go as is and meets the requirements of the original issue (#7392). Adding adjustments for Windows seemed to be a bit out of scope, and the current Windows named pipe hack has been in place for a while, so it's being used in production. Wiring up something like WINDOWS_NAMED_PIPE didn't seem to be as pressing since we'll still have to support the PORT hack for Windows anyways. Unless of course we deprecate the Windows PORT approach, but we'll definitely have to handle that situation carefully.

[aogaili]

Shouldn't host map to localIp?

[kentonv]

Hi all. This is largely our fault, not yours, but I wanted to note for informational purposes that this change played a part in a global outage of self-hosted Sandstorm.io servers. It turns out our startup code would often set a PORT variable with multiple ports, comma-separated, like "80,443". In the past, Meteor would use the first port, and other code in our server would parse out the remaining ports and do other things with them. This change implemented more strict parsing of ports, so broke that configuration. Unfortunately, our usual testing configurations did not happen to have multiple ports set, so the problem was not caught until real servers started dying after auto-updating.

Again, this is our fault for questionable code (we should never have relied on lenient PORT parsing) and insufficient testing, but I thought having visibility into this kind of issue would be informative.

Sandstorm issue and postmortem here: sandstorm-io/sandstorm#2993