Replace ws with uws #2807

mikoim · 2017-05-05T08:20:46Z

This PR may make streaming server faster over 10x. At least it relieve a performance bottleneck of ws.

uws: https://github.com/uWebSockets/uWebSockets
benchmark results: https://github.com/uWebSockets/uWebSockets/blob/master/benchmarks/benchmark_results_scalability

Wonderfall · 2017-05-05T12:17:29Z

Tried with no success. Any instruction?

mikoim · 2017-05-05T12:59:39Z

@Wonderfall It needs no additional operations. Could you upload logs with errors?

Gargron · 2017-05-05T15:38:00Z

I am in favour of this PR but will wait for @Wonderfall's logs before merging.

Wonderfall · 2017-05-05T15:49:18Z

> mastodon@ start /mastodon
> babel-node ./streaming/index.js --presets es2015,stage-2

/mastodon/node_modules/uws/uws.js:38
            throw new Error('Compilation of µWebSockets has failed and there is no pre-compiled binary ' +
            ^

Error: Compilation of µWebSockets has failed and there is no pre-compiled binary available for your system. Please install a supported C++11 compiler and reinstall the module 'uws'.
    at e (/mastodon/node_modules/uws/uws.js:38:19)
    at Object.<anonymous> (/mastodon/node_modules/uws/uws.js:42:3)
    at Module._compile (module.js:570:32)
    at Module._extensions..js (module.js:579:10)
    at Object.require.extensions.(anonymous function) [as .js] (/mastodon/node_modules/babel-register/lib/node.js:152:7)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)

npm ERR! Linux 4.10.1
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "start"
npm ERR! node v6.10.1
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! mastodon@ start: `babel-node ./streaming/index.js --presets es2015,stage-2`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the mastodon@ start script 'babel-node ./streaming/index.js --presets es2015,stage-2'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the mastodon package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     babel-node ./streaming/index.js --presets es2015,stage-2
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs mastodon
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls mastodon
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /mastodon/npm-debug.log

Gargron · 2017-05-05T15:50:52Z

@Wonderfall This could be because your BUILD_DEPS are deleted by the time yarn install happens, maybe? Something along those lines.

Wonderfall · 2017-05-05T15:51:51Z

I use the latest Dockerfile, so BUILD_DEPS packages are not removed. I'm looking into it.

Gargron · 2017-05-05T15:52:51Z

Oh damn, maybe it's incompatible with Alpine?

Wonderfall · 2017-05-05T15:53:23Z

Alpine 3.4 in particular, maybe. I'll try to use some newer packages

Wonderfall · 2017-05-05T16:05:52Z

Tried with gcc 6 but I ran into the same issue as #2804

mikoim · 2017-05-05T16:06:43Z

Build successful.
https://gist.github.com/mikoim/cc40652e31d247f9e2d5bc187b3196e2

Try docker-compose build --no-cache or docker build --no-cache to build images without cache.

Wonderfall · 2017-05-05T16:07:35Z

Building the image is not the problem here, but running it (with gcc 5) fails.

mikoim · 2017-05-05T16:11:33Z

Oops sorry. It seems to be working... why?

> docker-compose up streaming
mastodon_redis_1 is up-to-date
mastodon_db_1 is up-to-date
Recreating mastodon_streaming_1
Attaching to mastodon_streaming_1
streaming_1  | 
streaming_1  | > mastodon@ start /mastodon
streaming_1  | > babel-node ./streaming/index.js --presets es2015,stage-2
streaming_1  | 
streaming_1  | info Starting streaming API server master with 1 workers 
streaming_1  | info Starting streaming API server worker on :::4000

Wonderfall · 2017-05-05T16:21:21Z

I don't know, but I also have an error in node_modules/uws/build_log.txt :

gyp info it worked if it ends with ok
gyp info using node-gyp@3.6.0
gyp info using node@6.10.1 | linux | x64
gyp http GET https://nodejs.org/download/release/v6.10.1/node-v6.10.1-headers.tar.gz
gyp http 200 https://nodejs.org/download/release/v6.10.1/node-v6.10.1-headers.tar.gz
gyp http GET https://nodejs.org/download/release/v6.10.1/SHASUMS256.txt
gyp http 200 https://nodejs.org/download/release/v6.10.1/SHASUMS256.txt
gyp info spawn /usr/bin/python2
gyp info spawn args [ '/usr/lib/node_modules/yarn/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/mastodon/node_modules/uws/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/usr/lib/node_modules/yarn/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/root/.node-gyp/6.10.1/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/root/.node-gyp/6.10.1',
gyp info spawn args   '-Dnode_gyp_dir=/usr/lib/node_modules/yarn/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=node.lib',
gyp info spawn args   '-Dmodule_root_dir=/mastodon/node_modules/uws',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory '/mastodon/node_modules/uws/build'
  CXX(target) Release/obj.target/uws/src/Extensions.o
  CXX(target) Release/obj.target/uws/src/Group.o
In file included from /root/.node-gyp/6.10.1/include/node/openssl/asn1.h:63:0,
                 from /root/.node-gyp/6.10.1/include/node/openssl/objects.h:965,
                 from /root/.node-gyp/6.10.1/include/node/openssl/evp.h:94,
                 from /root/.node-gyp/6.10.1/include/node/openssl/x509.h:73,
                 from /root/.node-gyp/6.10.1/include/node/openssl/ssl.h:156,
                 from ../src/Networking.h:70,
                 from ../src/WebSocketProtocol.h:5,
                 from ../src/WebSocket.h:4,
                 from ../src/Group.h:4,
                 from ../src/Group.cpp:1:
/root/.node-gyp/6.10.1/include/node/openssl/e_os2.h:2:1: error: unterminated comment
 /* ====================================================================
 ^
In file included from /root/.node-gyp/6.10.1/include/node/openssl/x509.h:83:0,
                 from /root/.node-gyp/6.10.1/include/node/openssl/ssl.h:156,
                 from ../src/Networking.h:70,
                 from ../src/WebSocketProtocol.h:5,
                 from ../src/WebSocket.h:4,
                 from ../src/Group.h:4,
                 from ../src/Group.cpp:1:
/root/.node-gyp/6.10.1/include/node/openssl/ec.h:9:1: error: unterminated comment
 /* ====================================================================
 ^
In file included from /root/.node-gyp/6.10.1/include/node/openssl/ecdsa.h:68:0,
                 from /root/.node-gyp/6.10.1/include/node/openssl/x509.h:87,
                 from /root/.node-gyp/6.10.1/include/node/openssl/ssl.h:156,
                 from ../src/Networking.h:70,
                 from ../src/WebSocketProtocol.h:5,
                 from ../src/WebSocket.h:4,
                 from ../src/Group.h:4,
                 from ../src/Group.cpp:1:
/root/.node-gyp/6.10.1/include/node/openssl/ec.h:76:0: error: unterminated #ifndef
In file included from /root/.node-gyp/6.10.1/include/node/openssl/ecdh.h:78:0,
                 from /root/.node-gyp/6.10.1/include/node/openssl/x509.h:91,
                 from /root/.node-gyp/6.10.1/include/node/openssl/ssl.h:156,
                 from ../src/Networking.h:70,
                 from ../src/WebSocketProtocol.h:5,
                 from ../src/WebSocket.h:4,
                 from ../src/Group.h:4,
                 from ../src/Group.cpp:1:
/root/.node-gyp/6.10.1/include/node/openssl/ec.h:76:0: error: unterminated #ifndef
In file included from /root/.node-gyp/6.10.1/include/node/openssl/x509.h:87:0,
                 from /root/.node-gyp/6.10.1/include/node/openssl/ssl.h:156,
                 from ../src/Networking.h:70,
                 from ../src/WebSocketProtocol.h:5,
                 from ../src/WebSocket.h:4,
                 from ../src/Group.h:4,
                 from ../src/Group.cpp:1:
/root/.node-gyp/6.10.1/include/node/openssl/ecdsa.h:75:1: error: expected initializer before 'extern'
 extern "C" {
 ^
In file included from /usr/include/c++/5.3.0/new:40:0,
                 from /usr/include/c++/5.3.0/ext/new_allocator.h:33,
                 from /usr/include/c++/5.3.0/x86_64-alpine-linux-musl/bits/c++allocator.h:33,
                 from /usr/include/c++/5.3.0/bits/allocator.h:46,
                 from /usr/include/c++/5.3.0/vector:61,
                 from ../src/Networking.h:72,
                 from ../src/WebSocketProtocol.h:5,
                 from ../src/WebSocket.h:4,
                 from ../src/Group.h:4,
                 from ../src/Group.cpp:1:
/usr/include/c++/5.3.0/exception:35:37: error: expected '}' before end of line
/usr/include/c++/5.3.0/exception:35:37: error: expected declaration before end of line
uws.target.mk:97: recipe for target 'Release/obj.target/uws/src/Group.o' failed
make: *** [Release/obj.target/uws/src/Group.o] Error 1
make: Leaving directory '/mastodon/node_modules/uws/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/yarn/node_modules/node-gyp/lib/build.js:285:23)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)
gyp ERR! System Linux 4.10.1
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/yarn/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /mastodon/node_modules/uws
gyp ERR! node -v v6.10.1
gyp ERR! node-gyp -v v3.6.0
gyp ERR! not ok

Wonderfall · 2017-05-05T17:23:06Z

So I built a new image based on xataz/node:7 (node:7-alpine also works) (which provides the latest node version on Alpine Linux 3.5) and I have no problem. But this is a specific Dockerfile for the streaming container...

I made more tests and I think upgrading to Alpine 3.5 is a solution. But the official ruby image is not updated, and Puma won't work with LibreSSL. You see how it becomes...

Wonderfall · 2017-05-06T13:09:45Z

Fuck. It seems to work now, and I don't know why. I was 100% convinced I built the image without the cache. No matter, I think it can be merged!

…upstream Merge upstream changes up to 670e465

mikoim added 2 commits May 5, 2017 17:01

Replace ws with uws

0fef799

fix indent

fd4a5a8

Adjust indentation

8dbb041

Gargron approved these changes May 5, 2017

View reviewed changes

remove trailing space

ff328bd

Gargron merged commit 67b7d3d into mastodon:master May 6, 2017

Gargron mentioned this pull request Jan 5, 2019

Migrate from uws to ws #9505

Closed

ClearlyClaire mentioned this pull request Mar 20, 2021

Migrate from discontinued cws to uWebSockets.js or similar #15184

Closed

ClearlyClaire added a commit to ClearlyClaire/mastodon that referenced this pull request Aug 8, 2024

Merge pull request mastodon#2807 from ClearlyClaire/glitch-soc/merge-…

e4e1173

…upstream Merge upstream changes up to 670e465

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace ws with uws #2807

Replace ws with uws #2807

mikoim commented May 5, 2017

Wonderfall commented May 5, 2017

mikoim commented May 5, 2017

Gargron commented May 5, 2017

Wonderfall commented May 5, 2017

Gargron commented May 5, 2017

Wonderfall commented May 5, 2017

Gargron commented May 5, 2017

Wonderfall commented May 5, 2017

Wonderfall commented May 5, 2017

mikoim commented May 5, 2017 •

edited

Loading

Wonderfall commented May 5, 2017

mikoim commented May 5, 2017 •

edited

Loading

Wonderfall commented May 5, 2017 •

edited

Loading

Wonderfall commented May 5, 2017 •

edited

Loading

Wonderfall commented May 6, 2017

Replace ws with uws #2807

Replace ws with uws #2807

Conversation

mikoim commented May 5, 2017

Wonderfall commented May 5, 2017

mikoim commented May 5, 2017

Gargron commented May 5, 2017

Wonderfall commented May 5, 2017

Gargron commented May 5, 2017

Wonderfall commented May 5, 2017

Gargron commented May 5, 2017

Wonderfall commented May 5, 2017

Wonderfall commented May 5, 2017

mikoim commented May 5, 2017 • edited Loading

Wonderfall commented May 5, 2017

mikoim commented May 5, 2017 • edited Loading

Wonderfall commented May 5, 2017 • edited Loading

Wonderfall commented May 5, 2017 • edited Loading

Wonderfall commented May 6, 2017

mikoim commented May 5, 2017 •

edited

Loading

mikoim commented May 5, 2017 •

edited

Loading

Wonderfall commented May 5, 2017 •

edited

Loading

Wonderfall commented May 5, 2017 •

edited

Loading