Skip to content

Node

Jump to bottom
Sam Roberts edited this page Jun 3, 2020 · 42 revisions

https://nodesource.com/blog/node-js-performance-monitoring-part-3-debugging-the-event-loop

https://v8project.blogspot.ca/ https://github.com/v8/v8.dev/blob/45872b0dd2f59ce9ac2d0cf7bbbceb9fa452a87c/src/docs/embed.md

https://github.com/tomato42/tlsfuzzer

Code style

StrongLoop DevOps tooling

module search

IBM tooling

IBM NodeReport (FFDC for Node.js):

IBM Health Center

https://www.ibm.com/blogs/bluemix/2015/07/how-to-analyze-performance-in-bluemix/

https://developer.ibm.com/node/2016/09/27/advances-in-core-dump-debugging-for-node-js

https://scottfrees.com/ebooks/nodecpp/

https://community.risingstack.com/using-buffers-node-js-c-plus-plus

Eclipse addons

Not sure how to install these.

Google Chrome DevTools

WebStorm V8 Profiler (GUI for v8-profiler)

http

Maintenance

git clone git://github.com/iojs/io.js.git <--- guarantee that push is not possible

git clone iojs MERGE-iojs

git config --global --add core.whitespace fix

https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/ https://github.com/iojs/roadmap https://github.com/iojs/website https://github.com/iojs/build https://github.com/iojs/io.js

git shortlog -n -s

OS Primitives

Streams

streams, useful modules (careful attention to error handling):

  • pump... does it really handle errors?
  • https://github.com/maxogden/mississippi
  • duplexify
  • end-of-stream:
    • http request end/close: end won't happen if client errors (it will come as client_error on the server). close will always happen after socket closes... but there could still be buffered data, so data might keep coming in...
  • pump: important replacement for .pipe(), that avoids fd leaks: there is a case to be made that pipe shouldn't show up in production code
  • pumpify: pass an array of streams
  • readable_stream: always use this, not node core...
  • except don't use that...: just use streams that use readable_stream
  • streams adventure..., streams handbook... already updated to streams3... but if you are running 0.10 you need to use readable_stream to get streams3
  • what happens when an object stream goes into a fs.writestream...

Doc tools

Docs

Modules

Zones/CLS

uv/eventloop

Performance

XXX http://www.brendangregg.com/perf.html

Absorb or record Netflix talk:

So "netstat -tnlp" could be replaced with "ss -tnlp". lsof -i tcp -s tcp:listen More or less what you're looking for as well? Or without nss port name resolution or hostname resolution: lsof -nPi tcp -s tcp:listen

Mem leaks:

lsof -i tcp -s tcp:listen More or less what you're looking for as well? Or without nss port name resolution or hostname resolution: lsof -nPi tcp -s tcp:listen

Tips and Tricks

The "EPERM" error can result from two conditions:

  • The user genuinely has no access to the file that NPM is trying to manipulate. I think that is very unlikely in this case.

  • When npm tries to move or delete the file, it's open by another process. In most cases this happens because some background process temporarily opens the file for some reason - typically this would be a virus scanner, a synchronization service (e.g. dropbox) or an indexing service (e.g. the windows built-in one).

The rest applies to the second hypothesis:

  • NPM's solution to dealing with EPERM errors on windows is to retry a couple of times and then give up. It does that not in a very smart way (it retries essentially in a tight loop) which may cause starvation of the process that's holding the file open, especially on slower machines. A solution may be to set a timer and retry after some time has elapsed, as was proposed in https://github.com/npm/npm/issues/12059. Another solution might be to simply keep retrying for a bit longer.

  • The easiest way to test this is to install a separate copy of npm in the user's home dir, patch that up, and invoke this copy (node npm-cli.js) to install APIC.

  • In order to find out what process is interfering, I recomment using Process Monitor (https://technet.microsoft.com/en-us/sysinternals/processmonitor.aspx). Make it capture all file system activity (by all processes) while npm runs. After npm stops (fails), stop logging events, and then search list of events for the filename that caused the error (the one that appeared in the error message), and you should be able to see whether/which other processes were interfering.

abort on uncaught exception, a flag to do that exists

core files... gdb macros to dump js stack frames and objects?

https://github.com/trevnorris/tracemeur

http://www.joyent.com/blog/walmart-node-js-memory-leak

  • process._rawDebug(msg): direct fprintf, doesn't interact with libuv or the event loop, does not support a format string

  • node --expose-internals: allows require('internal/xxx') to access lib/internal/xxx.js

Clean close:

process._getActiveHandles()
process._getActiveRequests()

process._getActiveHandles().forEach(function(handle) {
  if (handle.unref) handle.unref()
})

    [2013-06-26 10:45:56 AM] Ben Noordhuis: that's for node handle/req wrap objects
    [2013-06-26 10:46:32 AM] Ben Noordhuis: in gdb, you can `call
    uv_print_all_handles(0)` and `call uv_print_active_handles(0)`
    [2013-06-26 10:47:36 AM] Ben Noordhuis: listing handles that 'become ready' is
    not really possible right now, there's not a single list of handles that are
    somehow primed
    [2013-06-26 10:48:02 AM] Ben Noordhuis: i've considered adding that but the
    current approach gives better performance because you can dispatch earlier
    [2013-06-26 10:49:20 AM] Ben Noordhuis: btw, the legenda for
    uv_print_{all,active}_handles is A=active, R=ref'd, I=internal

(node) warning: Recursive process.nextTick detected. This will break in the next version of node. Please use setImmediate for recursive deferral. RangeError: Maximum call stack size exceeded

The weird thing is, courtesy of grep I'm confident that we do not utilize process.nextTick anywhere; neither in our code nor in any modules we use.

Any advice on figuring this one out?

--trace-gc: https://github.com/joyent/node/issues/6258

You can start node with --trace-deprecation or --throw-deprecation to see where the warnings come from.

  • How to do above from inside mocha? Can be set in env?

https://github.com/iojs/io.js/pull/1077#discussion_r26780492

Forgot to mention, you can get a reasonable approximation of non-idle time by passing -j or --js to the tick processor. That filters out samples that aren't accountable to JS land.

https://gist.github.com/trevnorris/7712539

If you've installed node from nodejs.org installer, how do you get the d8 tick processor? do you have to download src, and build? if you build, won't the
addresses be different, so an existing v8.log won't be processable?
indeed, I'm not seeing it failing now too
octetcloud: npm install -g tick-processor
octetcloud: what indutny said
octetcloud: btw, try #nodejs channel for this ;)
octetcloud: you can build from deps/v8
tick-processor module will fail if running stuff on master
tjfontaine: master is not on nodejs.org
octetcloud: here are some steps: https://gist.github.com/trevnorris/7712539
octetcloud: alternatively you could use dtrace ;)
indutny: thanks, I'll try the npm install, just to see what it looks like for people who don't run from src
octetcloud: once spawnSync makes it into node we can easily shim the environment and make the tick processor run in node with proper symbol resolution, hmm?
what does spawnSync have to do with that?
tjfontaine: the tick processor script calls some d8-specific functions to spawn nm
oh that sort of stuff

tracing

Conversation was pretty high level, mostly gave a sense of what kinds of things people are interested in.

Reconstruction of async state:

  • async-listener is considered a good user-facing API

    We should have a stance on that. Is it good enough for us? Is it not? Is async-tracker better/worse, why? Can we build zones & strong-agent on top of async-listener? If not, why?

  • async_wrap is considered not good enough for async-listener

    Why? What are its issues? If it has them, we should comment on them.

Other:

https://github.com/thlorenz/cpuprofilify https://github.com/thlorenz/flamegraph https://www.npmjs.com/package/transaction-tracer https://www.npmjs.com/package/async-listener https://github.com/Raynos/leaked-handles https://github.com/thlorenz/traceviewify#screenshots

https://github.com/mafintosh/why-is-node-running

  var dump = {
    handles: function() {
      var fmt = require('util').format;
      var inspect = require('util').inspect;
      var handles = process._getActiveHandles();

      console.log('ActiveHandles:');
      handles.forEach(function(h, i) {
        console.log('%d: %s', i, pretty(h));
      });

      function pretty(h) {
        if (h === process.stderr)
          return fmt('process.stderr/type=%s/fd=%j', h._type, h.fd);
        if (h === process.stdout)
          return fmt('process.stdout/type=%s/fd=%j', h._type, h.fd);
        if (h === process.stdin)
          return fmt('process.stdin/type=%s/fd=%j', h._type, h.fd);
        if ('_idleNext' in h)
          return fmt('timer:msecs=%s:count=%d', h.msecs, timerCount(h));
        if ('_connections' in h)
          return fmt('server:%j', h.address());
        return inspect(h, {depth: 1});
      }

      function timerCount(h) {
        // XXX the circular list always has one extra entry for each setTimeout
        // call, why?
        var count = -1;
        var first = h._idleNext;
        var next = first;
        while (next) {
          next = next._idleNext;
          count++;
          if (next === first)
            break;
        }
        return count;
      }
    },
    requests: function() {
      console.log('ActiveRequests:', process._getActiveRequests());
    },
  };
  dump.handles();
  dump.requests();

@fadi could you point to the js api used in chrome to get events? and is it targeted to go in v8, or just the trace_event.h c++ API?

https://codereview.chromium.org/827993003/patch/20001/30008

@sam, in the doc there is another CL for chrome The original doc shared by nduca

https://github.com/johnmccutchan/adb_trace

https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit

https://codereview.chromium.org/707273005/

@fadi: in above I don't see any api, wherein in javascript, you can recover the trace events that have been added fair enough, so it could be first consumer to register wins

https://github.com/chrisa/node-dtrace-provider/blob/master/dtrace_provider.cc#L116

from nduca: https://code.google.com/p/chromium/codesearch#chromium/src/base/trace_event/trace_event_impl.cc&q=trace_event_impl&sq=package:chromium&l=1995

trace_event.h

event_tracing.cc

what version of v8 is a stable version of the trace_event system targeted for? not sure at the moment

https://github.com/iojs/io.js/issues/877

https://docs.google.com/a/strongloop.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU

https://codereview.chromium.org/827993003

https://code.google.com/p/chromium/codesearch#chromium/src/base/trace_event/&q=trace_event&sq=package:chromium&type=cs

https://docs.google.com/a/strongloop.com/document/d/1l0N1B4L4D94andL1BY39Rs_yXU8ktJBrKbt9EvOb-Oc/edit#heading=h.3hymw1suafwo

https://github.com/iojs/io.js/issues/671#issuecomment-74737098

https://docs.google.com/a/strongloop.com/document/d/1_ApOMt03xHVkaGpTEPMDIrtkjXOzg3Hh4ZcyfhvMHx4

Clone this wiki locally