Looking to build a plugin offering APM support

[tlhunter]

I'm working on adding bundler support for the Datadog Node.js APM library. Currently, from my research, none of the popular APM tools fully support bundlers (like esbuild, webpack, etc.) Some of them partially support bundlers, à la adding instrumented modules to the externals list in webpack, but that results in modules being left outside of the bundle and isn't ideal for the end developer. My goal with this issue is to pave a way for Datadog and other APM libraries to support a bundler.

At a high level the way that Datadog and other APM tools work is that they're delivered as an npm package and contain a list of supported third party packages and version ranges, like pg v3-v4 or redis v5.2-v6.3. Node.js's require function is replaced at runtime in some manner so that when a module in node_modules/ is loaded the name and version are compared with this list. The require-in-the-middle npm package illustrates this. If there's a match then some magic happens, like having package methods get wrapped in a function to track the timing and query information.

When it comes to building Node.js apps, esbuild concatenate userspace modules into a bundle, basically a single module JavaScript file, and calls to require for userspace modules are instead replaced with a function to lookup the bundled version of the module. Internal Node.js modules like http can be exposed using the original Node.js require call. At this point most APM tools are able to wrap the internal Node.js modules but not third party node_modules/.

Basic esbuild output might look something like this:

var __getOwnPropNames = Object.getOwnPropertyNames;
var __commonJS = (cb, mod) => function __require() {
  return mod || (0, cb[__getOwnPropNames(cb)[0]])((mod = { exports: {} }).exports, mod), mod.exports;
};

// foo.js
var require_foo = __commonJS({
  "foo.js"(exports, module2) {
    var fs = require("fs");
    module2.exports = "hello";
  }
});

// app.js
var foo = require_foo();
console.log(foo);

At this point I have an idea for to implement such an APM plugin. First, the plugin would need to ship with the list of packages / versions to instrument. At build time the plugin would maintain a map of module locations and versions. It would need to hook into the events for each module that gets loaded. If the name matches then read the appropriate package.json file on disk to check the version. If that happens, then add the module and version information to the map. Once the modules have all been read the map would then be injected into the resulting bundle somehow. The APM library itself would be a dependency of the project and would be included as well. The APM library would then read the injected module / version map somehow. Finally, the APM library would still wrap the built-in require function but would also wrap the __commonJS() function provided by esbuild.

With this in mind I'm having trouble finding the appropriate plugin APIs to support this. Would anyone be able to tell me if this plan is currently implementable and which APIs correlate to these steps? Also, if you specifically know that any of these operations aren't supported, please let me know as well so that a PR could be made.

[evanw]

If you want to change what module a given import path resolves to, you want to use a plugin with an onResolve callback. So something like this:

let examplePlugin = {
  name: 'example',
  setup(build) {
    build.onResolve({ filter: /.*/ }, args => {
      if (args.path === 'pg') {
        return { path: require.resolve('alt-pg') }
      }
    })
  },
}

This swaps the pg module for the alt-pg module. If you want to swap a module out for a virtual module that's generated at compile time and doesn't exist on the file system, then it's a little more complicated and needs an onLoad callback as well as a namespace (to tell esbuild that it's not on the file system). Something like this:

let examplePlugin = {
  name: 'example',
  setup(build) {
    build.onResolve({ filter: /.*/ }, args => {
      if (args.path === 'pg') {
        return { path: 'pg', namespace: 'my-ns' }
      }
    })

    build.onLoad({ filter: /.*/, namespace: 'my-ns' }, args => {
      let version = getVersionOf(args.path)
      let contents = `
        import { addToMap, wrapSomehow } from 'my-helpers'
        addToMap(${JSON.stringify(args.path)}, ${JSON.stringify(version)})
        let mod = require(${JSON.stringify(args.path)})
        module.exports = wrapSomehow(mod)
      `
      return { contents}
    })
  },
}

This should in theory (I didn't test it) replace the pg module with a virtual module that looks like this:

import { addToMap, wrapSomehow } from 'my-helpers'
addToMap("pg", "0.0.1")
let mod = require("pg")
module.exports = wrapSomehow(mod)

Information about how to do all of this should already be in the documentation: https://esbuild.github.io/plugins/. For example, the WebAssembly sample plugin already demonstrates how to wrap modules: https://esbuild.github.io/plugins/#webassembly-plugin.

[evanw]

Closing this issue due to age, and because this question was answered.