Feature Request: Webpack Support #827

mruoss · 2020-01-16T09:29:16Z

Hi there. I set up dd-trace according to the setup instructions and find that modules like http or express are not instrumented. I tried to find out the reason, digging in some of the dd-trace code and finding that e.g. express is not in the require.cache because it only contains numeric values. Although I did not find anything in the instructions about it, I set the webpack optimization moduleIds to named. Now I see node module paths in require.cache but still express is not there.

Is this a known problem? Am I doing somehting wrong here?

Thanks,
Michael

Environment

Operation system: Mac OSX
Node version: 12.13.0
Tracer version: 0.16.3

The text was updated successfully, but these errors were encountered:

mruoss · 2020-01-16T12:55:47Z

Alright, by now I figured out I had to externalize the node_modules using webpack-node-externals. Is this a required step in the setup? If it is, it should probably be mentioned in the setup instructions, no?

rochdev · 2020-01-16T21:09:47Z

We could indeed recommend this module when using Webpack. It's possible that it won't be possible to use it in some setups, but I think it would work for most cases.

In all honesty, I didn't know about this module which is the why it's not currently documented, so thanks for the tip!

armenr · 2020-01-17T15:29:30Z

I'm not sure whether we're getting bitten by something similar - kinda a newb here - but we use babel to transpile our ES6 before shipping it. Would this be affecting us too - I realize, naturally, that we're not talking about WebPack, but I figure it can't hurt to ask.

mruoss · 2020-01-17T17:37:05Z

Hi @armenr
In our case the application and the node modules are bundled into a single JS file. The problem is that somehow the "require-in-the-middle" module which is used by dd-trace-js doesn't trigger if the node modules are bundled.

If you're just using babel to transpile ES6 to JS and are not bundling, you probably get bitten by something else...

armenr · 2020-01-18T19:21:35Z

@mruoss - I've used PKG, NCC, and NEXE - and have been bitten in ALL three cases by EXACTLY what you've been bitten by. I moved our entire setup (dev & prod) to Docker, JUST to get a "vanilla" set up, as an attempt to benefit from auto-instrumentation...but it doesn't look like it's instrumented fully.

"If you're just using babel to transpile ES6 to JS and are not bundling, you probably get bitten by something else..."

:) - How can I go about finding out?

mruoss · 2020-01-20T07:56:26Z

:) - How can I go about finding out?

If you have no node_modules folder inside your docker image, this is your issue.

mruoss · 2020-01-20T14:04:18Z

Alright, see the referenced issue above. Bundling node modules is not supported by the require-in-the-middle module. As mentioned before, it might be helpful to mention this also here in the setup instructions somewhere.

rochdev · 2020-01-20T18:46:14Z

@armenr Simply using Babel should not cause any problems, although you have to make sure to avoid hoisting, similar to what we recommend for TypeScript in the setup docs, otherwise auto-instrumentation will not work properly.

armenr · 2020-01-21T09:07:44Z

Thanks for the quick and helpful response, @rochdev! :)

We utilize an entrypoint file to fire up our NodeJS APIs in the following pattern/manner:

/* eslint-disable global-require */
const { join } = require('path');
require('dotenv').config({ path: join(__dirname, '.env') });

if (process.env.NODE_ENV === 'production') {
  require('./dist/connectors/connectTracer.js');
  require('./dist/server.js');
} else {
  require('./src/connectors/connectTracer.js');
  require('./src/server.js');
}

Is this sufficient/correct?

rochdev · 2020-01-22T02:03:33Z

@armenr Assuming that the tracer is initialized synchronously in connectTracer.js, then it looks correct. Which auto-instrumentation seems to be missing?

armenr · 2020-01-22T14:51:22Z

@rochdev --

synchronously, yes. --> see below

const tracer = require('dd-trace');

const {
  DD_ENV,
  DD_TRACE_DEBUG,
  DD_SERVICE_NAME,
  DD_TRACE_ENABLED,
  DD_LOGS_INJECTION,
  DD_PLUGINS_ENABLED,
  DD_RUNTIME_METRICS_ENABLED,
} = process.env;

module.exports = tracer.init({
  analytics: true,
  debug: DD_TRACE_DEBUG,
  enabled: DD_TRACE_ENABLED,
  env: DD_ENV,
  logInjection: DD_LOGS_INJECTION,
  plugins: DD_PLUGINS_ENABLED,
  runtimeMetrics: DD_RUNTIME_METRICS_ENABLED,
  service: DD_SERVICE_NAME,
});

williancolognesitrimble · 2020-06-09T13:02:30Z

Alright, by now I figured out I had to externalize the node_modules using webpack-node-externals. Is this a required step in the setup? If it is, it should probably be mentioned in the setup instructions, no?

Hey @mruoss how could you fix that to use dd-trace in your webpack application? I've been falling in the same problem here a few days...

mruoss · 2020-06-09T13:06:06Z

@williancolognesitrimble

my webpack config currently looks like this:

import nodeExternals from 'webpack-node-externals'


const config = {
  	entry: '...',
  	externals: [nodeExternals()],

        ... other_configuration
}

maschmann · 2020-07-09T12:12:37Z

I've also encountered this problem with node10 and dd-trace while trying to bundle with webpack:

ERROR in ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js
Module not found: Error: Can't resolve 'graphql/language/printer' in '/home/maschmann/git<package>/node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools'
 @ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js 10:18-53
 @ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/signature.js
 @ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/index.js
 @ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/index.js
 @ ./node_modules/dd-trace/packages/dd-trace/src/plugins/index.js
 @ ./node_modules/dd-trace/packages/dd-trace/src/platform/node/index.js
 @ ./node_modules/dd-trace/packages/dd-trace/index.js
 @ ./node_modules/dd-trace/index.js
 @ ./app.js
 @ multi ./app.jsERROR in ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js
Module not found: Error: Can't resolve 'graphql/language/visitor' in '/home/maschmann/git/<package>/node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools'
 @ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js 9:18-53
 @ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/signature.js
 @ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/index.js
 @ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/index.js
 @ ./node_modules/dd-trace/packages/dd-trace/src/plugins/index.js
 @ ./node_modules/dd-trace/packages/dd-trace/src/platform/node/index.js
 @ ./node_modules/dd-trace/packages/dd-trace/index.js
 @ ./node_modules/dd-trace/index.js
 @ ./app.js
 @ multi ./app.jsERROR in ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js
Module not found: Error: Can't resolve 'graphql/utilities' in '/home/maschmann/git/<package>/node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools'
 @ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js 11:20-48
 @ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/signature.js
 @ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/index.js
 @ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/index.js
 @ ./node_modules/dd-trace/packages/dd-trace/src/plugins/index.js
 @ ./node_modules/dd-trace/packages/dd-trace/src/platform/node/index.js
 @ ./node_modules/dd-trace/packages/dd-trace/index.js
 @ ./node_modules/dd-trace/index.js
 @ ./app.js

kalebdf · 2020-07-09T18:51:54Z

We also ran into this issue adding dd-trace to our Angular app (Angular v9.1 universal using webpack). It looks like it could be a peer dependency issue on the plugins. We are going to try your approach @mruoss as a temporary workaround.

Let us know if you need more information @rochdev.

ERROR in ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js
Module not found: Error: Can't resolve 'graphql/language/printer' in '/Users/kfulgham/dev/flo/flosports-webapp/node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools'

ERROR in ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js
Module not found: Error: Can't resolve 'graphql/language/visitor' in '/Users/kfulgham/dev/flo/flosports-webapp/node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools'

ERROR in ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js
Module not found: Error: Can't resolve 'graphql/utilities' in '/Users/kfulgham/dev/flo/flosports-webapp/node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools'

ERROR in ./node_modules/spdx-expression-parse/scan.js
Module not found: Error: Can't resolve 'spdx-exceptions' in '/Users/kfulgham/dev/flo/flosports-webapp/node_modules/spdx-expression-parse'

ERROR in ./node_modules/spdx-correct/index.js
Module not found: Error: Can't resolve 'spdx-license-ids' in '/Users/kfulgham/dev/flo/flosports-webapp/node_modules/spdx-correct'

ERROR in ./node_modules/spdx-expression-parse/scan.js
Module not found: Error: Can't resolve 'spdx-license-ids' in '/Users/kfulgham/dev/flo/flosports-webapp/node_modules/spdx-expression-parse'

ERROR in ./node_modules/spdx-expression-parse/scan.js
Module not found: Error: Can't resolve 'spdx-license-ids/deprecated' in '/Users/kfulgham/dev/flo/flosports-webapp/node_modules/spdx-expression-parse'

kalebdf · 2020-07-10T20:50:08Z

Until a fix is made, our team found a temporary workaround for Angular 9 CLI & webpack when using Angular Universal (server-side rendering).

Created an externals reference:

// webpack.config.js
module.exports = {
  externals: [/^dd-trace/]
}

And added a customWebpackConfig property to to our angular.json in projects.our-project-name.architect.server

{
	"builder": "@angular-builders/custom-webpack:server",
	"options": {
		"customWebpackConfig": {
			"path": "./webpack.config.js"
		},
		"outputPath": "dist/flosports-webapp/server",
		"main": "server.ts",
		"tsConfig": "tsconfig.server.json",
		"externalDependencies": ["@firebase/firestore"]
	}
        //...
}

ashatch · 2020-07-14T16:45:44Z

We're running into the same issue as @kalebdf and @maschmann - webpack is failing for our server-side project. We're following the guidance here https://docs.datadoghq.com/tracing/setup/nodejs/

rochdev · 2020-07-15T16:22:17Z

For graphql, it needs to be set as an external in the Webpack config since it's only required when the plugin is loaded, which is not the case when graphql is not used. This is because Webpack doesn't work with conditional requires which are used in that plugin.

For spdx-license-ids, it's a transitive dependency from one of our dependencies (read-pkg) and it should be included by Webpack, not sure why this isn't the case, I'll have to look into it.

amitizle · 2020-10-26T17:15:43Z

Thanks for responding and for this package! I'll be blunt and ask, any update @rochdev ?

rochdev · 2020-10-26T18:11:17Z

@amitizle Are you referring to the issue with spdx-license-ids? Is this still an issue?

amitizle · 2020-10-26T20:39:09Z

@rochdev yes it is.

rochdev · 2020-10-26T21:16:01Z

@amitizle Ok, for some reason I assumed it was fixed. I'll look into it this week.

rochdev · 2020-10-26T21:31:48Z

@amitizle I tried to take a quick look at the dependency tree but nothing stood out. I'm not super well versed Webpack wise, could you create a small replication snippet I could use as the basis of the investigation?

amitizle · 2020-10-27T06:56:11Z

@rochdev me neither 😄
Let me try and write an entire setup that is similar to ours, I'll post in here with an update. Thanks a lot for the quick responses.

samsaggace · 2020-12-01T11:09:05Z

I think I found the issue with the spdx-* dependencies :
it's because these packages only contain index.json files and this extension was not listed in my webpack.config.js.
I solved my problem with :

module.exports = {
  resolve: {
    extensions: ['.js', '.jsx', '.ts', '.tsx', '.json'],
  {...}
}

Hope it solves the problem for everyone!

kty1965 · 2020-12-15T04:14:51Z

I am using nestjs monorepo webpack build.
works for me

const mergedOptions = merge(options, {
    resolve: {
      extensions: ['.js', '.ts', '.json'],
    },
    plugins: [
      new webpack.IgnorePlugin({
        checkResource(resource) {
          const lazyImports = [
            //...
            'graphql/language/printer',
            'graphql/language/visitor',
            'graphql/utilities',
            'spdx-exceptions', 
            'spdx-license-ids',
            'spdx-license-ids/deprecated',
          ];
          if (!lazyImports.includes(resource)) {
            return false;
          }
          try {
            require.resolve(resource, {
              paths: [process.cwd()],
            });
          } catch (err) {
            return true;
          }
          return false;
        },
      }),
    ],
  });

rochdev · 2021-05-18T23:04:36Z

We've removed the dependency on read-pkg so the issue with spdx-license-ids should not longer happen.

SasSam · 2021-08-18T15:38:40Z

We had the same issue with dd-trace like @maschmann had. The solution was that we had to set externals for those graphql packages within the webpack config file.

    externals: [
        'graphql/language/visitor',
        'graphql/language/printer',
        'graphql/utilities'
    ],

This resolved the issue for us.

vanseverk · 2022-05-10T15:26:31Z

Hi team, we are having a simlilar issue with dd-trace for a large number of Lambda's which are running on NodeJS. They are created using Yarn 2 and webpack to a single index.js file. Because of the treeshaking and nature of the Yarn PnP we don't have any node_modules, so using webpack-node-externals isn't an option either - as we don't have any node_modules to copy in the zip we place on AWS Lambda.

In case a fix in the library or its dependencies isn't possible, is there any alternative workaround in existence?

rochdev · 2022-06-02T15:39:17Z

@vanseverk There was a mention in another issue of webpack-pnp-externals, I would give that a try for now. We do plan to support Webpack and PnP officially eventually but not in the short term.

tlhunter · 2023-12-18T21:51:59Z

Note that if bundling is important for your application that we do now have official esbuild support.

mruoss added the bug Something isn't working label Jan 16, 2020

rochdev added community question Further information is requested and removed bug Something isn't working labels Jan 16, 2020

mruoss mentioned this issue Jan 20, 2020

Use this module with bundled node modules (using webpack) elastic/require-in-the-middle#35

Closed

bengl added the webpack label May 18, 2021

rochdev mentioned this issue Sep 30, 2022

Can't resolve 'graphql/language/visitor' when bundling using webpack for node target #2413

Closed

rochdev mentioned this issue Oct 15, 2022

Impossible to bundle with Rollup: GraphQL should not be a devDependency #1467

Open

tlhunter added bundler and removed webpack labels Nov 30, 2022

tlhunter changed the title ~~Modules are not instrumented (using webpack)~~ Feature Request: Webpack Support Dec 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature Request: Webpack Support #827

Feature Request: Webpack Support #827

mruoss commented Jan 16, 2020 •

edited

Loading

mruoss commented Jan 16, 2020

rochdev commented Jan 16, 2020

armenr commented Jan 17, 2020

mruoss commented Jan 17, 2020 •

edited

Loading

armenr commented Jan 18, 2020 •

edited

Loading

mruoss commented Jan 20, 2020

mruoss commented Jan 20, 2020

rochdev commented Jan 20, 2020

armenr commented Jan 21, 2020

rochdev commented Jan 22, 2020

armenr commented Jan 22, 2020

williancolognesitrimble commented Jun 9, 2020 •

edited

Loading

mruoss commented Jun 9, 2020

maschmann commented Jul 9, 2020 •

edited

Loading

kalebdf commented Jul 9, 2020

kalebdf commented Jul 10, 2020

ashatch commented Jul 14, 2020

rochdev commented Jul 15, 2020

amitizle commented Oct 26, 2020 •

edited

Loading

rochdev commented Oct 26, 2020

amitizle commented Oct 26, 2020

rochdev commented Oct 26, 2020

rochdev commented Oct 26, 2020

amitizle commented Oct 27, 2020

samsaggace commented Dec 1, 2020 •

edited

Loading

kty1965 commented Dec 15, 2020 •

edited

Loading

rochdev commented May 18, 2021

SasSam commented Aug 18, 2021

vanseverk commented May 10, 2022

rochdev commented Jun 2, 2022 •

edited

Loading

tlhunter commented Dec 18, 2023

Feature Request: Webpack Support #827

Feature Request: Webpack Support #827

Comments

mruoss commented Jan 16, 2020 • edited Loading

mruoss commented Jan 16, 2020

rochdev commented Jan 16, 2020

armenr commented Jan 17, 2020

mruoss commented Jan 17, 2020 • edited Loading

armenr commented Jan 18, 2020 • edited Loading

mruoss commented Jan 20, 2020

mruoss commented Jan 20, 2020

rochdev commented Jan 20, 2020

armenr commented Jan 21, 2020

rochdev commented Jan 22, 2020

armenr commented Jan 22, 2020

williancolognesitrimble commented Jun 9, 2020 • edited Loading

mruoss commented Jun 9, 2020

maschmann commented Jul 9, 2020 • edited Loading

kalebdf commented Jul 9, 2020

kalebdf commented Jul 10, 2020

ashatch commented Jul 14, 2020

rochdev commented Jul 15, 2020

amitizle commented Oct 26, 2020 • edited Loading

rochdev commented Oct 26, 2020

amitizle commented Oct 26, 2020

rochdev commented Oct 26, 2020

rochdev commented Oct 26, 2020

amitizle commented Oct 27, 2020

samsaggace commented Dec 1, 2020 • edited Loading

kty1965 commented Dec 15, 2020 • edited Loading

rochdev commented May 18, 2021

SasSam commented Aug 18, 2021

vanseverk commented May 10, 2022

rochdev commented Jun 2, 2022 • edited Loading

tlhunter commented Dec 18, 2023

mruoss commented Jan 16, 2020 •

edited

Loading

mruoss commented Jan 17, 2020 •

edited

Loading

armenr commented Jan 18, 2020 •

edited

Loading

williancolognesitrimble commented Jun 9, 2020 •

edited

Loading

maschmann commented Jul 9, 2020 •

edited

Loading

amitizle commented Oct 26, 2020 •

edited

Loading

samsaggace commented Dec 1, 2020 •

edited

Loading

kty1965 commented Dec 15, 2020 •

edited

Loading

rochdev commented Jun 2, 2022 •

edited

Loading