Error: Module "fabric-network" could not be loaded: Error: Cannot find module 'fs-extra'

[feickertm]

Context

I am trying to run one of the fabric sample benchmakrs. For this I am following the documentation on how to run caliper using the docker image.

I am using the following docker-compose script:

version: '2'

services:

    caliper:
        container_name: caliper
        image: hyperledger/caliper:0.4.0
        command: launch manager
        tty: true
        environment:
        - CALIPER_BIND_SUT=fabric:1.4.11
        - CALIPER_BENCHCONFIG=benchmarks/samples/fabric/marbles/config.yaml
        - CALIPER_NETWORKCONFIG=networks/fabric/v1/v1.4.1/2org1peergoleveldb/fabric-go.yaml
        volumes:
        - D:\git\caliper-benchmarks:/hyperledger/caliper/workspace

Expected Behavior

The benchmark is running without error.

Actual Behavior

When I try to spin up the containers I get the following result.

$ docker compose up
Container caliper  Created
Attaching to caliper
caliper  | 2021.06.12-11:43:40.738 info  [caliper] [cli-launch-manager]         Binding specification is present, performing binding for "fabric:1.4.11"
caliper  | 2021.06.12-11:43:40.741 warn  [caliper] [bind]       Working directory not specified. Using "/hyperledger/caliper/workspace"
caliper  | 2021.06.12-11:43:40.741 info  [caliper] [bind]       User-provided arguments for npm: "-g"
caliper  | 2021.06.12-11:43:40.741 info  [caliper] [bind]       Binding for fabric@1.4.11. This might take some time...
caliper  | 2021.06.12-11:43:40.741 info  [caliper] [bind]       Using working directory: /hyperledger/caliper/workspace
caliper  | 2021.06.12-11:43:40.742 info  [caliper] [bind]       Calling npm with: install -g fabric-client@1.4.11 fabric-protos@2.0.0-snapshot.1 fabric-network@1.4.11
npm WARN deprecated grpc@1.24.2: This library will not receive further updates other than security fixes. We recommend using @grpc/grpc-js instead.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated grpc@1.14.2: This library will not receive further updates other than security fixes. We recommend using @grpc/grpc-js instead.
+ fabric-network@1.4.11
caliper  | + fabric-client@1.4.11
caliper  | + fabric-protos@2.0.0-snapshot.1
caliper  | updated 3 packages in 8.236s
caliper  | 2021.06.12-11:43:49.383 info  [caliper] [cli-launch-manager]         Set workspace path: /hyperledger/caliper/workspace
caliper  | 2021.06.12-11:43:49.384 info  [caliper] [cli-launch-manager]         Set benchmark configuration path: /hyperledger/caliper/workspace/benchmarks/samples/fabric/marbles/config.yaml
caliper  | 2021.06.12-11:43:49.384 info  [caliper] [cli-launch-manager]         Set network configuration path: /hyperledger/caliper/workspace/networks/fabric/v1/v1.4.1/2org1peergoleveldb/fabric-go.yaml
caliper  | 2021.06.12-11:43:49.384 info  [caliper] [cli-launch-manager]         Set SUT type: fabric
caliper  | 2021.06.12-11:43:49.419 info  [caliper] [benchmark-validator]        No observer specified, will default to `none`
caliper  | 2021.06.12-11:43:49.419 info  [caliper] [caliper-engine]     Starting benchmark flow
caliper  | 2021.06.12-11:43:49.757 error [caliper] [cli-launch-manager]         Unexpected error during benchmark execution: Error: Module "fabric-network" could not be loaded: Error: Cannot find module 'fs-extra'
caliper  | Searched paths: /home/node/.npm-global/lib/node_modules/@hyperledger/caliper-core/lib/common/utils/node_modules,/home/node/.npm-global/lib/node_modules/@hyperledger/caliper-core/lib/common/node_modules,/home/node/.npm-global/lib/node_modules/@hyperledger/caliper-core/lib/node_modules,/home/node/.npm-global/lib/node_modules/@hyperledger/caliper-core/node_modules,/home/node/.npm-global/lib/node_modules/@hyperledger/node_modules,/home/node/.npm-global/lib/node_modules,/home/node/.npm-global/node_modules,/home/node/node_modules,/home/node_modules,/node_modules
caliper  | Usage:
caliper  |  caliper launch manager --caliper-bind-sut fabric:1.4 [other options]
caliper  |
caliper  | Options:
caliper  |   --help, -h           Show usage information  [boolean]
caliper  |   --version            Show version information  [boolean]
caliper  |   --caliper-bind-sut   The name and version of the platform to bind to  [string]
caliper  |   --caliper-bind-cwd   The working directory for performing the SDK install  [string]
caliper  |   --caliper-bind-args  Additional arguments to pass to "npm install". Use the "=" notation when setting this parameter  [string]
caliper  |   --caliper-bind-file  Yaml file to override default (supported) package versions when binding an SDK  [string]
caliper  |
caliper  | Error: Module "fabric-network" could not be loaded: Error: Cannot find module 'fs-extra'
caliper  | Searched paths: /home/node/.npm-global/lib/node_modules/@hyperledger/caliper-core/lib/common/utils/node_modules,/home/node/.npm-global/lib/node_modules/@hyperledger/caliper-core/lib/common/node_modules,/home/node/.npm-global/lib/node_modules/@hyperledger/caliper-core/lib/node_modules,/home/node/.npm-global/lib/node_modules/@hyperledger/caliper-core/node_modules,/home/node/.npm-global/lib/node_modules/@hyperledger/node_modules,/home/node/.npm-global/lib/node_modules,/home/node/.npm-global/node_modules,/home/node/node_modules,/home/node_modules,/node_modules
caliper  |     at Function.loadModule (/home/node/.npm-global/lib/node_modules/@hyperledger/caliper-core/lib/common/utils/caliper-utils.js:118:19)
caliper  |     at Function.moduleIsInstalled (/home/node/.npm-global/lib/node_modules/@hyperledger/caliper-core/lib/common/utils/caliper-utils.js:181:26)
caliper  |     at new FabricConnector (/home/node/.npm-global/lib/node_modules/@hyperledger/caliper-cli/node_modules/@hyperledger/caliper-fabric/lib/fabric-connector.js:63:26)
caliper  |     at CaliperEngine.connectorFactory [as adapterFactory] (/home/node/.npm-global/lib/node_modules/@hyperledger/caliper-cli/node_modules/@hyperledger/caliper-fabric/lib/connectorFactory.js:26:23)
caliper  |     at CaliperEngine.run (/home/node/.npm-global/lib/node_modules/@hyperledger/caliper-core/lib/manager/caliper-engine.js:93:36)
caliper  |     at Function.handler (/home/node/.npm-global/lib/node_modules/@hyperledger/caliper-cli/lib/launch/lib/launchManager.js:62:43)
caliper  |     at process._tickCallback (internal/process/next_tick.js:68:7)
caliper exited with code 1

The documentation also mentions:
Note: the above network configuration file contains a start script to spin up a local Docker-based Fabric network, which will not work in this form. So make sure to remove the start (and end) script, and change the node endpoints to remote addresses.

So I am not sure if this is the expected behavior. But to me it seems I am missing something during the installation.

Possible Fix

Steps to Reproduce

Follow the documentation on installing caliper using the docker image.

Existing issues

#374
#844

Context

Your Environment

• Version used:
• Environment name and version (e.g. Chrome 39, node.js 5.4): Docker Desktop v20.10.6, using the WSL 2 based engine
• Operating System and version (desktop or mobile): Windows 10
• Link to your project:

[davidkel]

This is a bug in the fabric node-sdk 1.4. They don't declare a dependency on fs-extra in their package.json and rely on it being provided by another of their dependencies, but it breaks caliper when running in a docker container. Unfortunately the node-sdk 1.4 is out of support so this will never be fixed and also fabric 1.4 is now end of life so people should really be using fabric 2.2 now. However there are a couple of options

1. bind to fabric:2.2. This will use the node sdk 2.2 version (gateway only so remember to specify --caliper-fabric-gateway-enabled option) but should still work with a 1.4 fabric
   2 . create a custom bind file in your workspace, eg a bind.yaml file

sut:
    # Specifies the available SDK bindings and their Node.js-specific install settings
    fabric:
        1.4.18: &fabric-v1-lts
            packages: ['fabric-client@1.4.18', 'fabric-protos@2.0.0-snapshot.1', 'fabric-network@1.4.18','fs-extra@8.1.0']
        1.4: *fabric-v1-lts

And then you can bind to 1.4.18 or just 1.4, for example on a CLI to launch caliper from it's image

docker run -e CALIPER_BIND_SUT=fabric:1.4 -e CALIPER_BIND_FILE=/caliper/bind.yaml -e CALIPER_BENCHCONFIG=/caliper/benchmarks/assetBenchmark.yaml -e CALIPER_NETWORKCONFIG=/caliper/networks/config.yaml -v $PWD:/caliper hyperledger/caliper:0.4.2 launch manager

specically note the -e CALIPER_BIND_FILE=/caliper/bind.yaml option