Allow interactive commands through @nrwl/workspace:run-commands

[JakeGinnivan]

Description

Sometimes you just want to be able to wrap a CLI tool for a project without using a plugin. For example:

        "serve": {
            "executor": "@nrwl/workspace:run-commands",
            "options": {
                "commands": [
                    {
                        "command": "shopify app serve"
                    }
                ],
                "cwd": "apps/shopify-app"
            }
        }

When you run this, sometimes it prompts you

~/_code/shopify-example/apps/shopify-app feature/nx-serve-functions* ❯ shopify app serve                                                                                                                                                                        13:22:24
? Do you want to convert mystore.myshopify.com to a development store?
Doing this will allow you to install your app, but the store will become transfer-disabled.
Learn more: https://shopify.dev/tutorials/transfer-a-development-store-to-a-merchant#transfer-disabled-stores
 (Choose with ↑ ↓ ⏎)
> 1. yes
  2. no

But all I see is

Motivation

There are so many useful CLI tools out there which do not default to --non-interactive (or even have a non-interactive version) and when they prompt you can't actually see what is printed to the console.

Suggested Implementation

I think there are multiple execution layers, i'm not sure how to do this. Given a few pointers I could take a crack.

Alternate Implementations

[jskrzypek]

@JakeGinnivan I think it should be straightforward and just come down to patching in a stdio-forwarding mode to the run-commands implementation here (run-commands.impl.ts#L174-L219).

If you look there you'll see it uses child_process's exec() & execSync(), which buffer the stdout & stderr and then resolve a promise with the buffered stdio (or just return it for execSync).

For the purpose you describe, this would probably be achievable by just adding a mode that uses child_process's spawn() & spawnSync() instead. Since you don't actually want to do anything programmatic with the output (just forward it back to the shell, you could probably just pass 'inherit' to spawn()'s options.stdio, which will tell the child process to inherit it's stdio from the parent, runner process. This means that the stdin, stdout, & stderr are shared between the processes. The only blocker I can think of would be if nx is doing something fancy with the stdio of the processes to make them cacheable or to for the build-analytics (I am pretty new to nx so I know nothing of how those systems work).

[JakeGinnivan]

For the moment I have just created my own executor in tools.

 "executor": "./tools/executors/workspace:run-command",
            "options": {
                "command": "dotnet watch msbuild /t:RunFunctions",
                "cwd": "apps/shopify-functions"
            }

tools/executors/workspace/package.json
{
    "executors": "./executor.json"
}

tools/executors/workspace/executor.json
{
    "executors": {
        "run-command": {
            "implementation": "./run-command/impl",
            "schema": "./run-command/schema.json",
            "description": "Runs a command"
        }
    }
}

tools/executors/workspace/run-command/schema.json
{
    "$schema": "http://json-schema.org/schema",
    "type": "object",
    "cli": "nx",
    "properties": {
        "command": {
            "type": "string",
            "description": "The command to run"
        },
        "cwd": {
            "type": "string",
            "description": "The working directory to run the command in"
        }
    },
    "required": [
        "command"
    ]
}

tools/executors/workspace/run-command/impl.ts
import execa from 'execa'

export default async function buildExecutor(options: {
    command: string
    cwd?: string
}) {
    console.info(`Executing workspace:run-command...`)

    await execa.command(options.command, {
        cwd: options.cwd,
        stdio: [process.stdin, process.stdout, 'pipe'],
    })

    return { success: true }
}

Happy to dig in, but a few of my other PRs have been closed with please discuss with the team the solution first. So now I don't invest my time into contributing without getting acknowledgement from the team about the solution

[dereekb]

I ran into this issue. In my case, I need to run the command inside a docker container so the environment has access to mongodb/etc.

        "watch": {
          "executor": "@nrwl/workspace:run-commands",
          "options": {
            "command": "./exec-with-server.sh 'npx nx run server:run-tests --watch'",
            "color": true
          }
        },
        "run-tests": {
          "executor": "@nrwl/jest:jest",
          "outputs": ["coverage/packages/server"],
          "options": {
            "jestConfig": "packages/server/jest.config.js",
            "passWithNoTests": true
          }
        },

When I run the watch target it doesn't pass through the interactivity. I thought jest was acting up at first.

For now I just have bash file that runs the watch command as presented above, but it would be nice to have interactivity when using @nrwl/workspace:run-commands, or a similar command that supported interactivity.

[jskrzypek]

@JakeGinnivan that makes a lot of sense, execa is a nice library. I'm pretty new to the nx community so that's a little disappointing that they treat PRs that way, but maintaining open source is a lot of work, just like contributing to open source, so I get it 🤷

In this case I do think there's totally room for the wrapper you built to be turned into a plugin that provides that executor. It wouldn't need to be merged with the core nx project that way... Let me know if that's something you're interested in doing and I'll see if I can help out!

[JakeGinnivan]

Yeah, that is my end goal. I actually don't use many of the core NX plugins because I use TypeScript project references, vitest and vite a lot. Time is the main blocker for getting things up and running.

[dustinsgoodman]

👍 to this issue. Thank you @JakeGinnivan for reporting and providing some workarounds!

Adding another use case and a bug this is causing:

• Intricate outputs from other CLIs can't forward their commands appropriately as nx is blocking causing important detail blocks to not be properly formatted. Example - serverless offline supplies a block of mapped routes when invoking that is getting muddled and makes the feature unusable:

  Actual:
  

  Expected:
  

• I've created a Node repl with history for testing (similar to a rails console). Unfortunately, the history is inaccessible as nx is blocking interactivity when run via run-commands so pressing the up arrow to go back in history causes the following result:
  

[dustinsgoodman]

Wanting to confirm that the workaround (quoted below) from @JakeGinnivan is a perfect solve to my above issues. Additionally, wanted to add a couple of tweaks/notes that would help others in the future using this executor:

1. Use execa version 5.1.1
2. I had to modify my impl.ts as follows to work:

import { commandSync } from 'execa';

export default async function buildExecutor(options: {
  command: string;
  cwd?: string;
}) {
  console.info(`Executing workspace:run-command...`);

  commandSync(options.command, {
    cwd: options.cwd,
    stdio: [process.stdin, process.stdout, 'pipe'],
  });

  return { success: true };
}

For the moment I have just created my own executor in tools.

 "executor": "./tools/executors/workspace:run-command",
            "options": {
                "command": "dotnet watch msbuild /t:RunFunctions",
                "cwd": "apps/shopify-functions"
            }

tools/executors/workspace/package.json
{
    "executors": "./executor.json"
}

tools/executors/workspace/executor.json
{
    "executors": {
        "run-command": {
            "implementation": "./run-command/impl",
            "schema": "./run-command/schema.json",
            "description": "Runs a command"
        }
    }
}

tools/executors/workspace/run-command/schema.json
{
    "$schema": "http://json-schema.org/schema",
    "type": "object",
    "cli": "nx",
    "properties": {
        "command": {
            "type": "string",
            "description": "The command to run"
        },
        "cwd": {
            "type": "string",
            "description": "The working directory to run the command in"
        }
    },
    "required": [
        "command"
    ]
}

tools/executors/workspace/run-command/impl.ts
import execa from 'execa'

export default async function buildExecutor(options: {
    command: string
    cwd?: string
}) {
    console.info(`Executing workspace:run-command...`)

    await execa.command(options.command, {
        cwd: options.cwd,
        stdio: [process.stdin, process.stdout, 'pipe'],
    })

    return { success: true }
}

Happy to dig in, but a few of my other PRs have been closed with please discuss with the team the solution first. So now I don't invest my time into contributing without getting acknowledgement from the team about the solution

[AgentEnder]

@JakeGinnivan For single command runs, or serial command runs, does this not already work? I.e. changing to use "command" instead of "commands", or adding "parallel: false" to your target. When running in parallel forwarding stdin doesn't make much sense, but in serial its doable (and the default behavior)

[dustinsgoodman]

@AgentEnder I don't think that's necessarily true. The run-commands implementation explicitly inherits its io settings from its parent process as seen here:

nx/packages/workspace/src/executors/run-commands/run-commands.impl.ts

Lines 212 to 219 in efedd2e

	function createSyncProcess(command: string, color: boolean, cwd: string) {
	execSync(command, {
	env: processEnv(color),
	stdio: ['inherit', 'inherit', 'inherit'],
	maxBuffer: LARGE_BUFFER,
	cwd,
	});
	}

I tested a fork of this implementation and switched the stdio settings to:

stdio: [process.stdin, process.stdout, process.stderr],

which has yielded a working result.

[dereekb]

@AgentEnder It does not. My example above uses a single command and it does not work.

[robbesettlemint]

Anyone know why when trying to run this executor I'm getting this common error:

import { commandSync } from 'execa';
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at Object.compileFunction (node:vm:352:18)
    at wrapSafe (node:internal/modules/cjs/loader:1026:15)
    at Module._compile (node:internal/modules/cjs/loader:1061:27)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1149:10)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:999:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at /Users/administrator/Repos/orange-ebsi/node_modules/nx/src/shared/workspace.js:122:28
    at /Users/administrator/Repos/orange-ebsi/node_modules/nx/src/commands/run.js:146:36

I mean normally u can fix this by adding for example "type": "module" but that didn't work. Does anyone know why I'm having this issue and u guys are not. Would it help to add a tsconfig?

[dustinsgoodman]

Anyone know why when trying to run this executor I'm getting this common error:

import { commandSync } from 'execa';
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at Object.compileFunction (node:vm:352:18)
    at wrapSafe (node:internal/modules/cjs/loader:1026:15)
    at Module._compile (node:internal/modules/cjs/loader:1061:27)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1149:10)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:999:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at /Users/administrator/Repos/orange-ebsi/node_modules/nx/src/shared/workspace.js:122:28
    at /Users/administrator/Repos/orange-ebsi/node_modules/nx/src/commands/run.js:146:36

I mean normally u can fix this by adding for example "type": "module" but that didn't work. Does anyone know why I'm having this issue and u guys are not. Would it help to add a tsconfig?

You probably need to ensure tslib is installed correctly. Also what command are you running that's producing the error?

[robbesettlemint]

I'm trying to run nx build eks-cluster && pulumi up -s stackname --cwd apps/eks-cluster, but getting the same when trying echo hello so it's definitely not the command.
also I have tslib installed in my root package.json: "tslib": "^2.0.0". and ran yarn again to be sure. So I don't think it's that either

[robbesettlemint]

I made it work by building the ts file and running the js file. Also changed it a bit so u can run multiple commands:

import { commandSync } from 'execa';

export default async function buildExecutor(options: {
  commands: string[];
  cwd?: string;
}) {
  console.info(`Executing workspace:run-command...`);

  options.commands.forEach(command => {
    commandSync(command, {
      cwd: options.cwd,
      stdio: [process.stdin, process.stdout, 'pipe'],
    });
  });


  return { success: true };
}

schema.json

{
    "$schema": "http://json-schema.org/schema",
    "type": "object",
    "cli": "nx",
    "properties": {
        "commands": {
            "type": "array",
            "description": "The commands to run"
        },
        "cwd": {
            "type": "string",
            "description": "The working directory to run the command in"
        }
    },
    "required": [
        "commands"
    ]
}

[JakeGinnivan]

@robbesettlemint make sure you install execa@^5 rather than the latest.

https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c latest version is ESM only

[notaphplover]

Hi @JakeGinnivan, thank you so much for your workaround. Regarding this, what are your thoughts about nx lifecycles? It seems the actual run and run-many ones only clear stdout lines, in my case is causing some trouble:

packages/nx/src/tasks-runner/life-cycles/dynamic-run-one-terminal-output-life-cycle.ts

const clearDependentTargets = () => {
  for (let i = 0; i < dependentTargetsNumLines; i++) {
    readline.moveCursor(process.stdout, 0, -1);
    readline.clearLine(process.stdout, 0);
  }
};

It seems these life cycles are not designed for this interactive approach, in my case I can successfully pipe stdio, but my output is cleared providing a poor experience.

I'm not an expert of the project but it seems nx@13 introduced some sort of life cycle API (I could not find any docs), so maybe there's still hope to deal with this in any way.

Edit: nvm, It seems NX_STREAM_OUTPUT env set to true does the trick. I'll let this comment in case someone faces the same issue.

[william-kerr]

@Coly010, thanks for linking that fix! I created a simple run-interactive-command executor and added "outputCapture": "direct-nodejs" in schema.json. Then, I was able to use execSync from child_process (instead of execa), like so:

execSync(command, {
  cwd: options.cwd,
  stdio: ['inherit', 'inherit', 'inherit'],
})

[zrisher]

+1 to fixing this in main. Appreciate the provided workarounds, but Nx should provide a command runner that does not swallow interactive output.

[fahimalizain]

Adding outputCapture: direct-nodejs in schema.json of the executor as suggested by @william-kerr enabled cli-select to work perfectly 👍🏼 It was missing out on options otherwise.

[enchorb]

Would be great to have a nx:run-interactive-commands executor provided by Nx directly rather than users having to create their own

[empeje]

I have a workaround for this.

1. Setup your interactive script in the package.json
2. Run your script using nx:run-script e.g. here's my case

    "deploy": {
      "executor": "nx:run-script",
      "outputs": ["{workspaceRoot}/coverage/{projectRoot}"],
      "options": {
        "cwd": "packages/hologram-bos",
        "script": "bos:deploy"
      }
    },

and id my script:

  "scripts": {
    "bos:deploy": "bos components deploy"
  }

[miluoshi]

The interactive commands were working until we updated from Nx 15.4.5 to 16.0.0.

I think this change might have broken it by removing the child process being created with inherited stdio.

Adding this line to run-commands.impl.ts fixes the issue for me:

process.stdin.pipe(childProcess.stdin);

[mattfysh]

thanks @JakeGinnivan for the nice solution! I'm using this to run docker-compose up and needed to attach stdin so the output would print in colour. Almost necessary when running a cluster as the colours indicate which container emitted each log line.

The only thing I find now is that when I want to bring down the cluster via CTRL+C, the NX process group is killed while docker-compose is still coming down, leading to some dangling log lines after control has been returned to the terminal

Does anyone know how this could be fixed? I've tried using exec and spawn, and then capturing SIGINT but couldn't get it orchestrated correctly

[enchorb]

This also breaks metro for React Native. 'R' to reload from the terminal no longer works because Nx swallows the input

[wardds]

Same issue here, we have a little CLI tool (using inquirer for interactive input). It uses executor nx:run-commands for a straightforward ts-node ./src/main.ts command, but it's broken on nx v16.0.3

[AlexJWayne]

When I upgraded to NX 16 prisma now thinks it's being run from a non-interactive prompt. See: https://www.prisma.io/docs/concepts/components/prisma-migrate/prisma-migrate-limitations-issues#prisma-migrate-in-non-interactive-environments

I've applied this patch hack and input does work in other scripts, but prisma still complains.

[crusoexia]

This would become a deal breaker.

[MinskLeo]

For the moment I have just created my own executor in tools.

 "executor": "./tools/executors/workspace:run-command",
            "options": {
                "command": "dotnet watch msbuild /t:RunFunctions",
                "cwd": "apps/shopify-functions"
            }

tools/executors/workspace/package.json
{
    "executors": "./executor.json"
}

tools/executors/workspace/executor.json
{
    "executors": {
        "run-command": {
            "implementation": "./run-command/impl",
            "schema": "./run-command/schema.json",
            "description": "Runs a command"
        }
    }
}

tools/executors/workspace/run-command/schema.json
{
    "$schema": "http://json-schema.org/schema",
    "type": "object",
    "cli": "nx",
    "properties": {
        "command": {
            "type": "string",
            "description": "The command to run"
        },
        "cwd": {
            "type": "string",
            "description": "The working directory to run the command in"
        }
    },
    "required": [
        "command"
    ]
}

tools/executors/workspace/run-command/impl.ts
import execa from 'execa'

export default async function buildExecutor(options: {
    command: string
    cwd?: string
}) {
    console.info(`Executing workspace:run-command...`)

    await execa.command(options.command, {
        cwd: options.cwd,
        stdio: [process.stdin, process.stdout, 'pipe'],
    })

    return { success: true }
}

Happy to dig in, but a few of my other PRs have been closed with please discuss with the team the solution first. So now I don't invest my time into contributing without getting acknowledgement from the team about the solution

Thanks for solution. I made this gist based on exactly this response for easier copy-paste.
https://gist.github.com/MinskLeo/749311c117996cf88e5594e61a4bb333

[PaulMest]

I couldn't get any of these custom executors solutions to work with nx 16.8. When I wrap a CLI with execa or node-pty outside of nx, it works perfectly.

I've also posted a related question on Stack Overflow. If anybody happens to solve this (or if NX fixes the likely culprit), feel free to answer on SO as well.

[codescalar]

Prisma is a pain to use with NX because of this CLI interactivity issue

[yeongjet]

Recommend: nx-prisma

[yeongjet]

@JakeGinnivan Good job, your method is feasible, if someone encounters the same problem as me that the last line prompt is missing:

you can refer to my code:

import { RunInteractiveCommandExecutorSchema } from './schema'
import execa from 'execa'

const sleep = async ms =>
    new Promise((resolve, reject) => {
        setTimeout(() => {
            resolve(1)
        }, ms)
    })

export default async function runExecutor(options: RunInteractiveCommandExecutorSchema) {
    console.log('Executing run-interactive-command...')
    try {
        let isSuccess = true,
            isEnd = false
        const childProcess = execa.command(`${options.command} ${options._.join(' ')}`, {
            cwd: options.cwd,
            stdio: [process.stdin, 'pipe', 'pipe']
        })
        childProcess.stdout?.on('data', data => {
            console.log(data.toString())
        })
        childProcess.stderr?.on('data', data => {
            console.error(data.toString())
            isSuccess = false
        })
        childProcess.addListener('exit', () => {
            isEnd = true
        })
        while (!isEnd) {
            await sleep(1000)
        }
        return {
            success: isSuccess
        }
    } catch (error) {
        console.error(error)
    }
}

[AlaaInflyter]

For the moment I have just created my own executor in tools.

 "executor": "./tools/executors/workspace:run-command",
            "options": {
                "command": "dotnet watch msbuild /t:RunFunctions",
                "cwd": "apps/shopify-functions"
            }

tools/executors/workspace/package.json
{
    "executors": "./executor.json"
}

tools/executors/workspace/executor.json
{
    "executors": {
        "run-command": {
            "implementation": "./run-command/impl",
            "schema": "./run-command/schema.json",
            "description": "Runs a command"
        }
    }
}

tools/executors/workspace/run-command/schema.json
{
    "$schema": "http://json-schema.org/schema",
    "type": "object",
    "cli": "nx",
    "properties": {
        "command": {
            "type": "string",
            "description": "The command to run"
        },
        "cwd": {
            "type": "string",
            "description": "The working directory to run the command in"
        }
    },
    "required": [
        "command"
    ]
}

tools/executors/workspace/run-command/impl.ts
import execa from 'execa'

export default async function buildExecutor(options: {
    command: string
    cwd?: string
}) {
    console.info(`Executing workspace:run-command...`)

    await execa.command(options.command, {
        cwd: options.cwd,
        stdio: [process.stdin, process.stdout, 'pipe'],
    })

    return { success: true }
}

Happy to dig in, but a few of my other PRs have been closed with please discuss with the team the solution first. So now I don't invest my time into contributing without getting acknowledgement from the team about the solution

FYI, doesn't work with env vars.
(for example if the command is something like FOO="bar" my-command)

[cmrfrd]

An important issue, I believe worth mentioning is this

Talking about using the cjs version of exa, I needed to install it to make this executor work.

For my particular use case I wanted to create this target to interactively use ipython within a poetry env.

        "shell": {
            "executor": "./tools/executors/workspace:run-command",
            "options": {
                "command": "poetry run ipython",
                "cwd": "libs/testproject"
            }
        }

but I needed to pass in stdin, stdout, & stderr as so:

const { execaCommandSync } = require('execa');

export default async function buildExecutor(options: {
    command: string;
    cwd?: string;
}) {
    console.info(`🟡 Executing workspace:run-command...`);
    await execaCommandSync(options.command, {
        cwd: options.cwd,
        stdout: process.stdout,
        stderr: process.stderr,
        stdin: process.stdin
    });
    console.info(`✅ Executing workspace:run-command finished!`);
    return { success: true };
}

resulting in this:

$ pnpx nx run testproject:shell
Packages: +111
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Progress: resolved 119, reused 111, downloaded 0, added 111, done

> nx run testproject:shell

>  NX  Falling back to ts-node for local typescript execution. This may be a little slower.
  - To fix this, ensure @swc-node/register and @swc/core have been installed
🟡 Executing workspace:run-command...
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.20.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: print("yay!")
yay!

In [2]:                                                                                                                            
Do you really want to exit ([y]/n)? y
✅ Executing workspace:run-command finished!

 —————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target shell for project testproject (10s)

[JakeGinnivan]

Just tested, works as expected!

Here is an example:

"drizzle-kit": {
    "executor": "nx:run-commands",
    "options": {
        "commands": [
            {
                "command": "drizzle-kit",
                "forwardAllArgs": true
            }
        ],
        "parallel": false,
        "cwd": "apps/chat"
    }
}

Thanks @AgentEnder and @FrozenPandaz for getting this resolved!

[AgentEnder]

Extra Kudos to @Cammisuli to - a lot of this rust stuff is absolutely thanks to him.

[mattfysh]

great work @Cammisuli 🎉 was able to remove the manual patch. One thing I noticed is that if I'm interacting with a menu, I need to press up/down a couple of times before it registers. Not sure if perhaps the input is buffering somewhere?

[hco]

It does not work when running with nx run-many though, right?

[Cammisuli]

@mattfysh there shouldn't be any buffering. We send all binary data to the underlying process without doing anything on our side. I just did some changes that affect unix systems that should make it more consistent though.

@hco no, it does not work when using nx run-many because its not possible to send 1 input to multiple processes.

[vezaynk]

@Cammisuli I want to run an interactive foreground task, and a supporting background task in parallel.

Can nx do that somehow?

[domvo]

If anyone comes here wondering why it does not work, parallel must be set to false in order for this to work.

e.g.:

{
  "name": "some-cli-tool",
  "$schema": "../../node_modules/nx/schemas/project-schema.json",
  "sourceRoot": "apps/some-cli-tool/src",
  "projectType": "application",
  "targets": {
    "start": {
      "executor": "nx:run-commands",
      "options": {
        "cwd": "apps/some-cli-tool/src",
        "commands": ["ts-node index.ts"],
        "parallel": false // this is mandatory
      }
    }
  }
}

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.