Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for piped input #66

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

stefanbuck
Copy link
Member

Resolves #65

To make it easier to operate on hundreds of repositories, this pull request allows you to pipe input to octoherd.

Example

list.txt

octoherd/script-hello-world
octoherd/script-rename-master-branch-to-main
// octokit/octokit.js
# sindresorhus/get-stdin
cat list.txt | octoherd run -S path/to/script.js -T $TOKEN

When piping a list of repositories, lines starting with a comment delimiter such as # or // will be ignored.

@stefanbuck stefanbuck changed the title Add support for piped input feat: add support for piped input Mar 24, 2022
@@ -108,7 +108,7 @@ export async function octoherd(options) {
octokit,
script: octoherdScript,
userOptions,
octoherdReposPassedAsFlag: !!octoherdRepos,
octoherdReposPassedAsFlag: octoherdRepos.length,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -20,6 +22,10 @@ const argv = await yargs(hideBin(process.argv))
.version(VERSION)
.epilog(EPILOG).argv;

const stdin = await getStdin();
Copy link
Member Author

@stefanbuck stefanbuck Mar 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get-stdin is responsible to get the piped input and returns a string

@stefanbuck
Copy link
Member Author

stefanbuck commented Mar 24, 2022

I just did another test and looks like piping into octoherd breaks the interactive mode somehow. IT just exists. I'll investigate

IMG_20220407_193447

@stefanbuck
Copy link
Member Author

Just a quick update. I made some progress (at least I understand the problem) a little better. However piping is a little more complicated than I expected.

The problem

Let's start simple with the following enquirer example

import enquirer from 'enquirer';

const prompt = new enquirer.Select({
  name: 'color',
  message: 'Pick a flavor',
  choices: ['apple', 'grape', 'watermelon', 'cherry', 'orange'],
});

prompt
  .run()
  .then((answer) => console.log('Answer:', answer))
  .catch(console.error);

When running node demo-1.js in the terminal, the user is able to select a flavour using the arrow keys.

demo1

Now if we pipe some random value echo "hi" | node demo-1.mjs in the terminal, enquirer by passes the interactive mode and selects the first value. This is what is currently happening with this pull request.

demo2

Research

ipt (Interactive Pipe To) is a cli written in Node.js and the piping works as expected. The project is using inquirer, and first I though inquirer is handling this piping case, but it is not. After reading through the source code I found https://github.com/ruyadorno/ipt/blob/8a9e6791ec7af3e471247192c42803c74c657c3b/src/cli.js#L106-L138 which seems to be the magic trick. I was able to simplify as follow:

import { promisify } from "util";
import enquirer from "enquirer";
import reopenTTY from 'reopen-tty';

function end() {
    process.exit(0);
}

function error(err) {
    console.error(err);
    process.exit(1);
}

Promise.all([promisify(reopenTTY.stdin)(), promisify(reopenTTY.stdout)()]).then(
    (stdio) => {
        const [ttyStdin, ttyStdout] = stdio;
        const stdin = ttyStdin;
        const stdout = ttyStdout;

        const prompt = new enquirer.Select({
            name: 'color',
            message: 'Pick a flavor',
            choices: ['apple', 'grape', 'watermelon', 'cherry', 'orange'],
            stdout,
            stdin,
        });

        prompt
            .run()
            .then((answer) => console.log('Answer:', answer))
            .then(end)
            .catch(error);
    }
);

Disclaimer, I'm new to this as well and this comment describes my current understanding (which might be wrong or partial incomplete).

From the TTY documentation page

When Node.js detects that it is being run with a text terminal ("TTY") attached, process.stdin will, by default, be initialized as an instance of tty.ReadStream and both process.stdout and process.stderr will, by default, be instances of tty.WriteStream.

When piping something to a Node.js process, TTY is undefined and process.stdin is initialized with an instance of Socket. Therefore there is not connection between the active terminal session and the running Node.js process. We need to recreate such a TTY stream by using reopen-tty and then pass the new stdin and stdout stream to enquirer.

This is how I understood how it after reading and try and error the code snippet. How to integrate this into Octoherd is something for later this week.

@gr2m
Copy link
Member

gr2m commented Mar 29, 2022

Thanks for doing all the research, this is great!

Maybe we could detect piping mode, and in that case require all arguments to be set by CLI flags? And if one is missing, throw an error. That way we would never get into an interactive mode. I think we also thought about adding an explicit flag to prevent interactive mode, which is useful e.g. scripts run in CI, so we would need to create that anyway. If it would help, maybe we do that first, as a preparation for piping support, as it seems to unblock it? What do you think @stefanbuck @oscard0m

@stefanbuck
Copy link
Member Author

Maybe we could detect piping mode, and in that case require all arguments to be set by CLI flags?

From my understanding we can use process.stdin.isTTY for that. It returns undefined in piping mode otherwise true (when calling the script straight away)

I think we also thought about adding an explicit flag to prevent interactive mode

So in comparison to --octoherd-bypass-confirms, this new flag will prevent the script from executing?

@gr2m
Copy link
Member

gr2m commented Mar 30, 2022

this new flag will prevent the script from executing?

only if a required argument was not set. The new flag would set --octoherd-bypass-confirms implicitly.

@oscard0m
Copy link
Member

oscard0m commented Apr 7, 2022

I just did another test and looks like piping into octoherd breaks the interactive mode somehow. IT just exists. I'll investigate

@stefanbuck be careful with publishing a screenshot with your GitHub Token. Make sure you expire/delete it and create a new one.

When piping something to a Node.js process, TTY is undefined and process.stdin is initialized with an instance of Socket. Therefore there is not connection between the active terminal session and the running Node.js process. We need to recreate such a TTY stream by using reopen-tty and then pass the new stdin and stdout stream to enquirer.

I don't have a deep experience with CLI's in Node.js and enquirer.
Searching a little bit, I come up with this nice CLI Best Practices Repo which suggests this for reading from stdin:

const readline = require("readline");

const rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout,
});

rl.question("What do you think of Node.js? ", (answer) => {
  // TODO: Log the answer in a database
  console.log(`Thank you for your valuable feedback: ${answer}`);

  rl.close();
});

Is not enquirer doing something similar?

  • I see they set process.stdin and process.stdout as default options.
  • Maybe we should check if this is a bug from enquirer or bad usage of the library? (from @stefanbuck I assume is a bug of the library?)

Maybe we could detect piping mode, and in that case, require all arguments to be set by CLI flags?

I think this is the user's responsibility. When piping something as input for the command, it should be read, but still, be in interactive mode for the rest of the flags (unless the rest of the flags were provided too).

I think we also thought about adding an explicit flag to prevent interactive mode, which is useful e.g. scripts run in CI

For CI environments I come up with the following utility library. I think we should open a different discussion for CI mode (I can create it if you are happy with it) and talk about piped input here (and link comments if they are interconnected of course).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support --from-file to allow reading repos from a text file
3 participants