Turning scripts into command line tools

[Hamatti]

I love command line and building tools for it.

Most often my things start as scripts (for example, I have a bunch of custom npm scripts to help me run different parts of my website) but I want to become better at turning them into more fleshed out and better documented command line tools so they become easier to extend and to remember how they work.

In this thread, I'll explore some options for tooling to build Javascript CLI tools and convert one of my messy Javascript scripts into one and a Python script into another Javascript CLI tool.

Welcome along the way!

[Hamatti]

I've used inquirer before in gym-leader-challenge-validator project. In that, I built an interactive menu since I run that tool once every 3 months and can't ever remember what commands to run.

In that tool, the main CLI part looked like this:

const inquirer = require("inquirer");

const { action } = await inquirer.prompt([
    {
      type: "list",
      name: "action",
      question: "What do you want to do?",
      choices: [
        { value: "list", name: "List sets in database" },
        { value: "download", name: "Download a new set from API" },
        { value: "banlist", name: "Adjust banlist" },
      ],
    },
  ]);

The prompt returns the value of the choice and then I use that to run other functions.

[Hamatti]

For these two tools that I'm looking into today, I don't necessarily need a menu. I want a solid, well-documented command line interface that makes running specific functions with specific flags and parameters easier.

So I'm looking into commander.js as its example looks ok:

const { Command } = require('commander');
const program = new Command();

program
  .name('string-util')
  .description('CLI to some JavaScript string utilities')
  .version('0.8.0');

program.command('split')
  .description('Split a string into substrings and display as an array')
  .argument('<string>', 'string to split')
  .option('--first', 'display just the first substring')
  .option('-s, --separator <char>', 'separator character', ',')
  .action((str, options) => {
    const limit = options.first ? 1 : undefined;
    console.log(str.split(options.separator, limit));
  });

program.parse();

I like how the interface is built by documenting what each command does and how to use it.

[Hamatti]

My first script is a tool I initially prototyped earlier today in Python to fetch these public note issue threads from GitHub and storing desired parts in a JSON file that I can then use with my Eleventy site to render individual read-only copies of these threads.

Since it was a quick prototype experimenting with the GitHub API, the interface became very hacky:

if __name__ == '__main__':
    if len(sys.argv) != 4:
        print('Usage: python downloader.py [user] [repository] [issue number]')
    
    _script, user, repository, issue = sys.argv
    main(*sys.argv[1:])

I want to start by turning this into a proper CLI with commander.

[Hamatti]

I need to add the project to my dev dependencies:

➜ npm install --save-dev commander

To replicate what I was thinking in Python (with improvements rather than direct 1:1 translation)

const { Command } = require("commander");

const program = new Command();

program
  .name("GitHub public notes downloader")
  .description(
    "Create read-only copies of the issue threads in my public-notes repository"
  )
  .version("1.0.0");

program
  .command("fetch")
  .description("Fetch issue and its comments and store them as JSON")
  .argument("<issue>", "Issue number")
  .option("-u, --user <user>", "GitHub user", "hamatti")
  .option("-r, --repository <repository>", "GitHub repository", "public-notes")
  .action((issue, options) => {
    console.log({ issue, options });
  });

program.parse();

Defining defaults on CLI option level is a nice touch.

Running the help function provides great usage documentation out of the box:

➜ node _scripts/public_notes.js help fetch  
Usage: GitHub public notes downloader fetch [options] <issue>

Fetch issue and its comments and store them as JSON

Arguments:
  issue                          Issue number

Options:
  -u, --user <user>              GitHub user (default: "hamatti")
  -r, --repository <repository>  GitHub repository (default: "public-notes")
  -h, --help                     display help for command

And when I run it, I get nice values to work with:

➜ node _scripts/public_notes.js fetch 1     
{
  issue: '1',
  options: { user: 'hamatti', repository: 'public-notes' }
}

[Hamatti]

Next up, I converted my messy Python script into a bunch of Javascript functions that do what I want it to do.

First, I modified the action to call main:

.action(async (issue, options) => {
    await main(issue, options);
  });

and then built the actual script. It first fetches the issue itself, then all the comments, adds the comments to the issue object and saves that to _data/public-notes where Eleventy is able to see them and can be used to build the actual pages.

const fs = require("fs");
const path = require("path");

async function main(issueNumber, { user, repository }) {
  const issue = await getIssue(issueNumber, user, repository);
  const comments = await getComments(issue.comments_url);

  issue.commentThread = comments;

  saveToJSON(issue, issueNumber, { user, repository });
  return { issue, comments };
}

async function getIssue(issueNumber, user, repository) {
  const url = `https://api.github.com/repos/${user}/${repository}/issues/${issueNumber}`;

  const response = await fetch(url);
  const issue = await response.json();
  return issue;
}

async function getComments(commentUrl) {
  const response = await fetch(commentUrl);
  const comments = await response.json();
  return comments;
}

function saveToJSON(issue, issueNumber, { user, repository }) {
  const filename = `public-note-${user}-${repository}-${issueNumber}.json`;
  const outputPath = path.join("_data", "public-notes", filename);

  fs.writeFileSync(outputPath, JSON.stringify(issue));
}

That was smooth operation and I really enjoy the commander.js's API. Reading it in the file gives me confidence to understand what it does and how to modify it and running it gives me good instructions for missing or misconfigured items.

[Hamatti]

When I originally created my Notion integration last August, I had an idea to download either by id or the latest post. But that process did not finish so I had an unfinished flag for latest that was hard coded not to do anything.

My actual usage of the script was as follows:

npm run notion | grep -iB1 <query>

where the first part would print out all the titles and ids and then grep would only find the lines that has <query>. I would then copy-paste the ID from that output and run

npm run notion <id>

to actually download the post.

I wanted to get rid of that grepping as it got annoying to write and I feel it should be part of the interface.

program
  .name("Notion CMS")
  .description("Manage blog posts with Notion CMS")
  .version("1.1.0");

program
  .command("query")
  .description("Find Notion ids based on title query")
  .argument("<query>", "What to search for?")
  .action(async (query) => {
    await queryNotion(query);
  });

program
  .command("fetch")
  .description("Fetch a blog post from Notion")
  .argument(
    "[notion-id]",
    "Notion id. Search for ids with query <search-query>"
  )
  .option("--latest", "Fetch latest post")
  .action(async (notionId, options) => {
    await fetchBlogPost(notionId, options);
  });

program.parse();

I created two commands: one to query to find an idea and another to fetch the post. And added a --latest flag that actually does something.

For this one, I also separated functionality for command line interface into its own file and imported only the entry points to all the functions that query, download and write blog posts.

I already had this in package.json as

"notion": "node _scripts/notion.js",

so I can run

npm run notion query python    # search any post with "python" in title

npm run notion fetch <id>      # download post with <id>
npm run notion fetch --latest  # download post that was edited latest

npm run notion fetch           # If ran without either, will result in error

I'm very happy with these improvements. The CLI now is documented better, is easier to modify and the entire flow is way nicer.