Hydrogen CLI

[cartogram]

Summary

hydrogen/cli provides interactive project scaffolding for hydrogen apps.

Reference implementations:

• [ci] release 2023-01 hydrogen#445
• [ci] release 2023-01 hydrogen#481

Motivation

• Provide developers an effortless way to start building on Hydrogen with all their favourite tools (jest, eslint, typescript, styled-components, etc...) configured and ready to go.
• Provide contributors an automated mechanism to test changes to Hydrogen across a matrix of uniquely configured apps.

If you prefer to see a demo before reading on, you can watch my initial recording in videobin.

Problems with existing solutions

If we take a look at Next.js, they provide a large examples directory showing various apps with a base Next.js structure plus one tool (with-jest, with-eslint, etc...). Some glaring limitations exist in this approach and this is why I think we shouldn't follow suit.

1. Maintenance: the toil of upgrading and testing examples when there is a core dependency, such as React or Vite, is prohibitive.
2. Combinations: While easily deployable and runnable, these examples only provide one of tools that they would ideally start with and leaves developers on their own to stitch together multiple tools.
3. Discoverability: unless the user knows what they are looking for (or even knows what all the tools are) these examples don't provide much guidance and there is no taxonomy/organization either.

I believe there is some value in having the physical examples exist (rather than generated), but we can be support and manage a smaller selection of "starter-templates" to accomplish this.

Core Framework

The core framework of hydrogen/cli consists of the following:

1. User Interface (UI): A general abstraction around all the user interface patterns. CLIs, web GUIs, VSCode plugins, etc... all share common functions (they display messages, prompt for inputs, etc...) this UI will attempt to provide an injectable common interface for these functions that the command scripts will interact with.
2. File System (FS): A manager for file system write and read operations. This should paper-over path normalization and other common file system annoyances. This component will also store an in-memory file record to optimize IO as we iteratively build up the file contents from the command scripts.
3. Workspace (WS): Provider for the context of a running command. This will store information that is project or user specific such as what dependencies they have installed and other functional or metadata attributes of a project. Just like the file system abstraction in the previous point, this provides more flexibility to build up changes iteratively.

Commands

These are the scripts that the CLI will run and are responsible for orchestrating the project scaffolding. They are simple functions that receive instances of UI, FS and WS (defined above).

Templates

In this initial iteration, templates are functions. These functions receive utility helper methods for common tasks and return markup as a string.

Next steps

The next iteration of this tool will replace these functions with static files that will be parsed into an Abstract Syntax Tree (AST) and passed through file transformations (using something like babel). This provides 2 benefits:

1. The file contents will be recognized in IDEs so we don't have to worry about formatting, syntax highlighting or syntax errors.
2. The visitor pattern used in these transforms will make it trivial to articulate and perform complex file changes.

Test Framework

The success of this tool will largely be determined by our ability to maintain it and so a robust test framework is vital. In this iteration, a command is tested in the following phases:

• Setup: An empty sandbox directory is created using the name of the specific test being run. This sandbox is injected into the test, allowing the user to build up the mock fixture files and perform other setup tasks. These setup tasks occur within the context of the test rather than in test hooks or separate fixture files, which are objectively more difficult to grep. This sandbox is wiped at the end of each test.
• Perform: Also injected into the test is a function to run the command with a unique set of prompt responses. The test invokes the command, gradually passing in the inputs to fulfil the prompts and commits the changes to the sandbox.
• Expect: After the command has finished we can inspect the sandbox and resulting project within it. This is where the user can make their assertions.

H2CLI.-.Demo.2.-.Testing.utilities.mp4

Phase 2

The next iteration of this tool goes one step further by actually installing the dependencies in the sandbox and running the project. We can then inspect the result using Playwright in a headless browser.

H2CLI.-.Demo.3.-.Browser.tests.mp4

The Future

My vision for this is far reaching and based on brief conversations with the google Aurora team, that vision for this type of developer tooling is shared. In addition to any Next Steps defined above, some other obvious features we could add are things like:

• Presets: Both standard and user defined (saved locally in a .rc file).
• Migrations: This should run on an existing project and perform migrations when we change the API of some part of Hydrogen. Frameworks commonly rely on codemods to do this today, but I think we can build in a better developer experience here.
• Auditing: More than just ESlint, we could scan every aspect of their project and flag issues. We could then define a fixer function for each issue that could be run to perform corrections.
• GUI: A hosted web based interface powered by a GraphQL API, this could be embedded in our docs.
• Deployment: This could have a tight integration with Oxygen, offering first-class support for deployment on our platform and offer support for other deployment environments.

Issues will be created for further discussion as follow up to this RFC.

Try it!

• Pull down either branch from the reference implementations
• Run yarn
• Run yarn test create to run the existing test suite.
• Run yarn hydrogen create or yarn h2 create to run the CLI yourself and generate a new app.
• Ping @cartogram if you have any problems getting this up and running.

[igrigorik]

Neat-o. I like the thinking and where this is headed. What kind of feedback are you looking for at this stage?

In the meantime, naive question: one beef I've had with many starter CLIs is that they get you going fast but then you're on your own whenever you want to change your starting state. It feels like the pattern you're proposing has the potential to be "re-entrant"... As in, I could init my repo without Tailwind do some hacking, but then go "oh wow, now I see why Tailwind is helpful" and rerun the CLI with this new option and on the existing project to drop it in? Hope that's coherent.

[cartogram]

@igrigorik yes exactly. Adding tailwind (as an example) either in an existing app or new would be the same under the hood as each would be broken up into a plugin that knows how to operate on an AST. As long as we can locate their main style sheet to add the directives @tailwind/typography, write a base tailwind.config, etc... from the CLI perspective it doesn't matter if you init with tailwind or add it 6 months down the road. This has worked well so far in my experiments, including some pretty interesting manipulations (instantiating hooks, adding props to existing components, etc...). We even use their existing prettier config to format the output and for small diff. You can dry-run as well.

But with that said, I need to prototype more use cases to see how viable this is for us to manage in the long term. What does adding sanity to an Hydrogen site look like? I18n? Bugsnag? Etc... to be more sure this model for code transformation is viable. More prototypes are for this flow are the next steps in the works. I think it will be hard for us to unring this bell if we release it, so I'm trying to be cautious here.

What kind of feedback are you looking for at this stage?

I have been focused on other tasks necessary for developer preview, but with that behind us now, I am coming back to this work. One major hesitation from me is how this meshes with our push for the best defaults for commerce. Does investing in tooling that unlocks a wider array of configuration push us away from that philosophy? Does this counter our ability to ship a well supported commerce framework? My goal here is to enable developers, but like anything, there are tradeoffs and I'd be curious if you feel those are worth accepting.

The other feedback I'd be looking for your opinion on is how this integrates with our existing Shopify CLI. I need to define what those touch points are and how we coordinate that. Do you have any thoughts on this?

[igrigorik]

But with that said, I need to prototype more use cases to see how viable this is for us to manage in the long term.

Yep. FWIW, we don't need to solve the world out of the gate. Nailing a common flow like tailwind add/remove for an existing project can be a starting point, and we can explore adding more over time.

Does investing in tooling that unlocks a wider array of configuration push us away from that philosophy? Does this counter our ability to ship a well supported commerce framework?

I don't think so. We provide a set of defaults, but it should be possible to swap them in and out. If Tailwind is not your thing and you have your own design system and CSS framework of choice, that's great. I think this reflects my earlier point: we don't need to build a fully generalized model, we should focus on paving common cowpaths for developers, and making them easy. Defaults are not about eliminating choice, they're about enabling a fast-start experience.

My goal here is to enable developers, but like anything, there are tradeoffs and I'd be curious if you feel those are worth accepting.

I think we can and should exercise opinions on what we bake in and enable by default. That's not the same as restricting developers from using what they want.

@jplhomer @morganmccunn curious to hear your take.

The other feedback I'd be looking for your opinion on is how this integrates with our existing Shopify CLI. I need to define what those touch points are and how we coordinate that. Do you have any thoughts on this?

Still forming a position on this one. I suspect @dalmaer might have relevant thoughts.

[ahmed-adly-khalil]

@cartogram this looks awesome, I want to try the CLI commands but the reference implementation links above give 404, can you share the branch names to pull it?

[cartogram]

@ahmed-adly-khalil those branches are from our earlier repository prior to going public, but you can use the hydrogen-cli now as it is public and on npm too. I have a few larger updates coming in the next few weeks and will post PRs here, really want to keep this discussion going and get your feedback.

[blittle]

A few thoughts.

1. This looks really awesome! Great work so far.
2. I tried to do just h2 create app and it isn't obvious to me how to start the project that it scaffolds. Also, it wasn't obvious to me that it would create the files within the directory I execute the command. I thought it would create a directory for me.
3. The points you make in the "motivation" and "Problems with existing solutions" really come down to a set of examples vs a fully managed application. I 100% agree that the latter is the ideal DX, when it works. The concern I have is that it can get to be very difficult to manage. Maybe something similar to h2 check could run, that would validate the project meets minimum criteria to be managed. I just imagine lots of issues coming in because people have customized their app in ways that make it impossible for us to manage nicely. It would be nice if there was a clear threshold where the developer has "ejected" from h2. And now they are on their own.
4. I love the testing framework that you have laid out. I think that will be essential to make this manageable.

[cartogram]

@blittle

I tried to do just h2 create app and it isn't obvious to me how to start the project that it scaffolds. Also, it wasn't obvious to me that it would create the files within the directory I execute the command. I thought it would create a directory for me.

Yeah I agree this should be fixed, I've been iterating on the DX as I go. You can pass a --root flag to run it inside another folder, but this still isn't great (if you don't know that flag exists, for example).

Maybe something similar to h2 check could run

Totally, I just opened a PR for how this could look. Would love your thoughts when you get a moment :)

[cartogram]

Some updates for folks following this discussion:

1. We have been contributing a number of features and bug fixes to the @shopify/ast-utilities repo to support the type of AST operations described in this discussion.

• New transform wrapJsxChildren() quilt#2086
• Adds replaceJsxBody() support for when parent node is root quilt#2085
• addImportSpecifier() prevent duplicates quilt#2081
• No duplicates in addComponentProps() quilt#2075
• Move babel-plugins to dependencies in ast-utilities package quilt#2072

2. We have a PR open that introduces check and add commands. The add command is where I imagine some core plugins will live, but also community contributions.
3. My next step is to show how this will look in practise. Will start with typescript support hydorgen add typescript and another UI library (maybe chakra) since these have been the most active discussions.

[dmi3y]

Hi, @cartogram thanks for your hard work! I see all the PRs are in the main is it good to poke around? I am particularly interested in the ability to set up an alternative library, Chakra UI. If are there any examples I can look at?

Some technical details about Chakra UI. It depends on the Emotion library, which has to be set up in a particular way for SSR.
Official Emotion docs are here. Chakra UI has an example of how to set it up with Remix here. And also the way it can be done on NextJS.