Infrastructure Guide

Monorepo Setup

Our monorepo is scaffolded with nx. The nx documentation has a lot of useful content if you're looking for a more general overview. This section contains information for key files and commands that we utilize in our monorepo setup.

Key nx Monorepo Files

Let's take a look at key files used in the nx monorepo setup.

workspace.json

• Maps to all of the projects
• This is hownx knows what to reference when we run commands such as nx run ui:build
  • The project name coming after nx rn needs to be included in this list, and the command after the colon must exist in that project's config!
• Referenced in each project'stsconfig as well
• When generating a new app or library using our generators these are updated

nx.json

• Global configuration for nx
• Don't need to touch this very often, but we did customize the targetDefaults:

 "targetDefaults": {
    "build": {
      "dependsOn": ["^build", "tsc"]
    }
  },

• When running build this also runs tsc so that we include TypeScript checks (tsc) since vite doesn't do this by default

Key App and Library Files

Let's dive a step deeper and look at key files in each project (apps and libraries).

project.json

Each project has a project.json generated at the root. This is a similar to the scripts section of a package.json file. It defines the commands and what they're executing. This is typically a plugin's executor.

Let's look at an example from the Hub app package:

"targets": {
    "build": {
      "executor": "@nxext/vite:build",

• In this example, the Hubbuild command targets the @nxext/vite:build plugin's build executor and runs that when running nx run hub-app:build
  • For a deeper understanding into what's going on behind the scenes, you can go to each plugin's code on GitHub and find the executors.json file which define the arguments for the executor
  • These args are where we get the options used in the `project.json

We utilize dependsOn and configurations in certain project's project.jsons. Let's take a look at an example from the ui library.

  "build": {
      "outputs": ["{options.outputPath}"],
      "executor": "@nrwl/workspace:run-commands",
      "options": {
        "outputPath": "dist/libs/ui",
        "command": "tsc --project libs/ui/tsconfig.lib.json --emitDeclarationOnly --skipLibCheck",
        "color": true
      },
      "dependsOn": [
        {
          "target": "vite-build",
          "projects": "self"
        },
        {
          "target": "build-storybook",
          "projects": "self"
        }
      ]
    },

• dependsOn
  • Can stack commands on top of each other
    • In the example from the ui library, in order to run build, vite-build and build-storybook need to run first
    • Any commands listed in the dependsOn array need to run successfully first before the project's build command will run

    "build-storybook": {
      "executor": "@nrwl/storybook:build",
      "outputs": ["{options.outputPath}"],
      "inputs": [{ "env": "NODE_ENV" }],
      "options": {
        "uiFramework": "@storybook/react",
        "outputPath": "dist/storybook/ui",
        "config": {
          "configFolder": "libs/ui/.storybook"
        },
        "quiet": false
      },
      "configurations": {
        "ci": {
          "quiet": true
        }
      }
    },

• configurations
  • We don't currently leverage this much, but this adds a subcommand (in the example, within storybook)
  • We use this in the build-storybook comamnd in the project.json from the ui library
• nx run ui: build-storybook:ci
  • Inheret all the config and also overwrite the quiet config value in the initial build command
• Can add configurations within each command -- such as subgraph parameters
  • Rinkeby as the key and then the options for the specific blockchain
• We'd want to chain the commands (such as build) that are used in our CI flows

CI Actions

Let's look deeper at our CI workflows. These are .yaml files located in the .github folder in the monorepo root. Each action has a different name that corresponds to the branch. For example, ci_develop.yaml is named CI Develop. Let's look at the CI Develop action - The CI Develop action is set to run on pushes directly to the develop branch and on PRs into develop

• Once the action is triggered, it creates a .env for the build. This includes a lot of key information:
• BASE branch:

  BASE: ${{ github.ref == 'refs/heads/develop' && 'origin/develop~1' || 'origin/develop' }}

- This sets the action ref as `develop` and includes any pushes directly to develop as well as PRs into `develop` (`origin/develop~1`) -- this is "1` commit below the head"
- If this isn't included the action wouldn't run on PRs into develop
- This structure enables our build action to run if we need to make a quick PR into `develop` (even though we don't typically do this) AND on PRs into `develop` (our current workflow)
- The `master` workflow is set up very similarly to run on merges directly into the branch for hotfixes as well as for our workflow of merging `develop` into `master`

• nx affected:command
  • If your code has changed relative to the base branch (BASE), then it'll run the connected command for each step
  • Every app/library should have a lint, test, and build command since these are used in our CI
    • If the app/library doesn't have a command of the same name, it won't run when affected runs -- name must be the same!
      • If the name is even slightly off, such as linter instead of lint it won't run in the CI step
• Test this with the generate commands to make sure that these exist
  • If any of these commands aren't supported we'd want to fork and customize for DH needs
    • Would be good to enforce at the code level than relying on maintaining docs about it
• The nx docs have additional information about how affected works

Secrets and Environment Variables

When the actions run, they create a .env with the environment variables we need and are configured to use ones stored within our GitHub Secrets.

• These are prefixed with secrets. -- this indicates that the secrets are safely stored in our GitHub Secrets
• It's critical to make sure that when we're adding new environment variables that we add them to the GitHub Secrets and to the Actions yaml file. If we don't do this, the env variables will be undefined at build and runtime!

Deployments and Services

We use Skynet Labs for decentralized hosting of our apps (Hub, Summoner, and Admin). Sia is like IPFS (where the files are pinned to) and Skynet is similar to Infura or Pinata. Skynet is a separate service that we're using. - Currently, pushes to develop trigger our build and deployment workflow whereas pushes to master are utilized for publishing to npm

• Our Actions build our apps and then deploy directly to Skynet for hosting
• Each time the build successfully runs there will be a corresponding entry in the Uploads list on the Skynet Dashboard
  • Name is the app name, such as hub-app
  • Skylink is a hash of the file itself
    • Every time the build files change (which is upon a successful build) there will be a new hash for that app
• The total storage we're allowed is configured across all apps connected with our account key, not on a per app basis

Skynet Portals and DNS

• Skynet contains portals, which are entry points into the network
  • We currently use siasky.net (this is a public portal)
    • We may need to switch to a different portal depending on the upcoming changes to Skynet Labs due to their recent sustainability announcement in August 2022
• For DNS, there is a useful tutorial in the Skynet Docs

Troubleshooting

• Refer to vite docs for frontend packaging and configuration explanations
• Forking and modifying nx generators is a helpful approach for deepending understanding
• Recognize when we're going down a bad rabbithole:
  • Indicators would be heavy changes to the vite config
  • Adding a bunch of polyfills to solve problems
  • Heavy modifications to the package.json outside of the defaults
  • Try to keep as much in the monorepo as possible
• Try to reproduce issues locally when troubleshooting building

Running Builds Locally

• You can test out the build command locally by running nx run project-name:build (such as nx run hub-app:build)
  • You can also use this to test individual steps in the build such as nx run ui:build-storybook
• If you're debugging errors on a deployed app you can test this locally beyond running the build command with the following steps (using Hub as an example):
  • Run the associated build command: nx run hub-app:build
  • Navigate into the output folder (default is dist unless changed in the configuration)
  • Serve this folder directly with an HTTP client and navigate to the provided URL
    • This can be done in VSCode by opening the hub-app found in the dist folder and then running VSCode's Live Server and heading to the served URL
    • This is essentially the process that the deployment workflow uses so you're able to troubleshoot for errors that may be occurring after build and deployment
  • Going through this process is helpful to debug and catch issues happening after build and deployment