Split client and server into separate packages (#223)

.eslintrc.json

@@ -1,5 +1,7 @@
 {
   "extends": ["react-app", "airbnb", "plugin:prettier/recommended"],
+  // unlike .eslintignore, these values will cascade
+  "ignorePatterns": ["node_modules/", "build/", "public/"],
   "rules": {
     "react/jsx-filename-extension": ["error", { "extensions": [".js"] }],
     "import/no-extraneous-dependencies": ["error", { "packageDir": "./" }],
@@ -13,17 +15,5 @@
     // these rules conflict with prettier; let prettier win
     "react/jsx-one-expression-per-line": 0,
     "react/jsx-curly-newline": 0
-  },
-  "env": {
-    "browser": true,
-    "jest": true
-  },
-  "overrides": [
-    {
-      "files": ["./server.js", "./server/**/*.js"],
-      "env": {
-        "node": true
-      }
-    }
-  ]
+  }
 }

.github/workflows/public-dashboard-client-ci.yml

@@ -0,0 +1,31 @@
+name: "CI: Public Dashboard API"
+
+on: push
+
+defaults:
+  run:
+    working-directory: public-dashboard-client
+
+jobs:
+  lint:
+    name: lint
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v1
+      - uses: actions/setup-node@v1.1.0
+        with:
+          node-version: "12.x"
+      - uses: c-hive/gha-yarn-cache@v1
+      - run: yarn install
+      - run: yarn lint
+  test:
+    name: test
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v1
+      - uses: actions/setup-node@v1.1.0
+        with:
+          node-version: "12.x"
+      - uses: c-hive/gha-yarn-cache@v1
+      - run: yarn install
+      - run: yarn test

.github/workflows/spotlight-api-ci.yml

@@ -0,0 +1,20 @@
+name: "CI: Spotlight API"
+
+on: push
+
+defaults:
+  run:
+    working-directory: spotlight-api
+
+jobs:
+  lint:
+    name: lint
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v1
+      - uses: actions/setup-node@v1.1.0
+        with:
+          node-version: "12.x"
+      - uses: c-hive/gha-yarn-cache@v1
+      - run: yarn install
+      - run: yarn lint

.gitignore

@@ -12,11 +12,11 @@
 
 # Production build directory
 build/
-/.firebase
+.firebase
 
 # Dependency directory
 node_modules/
-/.pnp
+.pnp
 .pnp.js
 
 # Code coverage directory
@@ -26,7 +26,6 @@ coverage/
 auth_config.json
 auth_config*.json
 gae*.yaml
-src/assets/scripts/metrics/*.json
 recidiviz-production-*.json
 recidiviz-dev-*.json
 

README.md

@@ -1,4 +1,20 @@
-This project was bootstrapped with [Create React App](https://github.com/facebook/create-react-app).
+# Spotlight
+
+This repository contains npm packages related to our Spotlight data publishing product, as well as some shared configuration and tooling that involves multiple packages.
+
+![CI: Public Dashboard API](https://github.com/Recidiviz/public-dashboard/workflows/CI:%20Public%20Dashboard%20API/badge.svg) ![CI: Spotlight API](https://github.com/Recidiviz/public-dashboard/workflows/CI:%20Spotlight%20API/badge.svg)
+
+## Packages in this repository
+
+More information about how to use each individual package can be found in their respective README files.
+
+### [Public Dashboard Client](public-dashboard-client/)
+
+A React application for the Spotlight website
+
+### [Spotlight API](spotlight-api/)
+
+A thin Node/Express backend that serves data for the Spotlight website.
 
 ## Development
 
@@ -20,120 +36,43 @@ First, build the app locally.
 
    `yarn install`
 
-That's it! We suggest installing a linting package for your preferred code editor that hooks into [eslint](#eslint). We recommend [linter-eslint](https://atom.io/packages/linter-eslint) if you're using Atom.
-
-At this point, you should be able to test your development environment via:
-
-    `yarn test`
-    `yarn lint`
-
-#### Environment variables
-
-Second and last, set up your environment variables.
-
-For the frontend: copy the `.env.frontend.example` file and set variables accordingly per environment. At the moment, the app is deployed to both staging and production environments. Staging relies on environment variables stored in `.env.development` and production relies on variables in `.env.production`. Local relies on `.env.development.local`.
-
-Expected frontend environment variables include:
-
-- `REACT_APP_API_URL` - the base URL of the backend API server. This should be set to http://localhost:3001 by default when running locally.
-- `REACT_APP_IS_DEMO (OPTIONAL)` - whether or not to run the frontend in demo mode, which will run the app without attempting to reach remote metric files. This should only be set when running locally and should be provided through the command line, along with the backend sibling below. To run the app in demo mode, use the following command: `./run_in_demo_mode.sh`
-
-The build process, as described below, ensures that the proper values are compiled and included in the static bundle at build time, for the right environment.
-
-For the backend: copy the `.env.backend.example` file into `.env` and set variables appropriate for your local environment. Set these same variables in your Google App Engine yaml files, if deploying to GAE. Those files are described later on.
-
-Expected backend environment variables include:
-
-- `GOOGLE_APPLICATION_CREDENTIALS` - a relative path pointing to the JSON file containing the credentials of the service account used to communicate with Google Cloud Storage, for metric retrieval.
-- `METRIC_BUCKET` - the name of the Google Cloud Storage bucket where the metrics reside.
-- `IS_DEMO` (OPTIONAL) - whether or not to run the backend in demo mode, which will retrieve static fixture data from the `server/core/demo_data` directory instead of pulling data from dynamic, live sources. This should only be set when running locally and should be provided through the command line, along with the frontend sibling above. To run the app in demo mode, use the following command: `./run_in_demo_mode.sh`
-
-### Running the application locally
-
-A yarn script is available for starting the development servers. The React frontend is served out of port `3000` and the Node/Express backend is served out of port `3001`. This will also automatically open a browser to localhost on the appropriate port, pointing to the frontend.
-
-`yarn dev`
-
-The development servers will remain active until you either close your terminal or shut down the entire setup at once using `control+c`.
-
-**Note:** The frontend server does not need to be restarted when frontend source code is modified. The assets will automatically be recompiled and the browser will be refreshed. The same is true true for the backend server, except in the case of changing fixture data in `/server/core/demo_data`.
-
-### Demo mode
-
-When running locally, you can run the app in demo mode to point the app to static data contained in `server/core/demo_data`. This is useful for debugging issues that materialize under specific data circumstances, for demonstrating the tool without exposing real data, for development when you don't have Internet access, and other use cases.
-
-You can launch in demo mode locally via: `./run_in_demo_mode.sh`
-
-Running via that command is important because environment variables are required for both the frontend and backend servers. Running with only one or the other in demo mode produces a fairly broken experience.
-
-## Deploys
-
-As noted above, the Dashboard is two components: a React frontend and a Node/Express backend providing a thin API. The app can be run locally, in staging, and in production. Deploying to staging and production are very similar, as described below.
-
-### Pre-requisites
-
-The frontend of the app is deployed to Firebase. To have deploy access, you need to be an admin on the frontend Firebase/GCP account.
-
-Once you have the required permissions, you can set up your environment for deploys by following [these instructions](https://firebase.google.com/docs/cli?install-cli-mac-linux). Specifically, follow the steps entitled "Install the Firebase CLI" and "Log in and test the Firebase CLI."
-
-The backend of the app is deployed to Google App Engine. Similarly, to have deploy access, you need to be an admin on the backend GCP account.
-
-Once you have the required permissions, you can set up your environment for deploys by following [these instructions](https://cloud.google.com/appengine/docs/standard/nodejs/setting-up-environment).
-
-### Deploying to Staging
-
-#### Frontend
-
-To generate a staging build of the frontend, invoke the following yarn script: `yarn build-staging`.
-
-Each time this is run, the `/build` directory will be wiped clean. A [bundle analysis](#Bundle-analysis) report, found in `build/report.html`, will also be generated on each invocation of this script. This will include the appropriate environment variables from `.env.development`.
-
-You should then test this locally by running `firebase serve`: it will run the staging build locally, pointed to the staging API backend--if you also have backend changes, deploy the backend as described in the next subsection. When you're satisfied, deploy the frontend to staging with `firebase deploy -P staging`. Test vigorously on staging before deploying to production.
-
-#### Backend
-
-We deploy the backend to Google App Engine with configured yaml files. Copy the `gae.yaml.example` file into files named `gae-staging.yaml` and `gae-production.yaml`, and set environment variables accordingly.
-
-Deploy the backend to staging Google App Engine with `gcloud app deploy gae-staging.yaml --project [project_id]`. This will upload any updated backend code/configuration to GAE and start the server (GAE runs `npm start` only once the deploy succeeds and is stable). Test vigorously on staging before continuing to production.
-
-### Deploying to Production
+That's it! We suggest installing a linting package for your preferred code editor that hooks into eslint. (Note that you may need to configure your linting plugin to use multiple package directories.)
 
-Follow the instructions described above, but with different commands for both frontend and backend deploys.
+Individual packages may require some additional configuration to work properly; refer to their respective README files for more information.
 
-Generate a production build of the frontend with `yarn build`. Test locally with `firebase serve`. Deploy the frontend with `firebase deploy -P production`.
+### Multi-package tools
 
-Deploy the backend to production GAE with `gcloud app deploy gae-production.yaml --project [project_id]`.
+We use [Yarn Workspaces](https://classic.yarnpkg.com/en/docs/workspaces/) for some light multi-package management on top of standard Yarn commands.
 
-Test vigorously! Don't be afraid to rollback the deploy of frontend or backend through the Firebase and GAE consoles.
+This means that running `yarn install` in the root or in any package directory will install the dependencies for all packages in the repository — so you only need to do it once no matter how many packages you wish to use.
 
-## Available Scripts
+It also allows you to run a Yarn command sequentially in each package with `yarn workspaces <command>` or individually with `yarn workspace <workspace-name> <command>` as an alternative to setting your working directory to the package's directory before executing it. For example, if you wanted to run lint tests for the entire repository, you could use `yarn workspaces run lint` to run the `lint` script in each package.
 
-Besides the scripts mentioned above for running and deploying the app, you can also run:
+In addition, there are some Yarn scripts defined in the root package as a convenience for running multiple packages in a coordinated fashion:
 
-### `yarn test`
+`yarn dev:pd` — starts the development servers for the Spotlight API and the Public Dashboard Client in a single terminal window. Be sure you have configured each of those applications with the necessary environment variables, as described in their README files, or this will not work!
 
-Launches the test runner in the interactive watch mode.
+`yarn demo:pd` — starts the development servers for the Spotlight API and the Public Dashboard client in "demo mode" by supplying the necessary environment variables in the command line. (This will run your local site off of fake data fixtures rather than live data from a calculation pipeline; see the [Spotlight API documentation](spotlight-api/#demo-mode) for more information.)
 
-We use [`@testing-library/react`](https://testing-library.com/docs/react-testing-library/intro)
-for component tests you will want to import the testing-library functions via `src/testUtils.js`
-instead; it re-exports the full API from `@testing-library/react` with a wrapper around `render`
-that includes any globally expected React Context providers (e.g. the `ThemeProvider` for `styled-components`).
+### Other tools
 
-See the section about [running tests](https://facebook.github.io/create-react-app/docs/running-tests) for more information.
+Style and formatting rules for this repository are defined with [ESLint](https://eslint.org/) and [Prettier](https://prettier.io/). The base configuration lives in the root of this repository and is extended by the individual packages as necessary. For this reason, it is important to run the individual `lint` commands for each package rather than trying to lint the entire repository at once with `eslint .` — this will exclude the nested configurations and produce inconsistent results.
 
-### `yarn lint`
+Linting rules (including auto-fixing) are applied to changed files in a pre-commit hook using [Husky](https://github.com/typicode/husky) and [lint-staged](https://github.com/okonet/lint-staged). These are configured in the root `package.json`.
 
-Runs the eslint checks against the repository to check for issues in code style.
+We use [Github Actions](https://docs.github.com/en/free-pro-team@latest/actions) for continuous integration tasks, which include lint checks as well as automated JavaScript tests where applicable.
 
-Eslint rules are configurable in `.eslintrc.json`. Any change to this file should be accompanied with an explanation
-for the change and why it should be merged.
+## Adding new packages
 
-### `yarn eject`
+Packages themselves do not have any special requirements under Yarn Workspaces, nor are they required to use the same packages or versions as any other Workspace; just create a new directory, **add it to the `workspaces` list in the root `package.json`**, and then run `yarn init` in your package directory as you normally would. (The entire repo is covered by a GPL V3 license, so make sure your package's `license` field conforms to that.)
 
-**Note: this is a one-way operation. Once you `eject`, you can’t go back!**
+Do note that there should only be one `yarn.lock` file for the entire repository; if one is created in your new package directory, you should delete it, make sure the directory is properly listed in the `workspaces` list, and re-run `yarn install`.
 
-If you aren’t satisfied with the build tool and configuration choices, you can `eject` at any time. This command will remove the single build dependency from your project.
+ESLint and lint-staged should be installed and applied to your new package directory automatically based on the configurations in the root directory. If you need to extend the default ESLint configuration, create a configuration file in your package directory; if you need to modify or add lint-staged rules, you can do so in the root `package.json`.
 
-Instead, it will copy all the configuration files and the transitive dependencies (webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except `eject` will still work, but they will point to the copied scripts so you can tweak them. At this point you’re on your own.
+In addition, there are some conventions you should follow when setting up your new package, unless you have a compelling and well-documented reason not to:
 
-You don’t have to ever use `eject`. The curated feature set is suitable for small and middle deployments, and you shouldn’t feel obligated to use this feature. However we understand that this tool wouldn’t be useful if you couldn’t customize it when you are ready for it.
+- The package name should be the same as the directory name
+- `yarn dev` should execute the main entry point for development (e.g., starting a development server)
+- `yarn lint` should run your lint tests
+- `yarn test` should run your JS tests, if you have any (which you should!)