add "overview" sections to package READMEs

README.md

@@ -59,11 +59,11 @@ Your code editor may need some additional configuration to properly integrate wi
 #### Visual Studio Code
 
 - install the [ESLint plugin](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint), and in your Workspace settings, set the ESLint Working Directories to an array of all the package directories.
-- install the [Jest plugin](https://marketplace.visualstudio.com/items?itemName=Orta.vscode-jest), and in your Workspace settings, set the "Jest command line" to `yarn test --`
+- install the [Jest plugin](https://marketplace.visualstudio.com/items?itemName=Orta.vscode-jest), and in your Workspace settings, set the "Jest command line" to `yarn test` (or one of the package-specific commands if you don't want to lint all packages; see `package.json` for details)
 
 ### Other tools
 
-Style and formatting rules for this repository are defined with [ESLint](https://eslint.org/) and [Prettier](https://prettier.io/). The base configuration lives in the root of this repository and is extended by the individual packages as necessary. For this reason, it is important to run the individual `lint` commands for each package rather than trying to lint the entire repository at once with `eslint .` — this will exclude the nested configurations and produce inconsistent results.
+Style and formatting rules for this repository are defined with [ESLint](https://eslint.org/) and [Prettier](https://prettier.io/). Each package maintains its own configuration, and there is no central one, and thus no single command that will lint the entire repository.
 
 Linting rules (including auto-fixing) are applied to changed files in a pre-commit hook using [Husky](https://github.com/typicode/husky) and [lint-staged](https://github.com/okonet/lint-staged). These are configured in the root `package.json`.
 

spotlight-api/README.md

@@ -1,6 +1,27 @@
 # Spotlight API
 
-This package is a Node/Express server application that provides a thin API backend for clients consuming public metrics from the Recidiviz data pipeline.
+This package is a Node/Express application that provides an API server for the `spotlight-client` web application package.
+
+## Application overview
+
+The Spotlight API is a thin backend that exposes metric data from the Recidiviz data platform to the open web. It has no access controls and therefore should traffic no data that is not fit for public consumption (in terms of privacy, security, confidentiality, etc.). Spotlight is a public-facing web application; while there are some access controls within the corresponding frontend application to allow for some privacy in a pre-release staging environment, you should exercise appropriate care when releasing new versions of this package.
+
+### Metric data
+
+This application consumes exported metric views from the Recidiviz data platform, which are just flat files retrieved from Google Cloud Storage. It passes these files through virtually untouched and generally strives to know as little about their contents as possible; all transformation logic — beyond translating them from JSON-Lines to proper JSON — is owned downstream by the frontend.
+
+Metric files are fetched from a designated bucket, specified per environment as discussed below. This allows us to maintain separate staging and production files. To enable a new Spotlight metric, a new metric view be created in the main Recidiviz platform application and exported to these buckets, and then that metric file must be registered in this application by adding it to the list maintained in `core/metricsApi`. This application should never serve an unregistered file to a client, even if it exists in the storage bucket.
+
+### Caching
+
+This application does some light in-memory caching, as the metric data is generally only updated once a day. In hotfix scenarios or others where the cache must be forcibly cleared, the deployed instances must be shut down and restarted. The easiest way to do this is generally to just redeploy the same version that's currently live.
+
+### Fixtures
+
+The data fixtures in `core/demo_data/` serve two purposes:
+
+1. They are served by this application when it is run in Demo Mode, in lieu of files fetched from GCS (as discussed below).
+1. They are consumed by various tests in the `spotlight-client` package that require realistic metric data. This is also accomplished by spinning up a server in Demo Mode, but is more sensitive to changes in file contents.
 
 ## Development
 
@@ -29,11 +50,11 @@ Expected backend environment variables include:
 
 `yarn dev` will run a development API server locally on port `3001`. The development server will remain active until you either close your terminal or shut it down using `control+c`.
 
-**Note:** The server does not need to be restarted when source code is modified. The assets will automatically be recompiled and the browser will be refreshed — except in the case of changing fixture data in `/server/core/demo_data` while running in demo mode (as described below).
+**Note:** The server does not need to be restarted when source code is modified. The assets will automatically be recompiled and the browser will be refreshed — except in the case of changing fixture data in `core/demo_data` while running in demo mode (as described below).
 
 ### Demo mode
 
-When running locally, you can run the app in demo mode to point the app to static data contained in `server/core/demo_data`. This is useful for debugging issues that materialize under specific data circumstances, for demonstrating the tool without exposing real data, for development when you don't have Internet access, and other use cases.
+When running locally, you can run the app in demo mode to point the app to static data contained in `core/demo_data`. This is useful for debugging issues that materialize under specific data circumstances, for demonstrating the tool without exposing real data, for development when you don't have Internet access, and other use cases. (Demo mode is not perfect, as some parts of the frontend expect historical data relative to the current date, which this application does not provide.)
 
 You can launch in demo mode by running `yarn demo`. Be sure your client application has its API url set to `localhost:3001` to consume this demo data! (For convenience, there is also [a helper script](../README.md#multi-package-tools) in the root package for running both client and server together with all proper settings.)
 
@@ -47,7 +68,7 @@ This application is deployed to Google App Engine. To have deploy access, you ne
 
 Once you have the required permissions, you can set up your environment for deploys by following [these instructions](https://cloud.google.com/appengine/docs/standard/nodejs/setting-up-environment).
 
-The Recidiviz environment configuration settings (including GCP project IDs) are not checked in, but if you need access to them (i.e., you are doing development work for Recidiviz and you have GCP admin access), just ask a member of the Recidiviz engineering staff to help you out!
+The Recidiviz environment configuration settings (including GCP project IDs) are not checked in, but if you need access to them (i.e., you are doing development work for Recidiviz and you have GCP admin access), just ask a member of the Recidiviz engineering staff to help you out! They are stored in a shared archive in the company password manager.
 
 ### Deploying to Staging
 

spotlight-client/README.md

@@ -1,6 +1,36 @@
 # Spotlight Client
 
-This package is a React client application for the next-generation Spotlight public data publishing website (not yet launched). It was bootstrapped with [Create React App](https://github.com/facebook/create-react-app) and is written in [TypeScript](https://www.typescriptlang.org/docs).
+This package is a React client application for the Spotlight public data publishing website. It was bootstrapped with [Create React App](https://github.com/facebook/create-react-app) and is written in [TypeScript](https://www.typescriptlang.org/docs).
+
+## Application overview
+
+The Spotlight Client is a single-page web application built with React; most of what you will find in `src/` is React components, organized by feature. The application also contains configuration files, a Metrics API client, and data models, all of which are discussed further below.
+
+The site consumes views of aggregate population data produced by the Recidiviz data platform (referred to here as "Metrics") and organizes them into thematic pages known as "Narratives," each of which contains an ordered set of sections displaying data visualizations and explanatory copy; it is not a "dashboard," really, although that label may be seen sometimes for mostly historical reasons.
+
+While Spotlight is developed and deployed as a single multi-tenant website, it is primarily consumed as separate single-tenant experiences, under `.gov` subdomains owned by our state partners (e.g. [dashboard.docr.nd.gov](https://dashboard.docr.nd.gov)). To keep our infrastructure simple, this "tenant lock" is implemented in application logic within the data models, based on the URL hostname at runtime. This is why the multi-tenant "homepage" is so plain; it is really only used internally, for convenience, in development and staging environments.
+
+### Configuration and content
+
+At its core this application is driven by a set of configuration objects, which are JavaScript objects that determine which states (or "Tenants") are displayed; which Narratives and Metrics will appear for each Tenant and what copy will appear on each of those pages (all of which is collectively referred to here as "Content"); and various other settings that can be changed per Tenant.
+
+There is one configuration file per state, each containing a single configuration object. These files, along with supporting logic, are found in `src/contentApi/`; more information about how to use these files can be found in the [Content README](src/contentApi/README.md).
+
+### The Metrics API
+
+The API client, found in `src/metricsApi/`, is the counterpart to the `spotlight-api` package, which runs our server application. In addition to fetching the metric data, it transforms the raw response contents into strongly typed "Records". These are more generic than the raw responses; multiple metrics may be mapped to the same underlying Record type, which in turn allows our data models to be more generic and makes it easier to visualize different metrics the same way. There is not necessarily a 1:1 correspondence between Record type and chart form, but they are strongly correlated.
+
+**UI components should never be interacting directly with the Metrics API!** This responsibility is handled by the data models, which are in turn consumed by UI components.
+
+### Data models
+
+This application uses [MobX](https://mobx.js.org/README.html) for state management; if you are unfamiliar with this package you should definitely spend some time studying their documentation, as it is quite different from other popular React state management libraries such as React Contexts or Redux. The [TL;DR](https://mobx.js.org/the-gist-of-mobx.html) on it, though, is that it is an object-oriented and reactive framework that encourages developers to centralize application state, decoupling it from the UI entirely, and to treat the UI as a side effect derived from that state. This stands in contrast to other libraries that favor a more functional style and map more closely to React's own internal rendering and state management features.
+
+To that end, state is owned by a group of MobX [data stores](https://mobx.js.org/defining-data-stores.html), found in `src/DataStore`. The `TenantStore` is the one mainly concerned with the content and metrics described above; it owns a set of domain objects, which are Mobx observable classes defined in `src/contentModels` that consume both the configuration objects and responses from `spotlight-api`.
+
+Each Record type defined by the API client has a corresponding Metric class; these classes use the API client to fetch data and own their own filtering and transformation logic. Filters are applied per metric (which, in practice, means per section on a given Narrative page). They are all extensions of the abstract `Metric` class, so start there if you want to understand their inner workings more deeply (e.g. to develop a new Metric type).
+
+We strive to do as much work as possible in these models and as little as possible in the UI components; generally speaking, the UI components connect the filters to their corresponding UI controls, and own the knowledge about how to translate from our relatively generic and descriptive record formats to the APIs of our visualization components (which wrap an external chart library). As much as possible, anything else that isn't purely display logic should be lifted up into the models and data stores.
 
 ## Development
 
@@ -110,7 +140,3 @@ You can also run either TS or ESLint individually; while there are not predefine
 **Note: this is a one-way operation. Once you `eject`, you can’t go back!**
 
 This package was bootstrapped with Create React App, which provides the option to `eject` its build tooling and configuration, allowing for full customization. See [the Create React App docs](https://create-react-app.dev/docs/available-scripts#npm-run-eject) for more information.
-
-## Adding new Tenants
-
-In addition to data being available from `spotlight-api`, adding a new tenant to the site also requires content for that tenant to be added to the Content API (which is included in the JS bundle, not served by the backend). See the [Content README](src/contentApi/README.md) for more information.