Merge branch 'main' of https://github.com/Recidiviz/public-dashboard …

…into nasaownsky/480-small-numbers

.github/workflows/main.yml

@@ -27,7 +27,7 @@ jobs:
         working-directory: spotlight-client
     env:
       REACT_APP_API_URL: http://localhost:3002
-      REACT_APP_ENABLED_TENANTS: US_ND,US_PA
+      REACT_APP_ENABLED_TENANTS: US_ID,US_ME,US_ND,US_PA,US_TN
     steps:
       - uses: actions/checkout@v2
       - uses: actions/setup-node@v1

.github/workflows/nightfall.yml

@@ -0,0 +1,20 @@
+name: nightfalldlp
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+jobs:
+  nightfalldlp:
+    name: nightfalldlp
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Repo Action
+        uses: actions/checkout@v2
+
+      - name: nightfallDLP action step
+        uses: nightfallai/nightfall_dlp_action@v2.0.1
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          NIGHTFALL_API_KEY: ${{ secrets.NIGHTFALL_API_KEY }}
+          EVENT_BEFORE: ${{ github.event.before }}

.github/workflows/spotlight-staging.yml

@@ -30,7 +30,7 @@ jobs:
           REACT_APP_AUTH_ENABLED: true
           REACT_APP_AUTH_ENV: development
           REACT_APP_API_URL: ${{ secrets.REACT_APP_API_URL }}
-          REACT_APP_ENABLED_TENANTS: US_ND,US_PA
+          REACT_APP_ENABLED_TENANTS: US_ID,US_ME,US_ND,US_PA,US_TN
 
         run: yarn build
       - name: Store build artifact

.nightfalldlp/config.json

@@ -0,0 +1 @@
+{ "detectionRuleUUIDs": ["c414b0c0-f92f-4ebb-ba37-2e928b7be924"] }

README.md

@@ -59,11 +59,11 @@ Your code editor may need some additional configuration to properly integrate wi
 #### Visual Studio Code
 
 - install the [ESLint plugin](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint), and in your Workspace settings, set the ESLint Working Directories to an array of all the package directories.
-- install the [Jest plugin](https://marketplace.visualstudio.com/items?itemName=Orta.vscode-jest), and in your Workspace settings, set the "Jest command line" to `yarn test --`
+- install the [Jest plugin](https://marketplace.visualstudio.com/items?itemName=Orta.vscode-jest), and in your Workspace settings, set the "Jest command line" to `yarn test` (or one of the package-specific commands if you don't want to lint all packages; see `package.json` for details)
 
 ### Other tools
 
-Style and formatting rules for this repository are defined with [ESLint](https://eslint.org/) and [Prettier](https://prettier.io/). The base configuration lives in the root of this repository and is extended by the individual packages as necessary. For this reason, it is important to run the individual `lint` commands for each package rather than trying to lint the entire repository at once with `eslint .` — this will exclude the nested configurations and produce inconsistent results.
+Style and formatting rules for this repository are defined with [ESLint](https://eslint.org/) and [Prettier](https://prettier.io/). Each package maintains its own configuration, and there is no central one, and thus no single command that will lint the entire repository.
 
 Linting rules (including auto-fixing) are applied to changed files in a pre-commit hook using [Husky](https://github.com/typicode/husky) and [lint-staged](https://github.com/okonet/lint-staged). These are configured in the root `package.json`.
 

spotlight-api/.env.example

@@ -1,2 +1,3 @@
 GOOGLE_APPLICATION_CREDENTIALS={PATH_TO_CREDENTIALS_FILE}
 METRIC_BUCKET={GCS_BUCKET_NAME}
+AUTH_ENABLED={AUTH_ENABLED}

spotlight-api/README.md

@@ -1,6 +1,27 @@
 # Spotlight API
 
-This package is a Node/Express server application that provides a thin API backend for clients consuming public metrics from the Recidiviz data pipeline.
+This package is a Node/Express application that provides an API server for the `spotlight-client` web application package.
+
+## Application overview
+
+The Spotlight API is a thin backend that exposes metric data from the Recidiviz data platform to the open web. It has no access controls and therefore should traffic no data that is not fit for public consumption (in terms of privacy, security, confidentiality, etc.). Spotlight is a public-facing web application; while there are some access controls within the corresponding frontend application to allow for some privacy in a pre-release staging environment, you should exercise appropriate care when releasing new versions of this package.
+
+### Metric data
+
+This application consumes exported metric views from the Recidiviz data platform, which are just flat files retrieved from Google Cloud Storage. It passes these files through virtually untouched and generally strives to know as little about their contents as possible; all transformation logic — beyond translating them from JSON-Lines to proper JSON — is owned downstream by the frontend.
+
+Metric files are fetched from a designated bucket, specified per environment as discussed below. This allows us to maintain separate staging and production files. To enable a new Spotlight metric, a new metric view must be created in the main Recidiviz platform application and exported to these buckets, and then that metric file must be registered in this application by adding it to the list maintained in `core/metricsApi`. This application should never serve an unregistered file to a client, even if it exists in the storage bucket.
+
+### Caching
+
+This application does some light in-memory caching, as the metric data is generally only updated once a day. In hotfix scenarios or others where the cache must be forcibly cleared, the deployed instances must be shut down and restarted. The easiest way to do this is generally to just redeploy the same version that's currently live.
+
+### Fixtures
+
+The data fixtures in `core/demo_data/` serve two purposes:
+
+1. They are served by this application when it is run in Demo Mode, in lieu of files fetched from GCS (as discussed below).
+1. They are consumed by various tests in the `spotlight-client` package that require realistic metric data. This is also accomplished by spinning up a server in Demo Mode, but is more sensitive to changes in file contents.
 
 ## Development
 
@@ -23,17 +44,18 @@ Expected backend environment variables include:
 
 - `GOOGLE_APPLICATION_CREDENTIALS` - a relative path pointing to the JSON file containing the credentials of the service account used to communicate with Google Cloud Storage, for metric retrieval.
 - `METRIC_BUCKET` - the name of the Google Cloud Storage bucket where the metrics reside.
+- `AUTH_ENABLED` - whether or not we should require authentication to access our endpoints. Currently only used in staging to make the entire site private. No need to enable this locally unless you are developing or testing something auth-related.
 - `IS_DEMO` (OPTIONAL) - whether or not to run the backend in demo mode, which will retrieve static fixture data from the `core/demo_data` directory instead of pulling data from dynamic, live sources. This should only be set when running locally and should be provided through the command line.
 
 ### Running the application locally
 
 `yarn dev` will run a development API server locally on port `3001`. The development server will remain active until you either close your terminal or shut it down using `control+c`.
 
-**Note:** The server does not need to be restarted when source code is modified. The assets will automatically be recompiled and the browser will be refreshed — except in the case of changing fixture data in `/server/core/demo_data` while running in demo mode (as described below).
+**Note:** The server does not need to be restarted when source code is modified. The assets will automatically be recompiled and the browser will be refreshed — except in the case of changing fixture data in `core/demo_data` while running in demo mode (as described below).
 
 ### Demo mode
 
-When running locally, you can run the app in demo mode to point the app to static data contained in `server/core/demo_data`. This is useful for debugging issues that materialize under specific data circumstances, for demonstrating the tool without exposing real data, for development when you don't have Internet access, and other use cases.
+When running locally, you can run the app in demo mode to point the app to static data contained in `core/demo_data`. This is useful for debugging issues that materialize under specific data circumstances, for demonstrating the tool without exposing real data, for development when you don't have Internet access, and other use cases. (Demo mode is not perfect, as some parts of the frontend expect historical data relative to the current date, which this application does not provide.)
 
 You can launch in demo mode by running `yarn demo`. Be sure your client application has its API url set to `localhost:3001` to consume this demo data! (For convenience, there is also [a helper script](../README.md#multi-package-tools) in the root package for running both client and server together with all proper settings.)
 
@@ -47,7 +69,7 @@ This application is deployed to Google App Engine. To have deploy access, you ne
 
 Once you have the required permissions, you can set up your environment for deploys by following [these instructions](https://cloud.google.com/appengine/docs/standard/nodejs/setting-up-environment).
 
-The Recidiviz environment configuration settings (including GCP project IDs) are not checked in, but if you need access to them (i.e., you are doing development work for Recidiviz and you have GCP admin access), just ask a member of the Recidiviz engineering staff to help you out!
+The Recidiviz environment configuration settings (including GCP project IDs) are not checked in, but if you need access to them (i.e., you are doing development work for Recidiviz and you have GCP admin access), just ask a member of the Recidiviz engineering staff to help you out! They are stored in a shared archive in the company password manager.
 
 ### Deploying to Staging
 

spotlight-api/app.js

@@ -19,6 +19,8 @@ const express = require("express");
 const cors = require("cors");
 const morgan = require("morgan");
 const helmet = require("helmet");
+const jwt = require("express-jwt");
+const jwks = require("jwks-rsa");
 const zip = require("express-easy-zip");
 const api = require("./routes/api");
 
@@ -33,6 +35,22 @@ app.use(morgan("dev"));
 app.use(helmet());
 app.use(zip());
 
+if (process.env.AUTH_ENABLED === "true") {
+  const checkJwt = jwt({
+    secret: jwks.expressJwtSecret({
+      cache: true,
+      rateLimit: true,
+      jwksRequestsPerMinute: 5,
+      jwksUri:
+        "https://recidiviz-spotlight-staging.us.auth0.com/.well-known/jwks.json",
+    }),
+    audience: "recidiviz-spotlight-staging",
+    issuer: "https://spotlight-login-staging.recidiviz.org/",
+    algorithms: ["RS256"],
+  });
+  app.use(checkJwt);
+}
+
 app.post("/api/:tenantId/public", express.json(), api.metricsByName);
 
 // uptime check endpoint

spotlight-api/gae.yaml.example

@@ -8,3 +8,4 @@ includes:
 env_variables:
   GOOGLE_APPLICATION_CREDENTIALS: {GOOGLE_APPLICATION_CREDENTIALS}
   METRIC_BUCKET: {METRIC_BUCKET}
+  AUTH_ENABLED: {AUTH_ENABLED}

spotlight-api/package.json

@@ -24,7 +24,9 @@
     "dotenv": "^8.2.0",
     "express": "^4.17.1",
     "express-easy-zip": "^1.1.5",
+    "express-jwt": "^6.0.0",
     "helmet": "^3.23.3",
+    "jwks-rsa": "^1.4.0",
     "morgan": "^1.10.0"
   },
   "devDependencies": {

spotlight-api/routes/api.js

@@ -21,6 +21,7 @@
  */
 
 const metricsApi = require("../core/metricsApi");
+const { AUTH0_APP_METADATA_KEY } = require("../utils/constants");
 const demoMode = require("../utils/demoMode");
 
 const isDemoMode = demoMode.isDemoMode();
@@ -39,14 +40,27 @@ function responder(res) {
 }
 
 function metricsByName(req, res) {
+  const { AUTH_ENABLED } = process.env;
+  const { tenantId } = req.params;
   const { metrics } = req.body;
+  const stateCode = req.user?.[
+    AUTH0_APP_METADATA_KEY
+  ]?.state_code?.toLowerCase();
   if (!Array.isArray(metrics)) {
     res
       .status(400)
       .json({ error: "request is missing metrics array parameter" });
+  } else if (
+    AUTH_ENABLED === "true" &&
+    stateCode !== tenantId.toLowerCase() &&
+    stateCode !== "recidiviz"
+  ) {
+    res.status(401).json({
+      error: `User is not a member of the requested tenant ${tenantId}`,
+    });
   } else {
     metricsApi.fetchMetricsByName(
-      req.params.tenantId,
+      tenantId,
       metrics,
       isDemoMode,
       responder(res)

spotlight-api/routes/api.test.js

@@ -0,0 +1,155 @@
+// Recidiviz - a data platform for criminal justice reform
+// Copyright (C) 2020 Recidiviz, Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with this program.  If not, see <https://www.gnu.org/licenses/>.
+// =============================================================================
+
+const { fetchMetricsByName } = require("../core/metricsApi");
+const { AUTH0_APP_METADATA_KEY } = require("../utils/constants");
+const { metricsByName } = require("./api");
+
+jest.mock("../core/metricsApi");
+
+// mocking the node env is esoteric, see https://stackoverflow.com/a/48042799
+const ORIGINAL_ENV = process.env;
+
+beforeEach(() => {
+  fetchMetricsByName.mockImplementation(
+    (tenantId, metrics, isDemoMode, responder) => {
+      responder(undefined, "passed");
+    }
+  );
+});
+
+afterEach(() => {
+  process.env = ORIGINAL_ENV;
+});
+
+test("retrieves metrics if auth is disabled", async () => {
+  process.env.AUTH_ENABLED = "false";
+  const mockFn = jest.fn();
+  metricsByName(
+    {
+      params: {
+        tenantId: "US_ND",
+      },
+      body: {
+        metrics: ["test_metric"],
+      },
+    },
+    {
+      send: mockFn,
+    }
+  );
+  expect(mockFn).toHaveBeenCalledWith("passed");
+});
+
+test("returns 401 if there is no metrics array in the request body", async () => {
+  process.env.AUTH_ENABLED = "false";
+  const mockSendFn = jest.fn();
+  const mockStatusFn = jest.fn();
+  metricsByName(
+    {
+      params: {
+        tenantId: "US_ND",
+      },
+      body: {},
+    },
+    {
+      status: () => ({ json: mockStatusFn }),
+      send: mockSendFn,
+    }
+  );
+  expect(mockSendFn).not.toHaveBeenCalled();
+  expect(mockStatusFn).toHaveBeenCalledWith({
+    error: "request is missing metrics array parameter",
+  });
+});
+
+test("retrieves metrics if auth is enabled and user state code is 'recidiviz'", async () => {
+  process.env.AUTH_ENABLED = "true";
+  const mockFn = jest.fn();
+  metricsByName(
+    {
+      params: {
+        tenantId: "US_ND",
+      },
+      body: {
+        metrics: ["test_metric"],
+      },
+      user: {
+        [AUTH0_APP_METADATA_KEY]: {
+          state_code: "recidiviz",
+        },
+      },
+    },
+    {
+      send: mockFn,
+    }
+  );
+  expect(mockFn).toHaveBeenCalledWith("passed");
+});
+
+test("retrieves metrics if auth is enabled and user state code matches the request param", async () => {
+  process.env.AUTH_ENABLED = "true";
+  const mockFn = jest.fn();
+  metricsByName(
+    {
+      params: {
+        tenantId: "US_ND",
+      },
+      body: {
+        metrics: ["test_metric"],
+      },
+      user: {
+        [AUTH0_APP_METADATA_KEY]: {
+          state_code: "us_nd",
+        },
+      },
+    },
+    {
+      send: mockFn,
+    }
+  );
+  expect(mockFn).toHaveBeenCalledWith("passed");
+});
+
+test("returns 401 if auth is enabled and user state code doesn't match the request param", async () => {
+  process.env.AUTH_ENABLED = "true";
+  const mockStatusFn = jest.fn();
+  const mockSendFn = jest.fn();
+  metricsByName(
+    {
+      params: {
+        tenantId: "US_PA",
+      },
+      body: {
+        metrics: ["test_metric"],
+      },
+      user: {
+        [AUTH0_APP_METADATA_KEY]: {
+          state_code: "us_nd",
+        },
+      },
+    },
+    {
+      status: () => ({ json: mockStatusFn }),
+      send: mockSendFn,
+    }
+  );
+  expect(mockSendFn).not.toHaveBeenCalled();
+  expect(mockStatusFn).toHaveBeenCalledWith({
+    error: "User is not a member of the requested tenant US_PA",
+  });
+});

spotlight-api/utils/constants.js

@@ -0,0 +1,5 @@
+const AUTH0_APP_METADATA_KEY = "https://recidiviz.org/app_metadata";
+
+module.exports = {
+  AUTH0_APP_METADATA_KEY,
+};