Monthly activity email cron job #4400

Vinnl · 2024-04-08T19:55:44Z

References:

Jira: MNTOR-3067
Figma: N/A

Description

This adds a MonthlyActivityEmail flag and a script that will send the monthly activity email to 10 (or $MONTHLY_ACTIVITY_EMAIL_BATCH_SIZE) Plus subscribers who have that flag enabled, whose monthly_email_optout is not true (@mansaj - is this column actually boolean, or is this the one where I need to check for 0 or 1 or -1 or something) and to whom we've not sent such an email in the last 30 days.

Importantly, it also sets up the build infrastructure to compile JSX and TypeScript for cron job scripts (though note that I haven't updated our cron jobs configuration yet to call the script - which will be at dist/cronjobs/monthlyActivity.js). The main thing here to realise is that this means that some code is now built by Next.js (using Webpack and SWC) when shipped as part of the website, and by ESBuild when shipped as part of the cron job - and that these have their own peculiarities. For example, JSX files that are also used in cron jobs need to explicitly import React, need to use a wrapper around import "server-only", need to load Fluent differently (see also my followup PR), and don't have access to variables from .env (possibly we should run dotenv for cron jobs like we do for the e2e tests).

I'll call out some more specific things inline.

How to test

Enable the MonthlyActivityEmail flag for your user, then run npm run dev:cronjobs. You may want to reset the value of their monthly_email_at column afterwards if you want to retry.

Checklist (Definition of Done)

Localization strings (if needed) have been added.
Commits in this PR are minimal and have descriptive commit messages.
I've added or updated the relevant sections in readme and/or code comments
I've added a unit test to test for potential regressions of this bug. - Somewhat, but there'd also be lots of mocking involved for more.
Product Owner accepted the User Story (demo of functionality completed) or waived the privilege.
All acceptance criteria are met. - Still need to actually have the cron job executed.
Jira ticket has been updated (if needed) to match changes made during the development process.
Jira ticket has been updated (if needed) with suggestions for QA when this PR is deployed to stage.

Vinnl

Called out some things to pay attention to.

Vinnl · 2024-04-08T19:56:29Z

esbuild.cronjobs.mjs

+  format: "esm",
+  outdir: "dist/cronjobs",
+  sourcemap: true,
+  target: "node20.12",


@rhelmer Here's another instance of the Node version we'd need to keep up-to-date.

Q: Should we start some document/wiki/README in this repo with all the places we need to bump Node.js versions (considering last week's sec releases and this week's scheduled sequel)?

I think it can be just node or node20 if you want to be more specific - do you think it's worth specifying it here? The environment running the cron job (k8s CronJob using the container we build in Dockerfile) will be what's actually available in the environment.

What I like about saying 20.12 is that it will show up on a project-wide search for that number, so that we won't forget to update it when updating the other instances. Using just 20 wouldn't be a problem for the coming future, but makes it easy to miss when we upgrade to Node 22.

As for the document/wiki/README, I'd actually like to write a small script that just checks all instances and errors out if they're not aligned, that we can run in CI.

Vinnl · 2024-04-08T19:58:03Z

src/app/functions/server/getSubscriberBreaches.ts

I had to move this into a separate file (de1c3d1) because the other functions in the module referenced the Next.js-specific l10n module, which calls require.context, which isn't available in plain Node.

Vinnl · 2024-04-08T19:58:41Z

src/app/functions/server/getSubscriberBreaches.ts

+ * @param params.countryCode
+ */
+export async function getSubscriberBreaches({
+  fxaUid,


I changed this to fxaUid from user because that was the only property we used, and user was the user's session - which there is none of when running cronjobs.

Other than that I did not change this function.

Vinnl · 2024-04-08T19:59:31Z

src/app/(proper_react)/(redesign)/(authenticated)/admin/emails/actions.tsx

@@ -102,7 +102,7 @@ export async function triggerMonthlyActivity(emailAddress: string) {
  const data = getDashboardSummary(
    latestScan.results,
    await getSubscriberBreaches({
-      user: session.user,
+      fxaUid: session.user.subscriber?.fxa_uid,


See my comment at the function definition of getSubscriberBreaches for why I changed this.

Vinnl · 2024-04-08T20:00:33Z

src/app/functions/server/notInClientComponent.ts

This file is needed because, while import "server-only" prevents us from loading a module in a client component, it also breaks when running in plain Node.

Vinnl · 2024-04-08T20:02:21Z

src/cronjobs/monthlyActivity.tsx

+    process.env.MONTHLY_ACTIVITY_EMAIL_BATCH_SIZE ?? "",
+    10,
+  );
+  const batchSize = Number.isNaN(batchSizeEnvVar) ? 10 : batchSizeEnvVar;


I figured I'd start with (configurable) small batches so that we can just run the cron job more often and not overload the server with a single big one, since we're doing multiple queries per user. Since this is limited to Plus users for now, there's not that many accounts we need to get through anyway. Also makes it easier to recover from errors.

Vinnl · 2024-04-08T20:03:19Z

src/cronjobs/monthlyActivity.tsx

+    fxaUid: subscriber.fxa_uid,
+    countryCode: countryCodeGuess,
+  });
+  const data = getDashboardSummary(latestScan.results, subscriberBreaches);


I still have to check with Tony whether we want to show totals, or only specifics for the past month.

Vinnl · 2024-04-08T20:04:32Z

src/db/tables/subscribers.js

+    query = query.andWhereRaw(
+      `EXISTS (
+        SELECT 1
+        FROM jsonb_array_elements(fxa_profile_json->'subscriptions') AS subscription
+        WHERE subscription::text = '"${MONITOR_PREMIUM_CAPABILITY}"'
+      )`
+    );


I know pretty little of SQL, so I'm not sure if this is the best way to check for rows where the fxa_profile_json column has an object with a subscriptions property containing an array of strings, of which one element is "monitor"?

I'm not sure if it's any more efficient but there is a ? operator https://www.postgresql.org/docs/9.4/functions-json.html that can check for existence:

WHERE (fxa_profile_json->'subscriptions')::jsonb ? 'monitor'

Hmm, I think I did try to use that, but it got interpreted as a parameter - and then my PG UI didn't support it either. But you just gave me the push to verify that running it in psql does work, and then digging through old Knex.js issues to find out how to escape it: ef2a47c

Vinnl · 2024-04-08T20:05:47Z

src/emails/getEmailL10n.node.ts

It's probably best to ignore this file for now - it's basically the existing Next.js getL10n, but with changes to make it compatible with plain Node. I've got a followup PR that untangles the five different runtimes that l10n currently has to take into account, and abstracts away the common parts.

Vinnl · 2024-04-08T20:06:30Z

src/emails/templates/EmailFooter.tsx

@@ -2,6 +2,7 @@
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, You can obtain one at http://mozilla.org/MPL/2.0/. */

+import React from "react";


Next.js auto-imports React in JSX files, but esbuild doesn't (at least no by default, maybe there's an option somewhere).

esbuild.cronjobs.mjs

pdehaan · 2024-04-08T20:50:52Z

esbuild.cronjobs.mjs

+  format: "esm",
+  outdir: "dist/cronjobs",
+  sourcemap: true,
+  target: "node20.12",


Q: Should we start some document/wiki/README in this repo with all the places we need to bump Node.js versions (considering last week's sec releases and this week's scheduled sequel)?

pdehaan · 2024-04-08T20:57:33Z

src/cronjobs/monthlyActivity.tsx

+
+async function run() {
+  const batchSizeEnvVar = Number.parseInt(
+    process.env.MONTHLY_ACTIVITY_EMAIL_BATCH_SIZE ?? "",


I'm confused. Why nullish coalesce to empty string which will NaN and then default to 10 below on L28?
Could we default here to "10" and skip L28, or am I missing something?

Hmm, so I was changing it to your suggestion, but then remembered why I didn't do that in the first place - I did still want to guard against passing in an invalid value for the env var, i.e. check Number.isNaN. But that does mean we'd ignore the env var value silently, so I'm now just throwing if it's wrong: 32e5b54.

src/db/tables/subscribers.js

rhelmer

This looks great, I have some quibbles that are pretty minor. I know it's safe as-is but I really think we should always use parameterized queries for knex queries.

rhelmer · 2024-04-08T21:19:55Z

esbuild.cronjobs.mjs

+  format: "esm",
+  outdir: "dist/cronjobs",
+  sourcemap: true,
+  target: "node20.12",


I think it can be just node or node20 if you want to be more specific - do you think it's worth specifying it here? The environment running the cron job (k8s CronJob using the container we build in Dockerfile) will be what's actually available in the environment.

src/cronjobs/monthlyActivity.tsx

rhelmer · 2024-04-08T21:27:57Z

src/db/tables/subscribers.js

@@ -8,6 +8,7 @@ import AppConstants from '../../appConstants.js'

 const knex = createDbConnection();
 const { DELETE_UNVERIFIED_SUBSCRIBERS_TIMER } = AppConstants
+const MONITOR_PREMIUM_CAPABILITY = "monitor";


Should this be in constants.ts, since it's used in more than one place?

Oh right, I was actually going to ask you, since I copied this pattern from you 😅

rhelmer · 2024-04-08T21:37:34Z

src/db/tables/subscribers.js

+    // Note: This will only match people who had a Plus subscription the last
+    //       time they signed in. We'd have to check with SubPlat directly to
+    //       find out if they have a subscription right now.


The Monitor database is updated by webhook as well, so we don't necessarily need to wait until the user logs in again.

This comment is relevant for dev environments that don't have webhook handling set up e.g. localhost.

Oh right, good one - clarified the comment in 9c81d47.

rhelmer · 2024-04-08T21:45:26Z

src/db/tables/subscribers.js

+    query = query.andWhereRaw(
+      `EXISTS (
+        SELECT 1
+        FROM jsonb_array_elements(fxa_profile_json->'subscriptions') AS subscription
+        WHERE subscription::text = '"${MONITOR_PREMIUM_CAPABILITY}"'
+      )`
+    );


I'm not sure if it's any more efficient but there is a ? operator https://www.postgresql.org/docs/9.4/functions-json.html that can check for existence:

WHERE (fxa_profile_json->'subscriptions')::jsonb ? 'monitor'

src/db/tables/subscribers.js

rhelmer · 2024-04-08T21:54:33Z

src/db/tables/subscribers.js

+  let query = knex('subscribers')
+    .select()
+    .where((builder) => builder.whereNull("monthly_email_optout").orWhere("monthly_email_optout", false))
+    .andWhere(builder => builder.whereNull("monthly_email_at").orWhereRaw('"monthly_email_at" < NOW() - INTERVAL \'30 days\''));


So this avoids re-sending because updateMonthlyEmailTimestamp() is called which sets this, right? I just want to make sure we're thinking through that path if there are any exceptions, so we can't e.g. send duplicate emails because something there fails after the email is sent but before this timestamp is updated.

Yes exactly, and that gets set before we send the email, so if an error occurs, the email won't get sent. Although given #4399 (comment), I'll probably need to create a new query once that lands.

Thinking through this more, what if we:

drop the columns (and data) of monthly_email_optout and monthly_email_at

add new cols monthly_monitor_report: boolean and monthly_monitor_report_at:timestamp

Basically do a reset since we don't want to keep the old data

Also I believe the data that exists should be inconsequential (very outdated, at least 6-12 months old)

Yeah that works for me. Wouldn't really change the logic here, just different column names, and a new query to update the timestamp when I send an email. (Although I imagine we do want to keep existing opt-outs of monthly_email_optout.)

OK, rebased on latest main that has the changes described above, and added e34b2e5 to make use of them.

mansaj · 2024-04-17T14:41:11Z

src/cronjobs/monthlyActivity.tsx

@@ -0,0 +1,108 @@
+/* This Source Code Form is subject to the terms of the Mozilla Public


Let's use src/scripts/cronjobs

With apologies for rebasing, but I was afraid I wasn't going to be able to get all the path changes correct if I spread this over fixup commits. Fixed in f753a0f and 93c298a.

mansaj · 2024-04-26T14:41:11Z

src/db/tables/subscribers.js

+  // show up as errors when we remove it from the flag list:
+  /** @type {import("./featureFlags.js").FeatureFlagName} */
+  const featureFlagName = "MonthlyActivityEmail";
+  const flag = await knex("feature_flags")


shouldn't we use a feature flag util function for this? If this specific one doesn't exist, maybe we should create it in the util file?

Also, for example, we need to make sure delete_at is null, since we don't actually delete old feature flags but rather archive them

Oh that's a good point, added that check in a util function in 1c425f7.

The main reason I initially shied away from the util function, is because I had to know both is_enabled and the allow_list, and thus was already replicating feature flag-specific logic. I've now created a function that does a check for the deleted_at, but otherwise just returns the full row. We'll probably replace it by experiments later on anyway.

mansaj · 2024-04-26T14:45:43Z

src/db/tables/subscribers.js

+      updated_at: knex.fn.now(),
+    })
+    .where("primary_email", subscriber.primary_email)
+    .andWhere("id", subscriber.id)


is it necessary to match both primary_email and id? Is there a case where these two won't match?

This is me being overly cautious after our security issues where people were able to have us email other people by adding their email address - by matching on ID in addition to email address, if we're somehow looking at someone else with the same email address, it won't match. But matching on just the ID should be sufficient, of course. There's no practical downside to matching on both though, is that correct?

mansaj · 2024-04-26T14:47:32Z

src/db/tables/subscribers.js

 /**
 * @param {string} email
+ * @deprecated Only used by the `send-email-to-unresolved-breach-subscribers.js`, which it looks like might not be sent anymore?


Suggested change

* @deprecated Only used by the `send-email-to-unresolved-breach-subscribers.js`, which it looks like might not be sent anymore?

* @deprecated Delete as a part of MNTOR-3077

mansaj · 2024-04-26T14:47:41Z

src/db/tables/subscribers.js

@@ -296,6 +349,7 @@ async function updateMonthlyEmailTimestamp (email) {
 * Unsubscribe user from monthly unresolved breach emails
 *
 * @param {string} token User verification token
+ * @deprecated


Suggested change

* @deprecated

* @deprecated Delete as a part of MNTOR-3077

mansaj · 2024-04-26T14:48:36Z

src/emails/getEmailL10n.node.ts

+  // const availableLocales = Object.keys(bundleSources);
+  // const filteredLocales =
+  //   process.env.APP_ENV === "heroku"
+  //     ? availableLocales
+  //     : availableLocales.filter((locale) => supportedLocales?.includes(locale));
+  // const currentLocales = negotiateLanguages(languages, filteredLocales, {


Suggested change

// const availableLocales = Object.keys(bundleSources);

// const filteredLocales =

// process.env.APP_ENV === "heroku"

// ? availableLocales

// : availableLocales.filter((locale) => supportedLocales?.includes(locale));

// const currentLocales = negotiateLanguages(languages, filteredLocales, {

I removed the whole file in #4401, but might as well fix it for a cleaner history :) 49c09b4

That's the only property that's actually used, and this allows callers to also obtain it from elsewhere. This will especially be relevant when sending emails, when there's no user session, and thus the UID will have to be fetched from the database.

This will allow us to import that function without also importing the other modules imported in that file. This will be extra relevant for cron jobs, which don't run Webpack and thus will trip over `require.context` being present in l10n modules that get transitively included for one of the other functions.

I'll do a pass over the three different l10n loading strategies we have right now and extract the common bits, to hopefully make it a bit more maintainable. Gotta love five different runtimes! (Yes, that's not an exaggeration: plain Node (for emails), server components, client components, Storybook, and Jest.)

This this is running in plain Node (i.e. not with the Next.js infrastructure), I had to make a couple of adjustments to some existing files. Specifically, files with JSX code need to explicitly import React, and code that shouldn't be run on the client-side can't use `import "server-only"`, because that only checks that it's being imported in a React server component. Additionally, I've added esbuild to bundle it up into a single file. Theoretically, this isn't really needed for Node.js scripts, since there are no HTTP requests we're trying to optimise. However, plain Node.js can't resolve module imports without file extensions (e.g. `"../db/tables/subscribers"`), so all import specifiers in *any* module that gets loaded by the cron jobs would need to use file extensions that match those of the generated files (i.e. .js rather than .ts), which is bound to cause its own issues in Next.js. Thus, esbuild is a compromise that can resolve these import specifiers for us.

This uses `tsx` ("TS execute") to strip the type annotations on the fly and then run the resulting output in Node. It uses ESBuild behind the scenes, so the results should be consistent with the built JS in production.

Vinnl requested review from rhelmer and mansaj April 8, 2024 19:55

Vinnl requested a review from flodolo as a code owner April 8, 2024 19:55

Vinnl commented Apr 8, 2024

View reviewed changes

Vinnl mentioned this pull request Apr 8, 2024

Deduplicate l10n loading #4401

Merged

Vinnl self-assigned this Apr 8, 2024

Vinnl added the Review: M Code review time: 1-2 hours label Apr 8, 2024

pdehaan reviewed Apr 8, 2024

View reviewed changes

rhelmer requested changes Apr 8, 2024

View reviewed changes

rhelmer mentioned this pull request Apr 8, 2024

MNTOR-3079: add monthly report column #4399

Merged

flodolo approved these changes Apr 9, 2024

View reviewed changes

Vinnl mentioned this pull request Apr 9, 2024

Node version consistency #4404

Merged

mansaj reviewed Apr 17, 2024

View reviewed changes

Vinnl force-pushed the MNTOR-3067-monthly-cron branch from 39704eb to f1faf2b Compare April 23, 2024 11:39

Vinnl requested review from rhelmer and mansaj April 23, 2024 12:12

rhelmer approved these changes Apr 26, 2024

View reviewed changes

mansaj reviewed Apr 26, 2024

View reviewed changes

mansaj approved these changes Apr 26, 2024

View reviewed changes

Vinnl added 4 commits April 29, 2024 15:48

Fix incorrectly capitalised brand term

fc9f4e2

Vinnl force-pushed the MNTOR-3067-monthly-cron branch from 49c09b4 to 8c86502 Compare April 29, 2024 13:57

Vinnl added 2 commits April 29, 2024 16:37

Add script to run cron job locally

fe56498

This uses `tsx` ("TS execute") to strip the type annotations on the fly and then run the resulting output in Node. It uses ESBuild behind the scenes, so the results should be consistent with the built JS in production.

Add some tests for the monthly activity email

93f3577

Vinnl force-pushed the MNTOR-3067-monthly-cron branch from 8c86502 to 93f3577 Compare April 29, 2024 14:37

Vinnl merged commit 56c50e9 into main Apr 29, 2024
14 checks passed

Vinnl deleted the MNTOR-3067-monthly-cron branch April 29, 2024 14:48

		@@ -0,0 +1,108 @@
		/* This Source Code Form is subject to the terms of the Mozilla Public

	* @deprecated Only used by the `send-email-to-unresolved-breach-subscribers.js`, which it looks like might not be sent anymore?
	* @deprecated Delete as a part of MNTOR-3077

Monthly activity email cron job #4400

Monthly activity email cron job #4400

Conversation

Vinnl commented Apr 8, 2024 • edited Loading

References:

Description

How to test

Checklist (Definition of Done)

Vinnl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vinnl Apr 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhelmer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mansaj Apr 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vinnl Apr 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vinnl commented Apr 8, 2024 •

edited

Loading

Vinnl Apr 8, 2024 •

edited

Loading

mansaj Apr 9, 2024 •

edited

Loading

Vinnl Apr 10, 2024 •

edited

Loading