New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create email job queue and Reaction.Email namespace #1282

Merged
merged 16 commits into from Aug 15, 2016

Conversation

3 participants
@jshimko
Contributor

jshimko commented Aug 11, 2016

  • creates a job queue for sending emails with error handling, retries, and logging
  • create Reaction.Email namespace
  • deprecate ReactionEmailTemplate() in favor of Reaction.Email.getTemplate() (same args)
  • deprecate Meteor's Email.send() in favor of Reaction.Email.send() (same args, plus job Q)
  • refactor/cleanup of any email-related methods
  • add DDP rate limiting for notifications and Meteor accounts methods
  • add placeholder content to empty fallback email template
  • add "order complete" email template to proper path (email template paths are in serious need of attention)
  • update Meteor and NPM deps
  • Fixes #959 and fixes #1200

This is also the first step toward replacing Meteor's accounts email methods with our own so we can use the email job queue for everything the app sends.

API changes

Sending Email

// Original Meteor Email  
Email.send({
  from: 'me@example.com',
  to: 'you@example.com',
  subject: 'RE: old email API',
  html: '<body>Heyo!</body>'
});

// is now
Reaction.Email.send({
  from: 'me@example.com',
  to: 'you@example.com',
  subject: 'RE: new email API',
  html: '<body>Heyo!</body>'
});

// identical except for "Reaction." in front of it

Retrieving Email Templates

// old
ReactionEmailTemplate('path/of/template');

// new
Reaction.Email.getTemplate('path/of/template');

// template paths are currently relative to the /private directory

Example usage

// prepare the template
SSR.compileTemplate('some-name', Reaction.Email.getTemplate('path/of/template.html'));

// render the HTML and send
Reaction.Email.send({
  from: 'me@example.com',
  to: 'you@example.com',
  subject: 'RE: new email API',
  html: SSR.render('some-name', { shopUrl: Meteor.absoluteUrl() })
});

That will add an email job to the job queue. The worker will then process it immediately (in batches of up to 10) and will retry failures up to 5 times (waiting 3 mins between each try) before failing completely. All email sending attempts are logged in the job collection, so these changes are also a first step toward being able to build a UI for admins to keep track of or control email notifications.

@@ -23,15 +23,15 @@
},
"dependencies": {
"accounting-js": "^1.1.1",
"authorize-net": "ongoworks/node-authorize-net",
"authorize-net": "github:ongoworks/node-authorize-net",

This comment has been minimized.

@zenweasel

zenweasel Aug 12, 2016

Member

Why this change? From NPM docs "As of version 1.1.65, you can refer to GitHub urls as just "foo": "user/foo-project". Just as with git URLs, a commit-ish suffix can be included. For example:"

This comment has been minimized.

@jshimko

jshimko Aug 12, 2016

Contributor

I didn't manually change that. npm update --save did it.

@zenweasel

This comment has been minimized.

Member

zenweasel commented Aug 12, 2016

If I create an anonymous order and then add an email I get this error when I mark it shipped:

I20160812-11:02:20.478(8)? 03:02:20.477Z ERROR Reaction: orders/shipmentShipped: Failed to send notification
I20160812-11:02:20.478(8)?   ReferenceError: Logger is not defined
I20160812-11:02:20.479(8)?       at Object.getTemplate (server/api/core/email.js:56:5)
I20160812-11:02:20.479(8)?       at [object Object].ordersSendNotification (server/methods/core/orders.js:352:45)
I20160812-11:02:20.479(8)?       at packages/check/match.js:107:1
I20160812-11:02:20.479(8)?       at [object Object]._.extend.withValue (packages/meteor/dynamics_nodejs.js:56:1)
I20160812-11:02:20.479(8)?       at Object.exports.Match._failIfArgumentsAreNotAllChecked (packages/check/match.js:106:1)
I20160812-11:02:20.479(8)?       at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1708:18)
I20160812-11:02:20.480(8)?       at packages/ddp-server/livedata_server.js:1624:18
try {
return Assets.getText(file);
} catch (e) {
Logger.warn(`Template not found: ${file}. Falling back to coreDefault.html`);

This comment has been minimized.

@zenweasel

zenweasel Aug 12, 2016

Member

missing Import here, causing the error

This comment has been minimized.

@jshimko

jshimko Aug 12, 2016

Contributor

nm. comment redacted

@zenweasel

This comment has been minimized.

Member

zenweasel commented Aug 12, 2016

Except for that minor error this all looks good to me. I got all the emails, etc.

import { Meteor } from "meteor/meteor";
import { check } from "meteor/check";
import { Job } from "meteor/vsivsi:job-collection";
import { Jobs, Packages, Templates } from "/lib/collections";

This comment has been minimized.

@jshimko

jshimko Aug 12, 2016

Contributor

Yes, the Assets import is missing, but that's not where it comes from. It's a Meteor global and...

Currently, it is not possible to import Assets as an ES6 module. Any of the Assets methods below can simply be called directly in any Meteor server code.

https://docs.meteor.com/api/assets.html

This comment has been minimized.

@jshimko

jshimko Aug 12, 2016

Contributor

And in that case, that import is correct. There is an Assets collection, but Assets.getText isn't associated. An unfortunate namespace collision. You wouldn't be able to use both in the same file without importing the Assets collection as something else.
And that's why globals are bad, Meteor.

This comment has been minimized.

@zenweasel

zenweasel Aug 12, 2016

Member

Yeah, my mistake. I had already removed my comment.

@jshimko

This comment has been minimized.

Contributor

jshimko commented Aug 12, 2016

Missing Logger import fixed.

@zenweasel

This comment has been minimized.

Member

zenweasel commented Aug 12, 2016

Approved

Approved with PullApprove

jshimko and others added some commits Aug 12, 2016

aaronjudd

@aaronjudd aaronjudd referenced this pull request Aug 14, 2016

Closed

[Tutorial]: Docs for email process / customization #78

0 of 3 tasks complete
Reaction.Email.send({
to: email,
from: `${shop.name} <${shop.emails[0].address}>`,
subject: `You have been invited to join ${shop.name}`,

This comment has been minimized.

@aaronjudd

aaronjudd Aug 14, 2016

Contributor

The email header needs to be i18n strings, and will also need to be editable in email template editor.
This work was in progress in:
#1203
#1136

@@ -220,17 +219,15 @@ Meteor.methods({
check(shipment, Object);
if (!Reaction.hasPermission("orders")) {
throw new Meteor.Error(403, "Access Denied");
Logger.error("User does not have 'orders' permissions");

This comment has been minimized.

@aaronjudd

aaronjudd Aug 14, 2016

Contributor

If this is the new standard, let's update the team and the style guidelines. (and a task for a global update).

const tpl = `orders/${order.workflow.status}`;
SSR.compileTemplate(tpl, Reaction.Email.getTemplate(tpl));
Reaction.Email.send({

This comment has been minimized.

@aaronjudd

aaronjudd Aug 14, 2016

Contributor

Another place we will need translations, perhaps another method like "Reaction.email.submit" that had all the translation logic before Reaction.email.send could make this a bit easier to replace through the codebase, and give a convenient location for future email transforms.

@aaronjudd

This comment has been minimized.

Contributor

aaronjudd commented Aug 14, 2016

👍 @jshimko I'm good with this as is, but made some comments that we should address. I think we've discussed these requirements as well. This pr will make the merging of #1136 and #1203 not very likely so I will close those, but translation logic is a requirement for v0.16.0.

Approved with PullApprove

try {
Email.send({ from, to, subject, html });
Logger.info(`Successfully sent email to ${to}`);

This comment has been minimized.

@aaronjudd

aaronjudd Aug 14, 2016

Contributor

Not sure it's a good practice to log emails out to the server, maybe "debug"? @Capt-Slow what say you?

This comment has been minimized.

@zenweasel

zenweasel Aug 14, 2016

Member

I thought of that too. Maybe just redact the full email and leave an identifiable stub?

aaronjudd
updated to remove name from email
should be enough info

@aaronjudd aaronjudd merged commit 5135cbe into development Aug 15, 2016

3 of 6 checks passed

bitHound - Code 95 failing files.
Details
bitHound - Dependencies 1 failing dependency.
Details
ci/circleci CircleCI is running your tests
Details
code-review/pullapprove Approved by aaronjudd, zenweasel
Details
codeclimate Code Climate found 6 fixed issues.
Details
licence/cla Contributor License Agreement is signed.
Details

@aaronjudd aaronjudd deleted the 959-email-defer branch Aug 15, 2016

@aaronjudd aaronjudd removed the in progress label Aug 15, 2016

This was referenced Aug 15, 2016

@zenweasel zenweasel referenced this pull request Aug 18, 2016

Merged

Release v0.15.0 #1308

kieckhafer added a commit that referenced this pull request Aug 18, 2016

Release v0.15.0 (#1308)
* create launchdock-connect plugin (#1241)

* rebuild launchdock-connect as a plugin module

* build more flexible admin email verification method

* add Stripe dependency for launchdock-connect

* fix linting issues

* Adjust inventory on shipment (#1240)

* Trap order-completed change

* Add missing import

* Add cartItemId field to order so that we can modify inventory

* Add new "inventory/sold" method

* Explicitly set from and to statuses

* Handle order insert hook to move ordered inventory to "sold" status

* Handle moving inventory from "reserved" to "sold" status

* Fix email import

* Fix tests

* Not so much logging

* Call status change method with explicit values

* Add test for moving inventory from "reserved" to "sold"

* Add test for moving inventory from "sold" to "shipped"

* Force eslint to recognize that switch/case should be indented (#1250)

* Force eslint to recognize that swtich/case should be indented
eslint/eslint#1797

* remove eslint-plugin-meteor

- we need to tune this, removing until we have time

* enable eslint-2 codeclimate

- remove meteor-eslint plugin, failing to load with eslint-2

* updated switch indentation

- test eslint switch, case indent

* Add route hooks API (#1253)

* build initial implementation of route hooks

* extend Router with Hooks and register all hooks

* more route hooks tweaks

* router namespace cleanup

* removed unused values from registry entry (linter)

* Refactor inventory (#1251)

* As noted in comments from previous PR, just some reorg kept separate from the functionality PR

* Correct matching

* Fix for decrement cart function  (#1273)

* Add failing test

* Fix for update Mongo command

* Add failing test for "decrease below zero"

* If removeQuantity is more than quantity remove the entire line

* updated linting error

* linting error fix

* listing issues

* more cleanup and linting tweaks

* fix eslint config for object key quotes

* fix tests after error message update

* revert quote-props linter change

* revert lint changes

* Simplify logic per CR comment

* import moment-timezone (#1261)

- fixes error .names undefined
- for loading tz in i18n settings

* Fix inventory tests (#1254)

* Reduce size of the cart so that orders don't take so long to process.

* Add a wait before pulling record

* Add a wait before pulling record

* Wait after calling Meteor method

* Longer timeout

* Longer timeout

* Temporarily bypass failiing inventory test (#1280)

* fix for #1072 (#1247)

fix for #1072

* Use forked version of authorize.net that doesn't have vulnerability (#1252)

* Update circle node (#1281)

* update circle node to v4.4.7

* remove unused packages

- remove eslint-plugin-meteor
- remove bunyan-loggly (insecure dependency)
- currently not in use

* updated package.json (#1286)

- updates jquery-i18next
- updates i18next-browser-languagedetector

* Create email job queue and Reaction.Email namespace (#1282)

* build job queue for email and refactor related methods

* put placeholder content in coreDefault email template

* add rate limiting for accounts and notification methods

* put order completed template at correct path

* update old jquery-i18next

* remove log

* remove unused var

* remove unnecessary filler in placeholder template

* log warning when email template isn’t found

* return job object from Reaction.Email.send()

* fix missing import

* fix comment for email method

* remove debugging logger

* updated to remove name from email

should be enough info

* logout and hasPermission updates (#1290)

* updated hasPermissions, router behaviour

- resolves #1122
- refactors client hasPermissions to wait for a userId if one isn’t
immediately found
- intentionally not redirecting to home page (not sure if that’s the
best behavior, better to have login?)
- adds subscription manager to a few more collections

* fix typo

* import lodash

might work better if _.find exists

* check for existing route table

* Fix PayPal PayFlow discounts and refunds (#1275)

* allow discounts and refunds with PayPal PayFlow

- Fix discounts so they are working
- Allow 100% discounts
- Fix refunds so they are working

* wrap PayPal PayFlow in wrapper for easier testing

* PayPal PayFlow refund test

* removed temporary file

* updated cc error message

* update cardNumber schemas

cardNumber schemas were not allowing valid credit cards, such as Amex
(15 digits), some Visa (13 digits), and foreign Maestro (12 - 19
digits). This updates to allow these lengths to be input.

* Remove bash script from postinstall to fix Windows installs (#1299)

* don’t run bash scripts in postinstall because Windows

* add fallback fonts back to public dir

* release 0.15.0

- updated package in preparation for new release

* Allow any user who has the createProduct permission to also delete products (#1263)

* - ignoring my custom entrypoint (for running prod image with src)

* - allow users with createProduct permission to delete products (for marketplace withmultiple sellers)

* - removing .gitignore files that is not welcome in repo

* Fix Braintree discounts and refunds (#1265)

* add Braintree Payment error to en.json

* enable discounts for Braintree payments

Braintree was not capturing discounts, instead using the amount in the
original authorization for the capturing. This update changes the
amount to the paymentMethod.amount, which includes discounts.

* removed else statement after an if containing a return

Since the return inside the IF statement will effectively kill the
process if it’s hit, there is no need for the else.

* added testing for Braintree refunds

This code still needs some love, just wanted to get it up for others to
take a look at.

* remove no longer needed commented code

* remove no longer needed commented code

* added field to expected response for testing

* moved braintree/refund/list methods into BraintreeApi wrapper

Creating a wrapper for Braintree in order to more easily perform testing

* move braintree methods into new file

* reconfigure all braintree payment code to no longer use ValidateMethod

* fixed lint issues

* removed code used to skip over 24 hour braintree delay (for testing)

* Rename braintreeapi.js to braintreeApi.js

* display absolute number of adjustedTotal

due to various instances of rounding numbers for display purposes, the
adjustedTotal would sometimes display -0.00, as the adjusted total was
technically -.00000000000000000000001, even though we display 0.00 when
we round it. This update just shows the absolute number, as this is
simple a display number and does not have any affect on what is being
sent to and from the payment provider.

* updated schema to match supported payment methods

The current Schema had a 16 number minimum for credit cards, however
braintree supports cards which have number lengths ranging from 12-19

* min -> max

* removed comments

* test testing

* update braintree test

* update exports of braintreeApi functions

* fixed callback error when action has no callback

* Updated error message to make more sense to a human user

* linter fixes

* updated 'Logger.info' to Logger.debug

* removed unused test

* removed commented callback

* don't log full order details on transaction error

* fix refunds and discounts for authorize.net (#1279)

- Fix discounts so that they are correctly sent to authorize.net

- Allow 100% discounts by adding the “voidTransaction” function and
voiding transactions

- Update error messaging for Refunds, as we do not (yet) allow them
from Authorize.net

* Fix Stripe refunds and Double Discounts (#1304)

* added comment to explain voiding vs applying discount

* add ___ for correctly calculating Stripe adjusted totals

Stripe doesn’t “do” discounts, instead they take the discount sent to
them, and apply it as a refund. This was causing any discounts to show
up twice - once as a discount, once as a refund - in our adjusted
total. This update fixes that by re-adding the discount price into the
adjusted total when the payment provider is Stripe.

* also 100% discounts on Stripe

* humanizing an error code

* removed unintentional text

* linting fixes

* removed unused commented code

* removed unused commented code

* Taxes (#1289)

* initial commit tax-rebase plugin

Initial commit for issue #972

* split providers to plugins/included

* Griddle updates

- fork MeteorGriddle into core/ui-grid
- move fetchTI
- add i18n taxSettings

* initial grid editing

* add taxSettings labels

* initial custom add / edit tax rates

- initial working forms, edit toggling.. bit rough..

* rename tax-base to taxes

* updated custom grid

 - adds row selection

* add custom rate form reset

* updated custom settings

* add cart hooks to trigger taxes/calculate

* update hooks, taxes/deleteRate

- also adds taxes/deleteRate

* migrate schemas to plugins

- also set taxCloud jobs to not run by default

* add custom tax rates calculation

- add taxes, tax to cart schema
- taxes not published to client
- updates global cartTotal helpers

* check for shipping

* migrate settings to plugins

* add tax hooks to plugins

 -abort idea on “provides” as a package property

* Avalara Tax Lookup

 - adds taxes/setRate method

* comment unused fields

- comment out first implementation unused fields

These are fields that will be used future enhancements to taxes.  They
are commented out for now.

* logger cleanup

- move info to debug

* updated product data with default taxable items

- add taxCode placeholder to schema

* taxCloud tax rate calculations

* disable taxjar, check for packages

- disable while module is not functional
- WIP fetch rates and api configuration

* custom rate ui cleanup

* taxes method test

- a weak attempt at a test but gets the ball rolling and actually fixes
a todo

* updated test, versions

- add done to test
- updated meteor package versions

* remove unused template registry entries

- final cleanup, ready for PR to development
- pending tests
- pending docs

* avoid running country-data unless address exists

* add tax rate delete

- adds delete and confirm to edit form

* linted griddle.js

* remove unused import

* fix display of taxes in order workflow

perhaps this should be updated to “tax”, instead of taxes in the
invoice object.

* line item calculations

- added for custom rates
- added for avalara

rates are calculated by individual line items but on a general rate for
the cart.

* update case linted

* updated variant methods

- update products/updateProductField to handle boolean
- kludge update of childVariants when parentVariant is updated
- remove default value from form
- small cleanups

jshimko added a commit that referenced this pull request Aug 23, 2016

Merge branch 'development'
* development: (27 commits)
  Release 0.15.1 (#1314)
  Fix wrong method hooks context (#1307)
  - function isSoldOut() should return true if quantity is <= 0, not only if it is === 0. (#1311)
  Remove jquery-ui (#1310)
  Taxes (#1289)
  Fix Stripe refunds and Double Discounts (#1304)
  fix refunds and discounts for authorize.net (#1279)
  Fix Braintree discounts and refunds (#1265)
  Allow any user who has the createProduct permission to also delete products (#1263)
  Remove bash script from postinstall to fix Windows installs (#1299)
  update cardNumber schemas
  Fix PayPal PayFlow discounts and refunds (#1275)
  logout and hasPermission updates (#1290)
  Create email job queue and Reaction.Email namespace (#1282)
  updated package.json (#1286)
  Update circle node (#1281)
  Use forked version of authorize.net that doesn't have vulnerability (#1252)
  fix for #1072 (#1247)
  Temporarily bypass failiing inventory test (#1280)
  Fix inventory tests (#1254)
  ...

# Conflicts:
#	.codeclimate.yml
#	.eslintignore
#	.meteor/versions
#	client/modules/core/main.js
#	imports/plugins/core/orders/client/templates/workflow/shippingTracking.js
#	imports/plugins/core/taxes/client/settings/custom.js
#	imports/plugins/core/taxes/client/settings/settings.js
#	imports/plugins/core/taxes/server/methods/methods.js
#	imports/plugins/included/authnet/server/methods/authnet.js
#	imports/plugins/included/braintree/server/methods/braintreeApi.js
#	imports/plugins/included/braintree/server/methods/braintreeMethods.js
#	imports/plugins/included/braintree/server/methods/braintreeapi-methods-refund.app-test.js
#	imports/plugins/included/example-paymentmethod/server/methods/example-payment-methods.app-test.js
#	imports/plugins/included/inventory/server/hooks/hooks.js
#	imports/plugins/included/inventory/server/hooks/inventory-hooks.app-test.js
#	imports/plugins/included/inventory/server/methods/inventory.js
#	imports/plugins/included/inventory/server/methods/statusChanges.js
#	imports/plugins/included/launchdock-connect/client/templates/dashboard.js
#	imports/plugins/included/paypal/server/methods/payflowpro-methods-refund.app-test.js
#	imports/plugins/included/paypal/server/methods/payflowproApi.js
#	imports/plugins/included/paypal/server/methods/payflowproMethods.js
#	imports/plugins/included/product-variant/client/templates/products/productDetail/variants/variantForm/variantForm.js
#	imports/plugins/included/taxes-avalara/server/hooks/hooks.js
#	imports/plugins/included/taxes-taxcloud/server/hooks/hooks.js
#	imports/plugins/included/taxes-taxjar/server/hooks/hooks.js
#	lib/collections/helpers.js
#	package.json
#	server/api/core/import.js
#	server/methods/catalog.js
#	server/methods/core/cart-remove.app-test.js
#	server/methods/core/orders.js
#	server/methods/core/shop.js
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment