An assistant for YNAB:
- Scrape Chase Bank real-time notification emails to create YNAB transactions in real-time.
- Scrape Amazon Order emails to add memos about what was purchased on unapproved YNAB transactions.
- Uses categorization rules to update categories for unapproved YNAB transactions.
Table of Contents
You must use YNAB.
If you are not a YNAB user, this program is not going to do anything for you. You'll need to get an API Key from YNAB and plug it into the .env file.
From there, things get a little murkier depending on which functionality you want to use and your bank / email provider.
If you want to get real-time transaction infromation fed into YNAB, you'll want an account at Chase Bank and a Gmail account to use this out-of-the-box.
- If you use a different email provider, you can likely use this with minimal modifications in code.
- If you use a different bank, you might still be able to use this, but there are some things to consider.
If you do use Chase Bank, then all you need to use this is to go into your account settings and turn on email notifications so that you get a notification for any transaction over $0.
If you bank elsewhere, then your bank neesd to offer a feature that sends transaction notification emails immediately when you make a transaction. If they can do this, then you can make that work with minimal modification to code to properly parse your bank's email messages.
Nothing special is needed for this other than a gmail account. Again, if you are using a different email provider you can likely still use this with minimal modification.
This part of the application can be used without any special requirements.
Connects to Gmail via IMAP and listens for emails. Refreshes basic data from YNAB every 15 minutes. When a Chase Transaction email or Amazon Order email comes in, it adds them to a database to be processed, then kicks off the processor, which:
- Scrapes transaction info from Chase Transaction Emails and adds transactions to YNAB where they don't already exist.
- Scrapes order details from Amazon Order Emails and adds product desriptions to memo field in YNAB on unapproved transactions that match Amazon order details.
- Applies categorization-rules to unapproved YNAB transactions.
A note on usage of flag colors. If a transaction already has a flag color, it will not be modified. When the memo field is filled, the flag color is set to blue so you can tell that information has been entered in the memo without having to open the transaction in the app, but again this does not get set if the transaction already had a flag color.
| File/Folder | Description |
|---|---|
| .vscode/ | Contains workspace settings for VSCode. User and Workspace Settings |
| ├─ extensions.json | Specifies recommended VSCode extensions to use when working with this project. Workspace recommended extensions |
| └─ settings.json | Specifies VSCode workspace settings specific to this project. Settings file locations |
| data/ | Local "database" to store info scraped from emails so we never scrape the same data twice. |
| ├─ amazon-orders-data | nedb database for Amazon orders data obtained from scraping emails. |
| └─ chase-transactions-data | nedb database for transactions data obtained from scraping emails. |
| dist/ | output from building typescript files as defined by tsconfig.json |
| node_modules/ | "npm install" will install the "dependencies" from package.json in this folder. npm-install |
| src/ | |
| ├─ config/ | |
| │ └─ env.ts | |
| ├─ data/ | |
| │ ├─ models/ | |
| │ │ ├─ amazon-order.ts | |
| │ │ ├─ email.ts | |
| │ │ └─ transaction.ts | |
| │ ├─ repositories/ | |
| │ │ ├─ amazon-orders-repository.ts | |
| │ │ └─ transactions-repository.ts | |
| │ ├─ base-repository.ts | |
| │ └─ database.ts | |
| ├─ integrations/ | |
| │ ├─ email-listener.ts | |
| │ └─ ynab.ts | |
| ├─ sandbox/ | |
| │ ├─ compact.ts | |
| │ └─ unprocessed.ts | |
| ├─ tasks/ | |
| │ ├─ auto-categorize-transactions.ts | |
| │ ├─ process-amazon-orders.ts | |
| │ └─ process-chase-transactions.ts | |
| ├─ tools/ | |
| │ ├─ amazon-email-scraper.ts | |
| │ ├─ amazon-order-details-scraper.ts | |
| │ └─ chase-email-scraper.ts | |
| └─ app.ts | |
| .editorconfig | |
| .env | |
| .env.example | |
| .eslintrc.js | |
| .gitignore | Specifies intentionally untracked files to ignore for version control. .gitignore |
| LICENSE | |
| package-lock.json | Ignored via .gitignore. It is automatically generated for any operations where npm modifies either the node_modules tree or package.json. Documentation seems to indicate that it is supposed to be committed to source repositories. npm-package-lock.json |
| package.json | Provides information to npm to identify the project as well as handle the project's dependencies. npm-package.json |
| README.md | This documentation serves as the starting point for anyone wishing to familiarize themselves with the project. It uses Markdown syntax for formatting. |
| rules.example.hjson | |
| rules.hjson | |
| tsconfig.json | TypeScript configuration file. |
Decided to use IMAP to deal with reading gmail messages, since their API does not work with API keys, only OAuth.
There are several libraries to use for IMAP in node.js:
imap-simple
node-imap
imap-flow
I went with the node-imap. I used imap-simple at first but it fell short by not allowing you to read the state of the connection to reconnect when needed.
This was pretty helpful, and it includes links that cover things like how to avoid getting shut down by Amazon:
https://zenscrape.com/how-to-scrape-amazon-product-information-with-nodejs-and-puppeteer/
Copyright (c) 2020 Nathan Fast
This project is licensed under the MIT license. See the LICENSE file for more details.
This is by no means a comprehensive list, but these are some things I thought it would be worth mentioning here because they impact how useful this might be to others wishing to use this program (or contribute to it!).
- This documentation is currently lacking a bit. Specifically, it needs to cover:
- More details on how to get set up.
- How to use rules.hjson to configure auto-categorization rules.
- Easier configuration options so you can use different functionality without having to modify code to comment out certain routines.
- Better error handling so that the program can continue to run when encountering recoverable errors.
- Improve some of the hard-coded strings or otherwise rigid logic that ties this to Chase Bank / Gmail. Some of these things could just be moved to the .env file. Others would need to be refactored to use regex first, and then the regex could be part of the .env file.