An AWS Lambda function that receives new emails (as S3 objects), parses highlights from iBooks/Kindle/Readmill, and adds them to a Siteleaf site.
JavaScript
Latest commit 8d0297f Feb 15, 2017 @sawyerh Check in node_modules
Permalink
Failed to load latest commit information.
node_modules
parsers
readme-images
.gitignore
debugging.js
index.js
package.json
readme.md
siteleaf.config.example.js

readme.md

Installation

Siteleaf

  1. Create a new Siteleaf site
  2. Create two collections, one for your "books" and another for your highlights. Take note of each collection's path, which can be found in the sidebar after it's created (click "Edit"). Typically it will be the slugified collection title (eg. "Books" is books).
  3. Grab your API keys from your account page
  4. Using the info from the above steps, set your config variables in siteleaf.config.js (see siteleaf.config.example.js). Note that site is the site id whereas books and highlights are the collections' path

AWS SES (and S3)

Setup AWS SES to store received email in an S3 bucket: http://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiving-email-setting-up.html

There are a couple ways you can set this up, but here's what I did:

Through my domain registar I created a specific subdomain for receiving my highlight emails (eg my-highlights.example.com) and verified my domain using a TXT record.

DNS Settings

In AWS SES I added my verified domain

SES Settings

and created a rule set that would send my emails to S3 (you can create the bucket in this step if one doesn't already exist)

SES Rule set

AWS Lambda

Create a new AWS Lambda function using the code in this repo (you can upload it as a Zip).

Lambda Settings

After your Lambda function is created, add an event source to it that listens for Object Created events in S3. Now your Lambda function will be called anytime a new object is added to the S3 bucket you setup to store your emails.

Lambda Settings

Saving Highlights

iBooks

  1. Go to the "Notes" tab
  2. Click the "Share" icon
  3. Email your highlights in the email's body

Kindle

If you're reading in a Kindle app (iOS/Android):

  1. Open the "Notebook" section of your book where your annotations are kept
  2. Tap the "Share" icon, then tap "Export as HTML"
  3. Email the export as an attachment

If you're reading on a Kindle device (and are unable to do the above):

  1. Install this bookmarklet and follow the instructions on the bookmarklet's page to export a JSON file with all Kindle highlights.
  2. Email the json file from step 1 and/or My Clippings.txt (see below) as attachments

Don't see all of your highlights? That's likely because you tried reading a book on your Kindle (or in the Kindle app) that wasn't purchased on Amazon. Yes, seriously. If you have a Kindle device, plug it into your computer and find documents/My Clippings.txt. If you're lucky, you'll find additional highlights in there.

Readmill

  1. Hopefully you downloaded the export file before they shut everything down on July 1, 2014.
  2. Email liked-highlights-data.json and/or reading-data.json as attachments.

Plain text

You can also email your highlights in the body of an email. This is handy if you can't access your Kindle highlights or if you'd like to share a highlight from an app that's not supported.

Your email must follow a specific format to work properly:

  1. Enter the title of the book/article/website/etc as the email's Subject.
  2. On the first line of the body, enter ++ followed by two empty lines. Optionally, enter the author's name directly after the two plus signs.
  3. Paste your highlights, separating each highlight with two blank lines.

Example email body:

Subject: Creativity Inc.

++Amy Wallace and Edwin Catmull


Be patient. Be authentic. And be consistent.


The more people there are in the room, the more pressure there is to perform well

Known issues

  • Due to AWS Lambda functions having a maximum runtime of 5 minutes, if you're trying to save more than 100 highlights, it might timeout before fully saving all the highlights. If all highlights don't save the first time, re-send the email and it will pick up where it left off, only adding highlights that weren't added the first time.