Skip to content
Permalink
master
Switch branches/tags
Go to file
 
 
Cannot retrieve contributors at this time

Airtable Expense Logging with Ledger Mode

summary = """Sometimes, we want to post individual ledger postings to some
external API. In this article, we explore ledger-mode's source and figure out
how to prepare the data for uploading to Airtable, a general-purpose
user-friendly database."""
  • 2019/4/13 My PR implementing ledger-xact-date got accepted, so had to rewrite a bunch.
    • Merged payee and date sections into new xact section.
    • Added utility function to get transaction amount value.
    • Changed implementation of final airtable exporting function to make use of ledger-xact-date and ledger-transaction-amount-value
    • Added metadata to display a custom summary, since the default doesn’t translate well.
  • 2019/4/11 First Release

Introduction

Many companies and organizations subsidize certain things like transit, food, and lodging for their employees, provided the latter log these expenses via some system defined by the former. My company uses Airtable. [fn:airtable]

If you use Ledger to maintain your personal accounting, or really any other method for that matter (but seriously. Use Ledger), it will be tedious to log your subsidized/reimbursable expenses on both books. Despite the inconvenience, this data is valuable to you, and you should keep track of it to better understand your lifestyle.

Luckily for us, Airtable has an API, and emacs is emacs. This article details my efforts in building a set of emacs functions for posting expenses from a Ledger file onto Airtable. Because I want this article to be as accessible and as informative as possible, I will go through the derivation of this script including exploring ledger-mode’s implementation.

Ledger

A Typical Receipt

Let’s say that earlier in the day, you needed to gas up, so you passed by Richard Stallman’s Gas Station. It just so happens that the Philippine Peso (PHP), despite all odds, has become the dominant currency of the United States. The acronym is no longer associated with PHP: Hypertext Processor, and the latter is now nothing more than a bad memory.

2019/03/29 Richard Stallman's Gas Station
    expenses:transport:fuel                     2000 PHP
    expenses:food                                200 PHP
    assets:cash                                -2200 PHP

You also spent 200 PHP on food because it was there, just begging to be bought, and human nature doesn’t change much between parallel realities.

In ledger, we call the snippet above an xact as a whole, rather than the more obvious “transaction”. To avoid confusion, we’ll have to define a few more terms.

Definition of Terms

payee

A description of the xact. Could be just as simple as “John” or “McDonald’s”, but sometimes it’s more detailed like “Martine, for that gallon of mayonnaise.”

account

A “bucket” of money. In Ledger, accounts can have a sort of tree structure where “tags” of increasing specificity are joined together by colons to give structure and grouping. e.g. expenses:transport:fuel, assets:cash, assets:savings:php

transaction

A line item detailing a change in the amount stored in an account. The account is separated from the amount by at least two spaces. Referred to in the Ledger source as acct-transaction. For example:

expenses:transport:fuel  2000 PHP

An xact is a group of transactions contextualized by a date and a payee.

Utility

Before we start tinkering, we have to first realize that since we’re using org-mode and babel, we can execute whatever code we write. We’re referring to that gas receipt above, so we need to find some way to easily refer to various parts of that receipt easily. org-babel-goto-named-src-block exists.

(save-excursion
  (org-babel-goto-named-src-block "Gas Receipt")
  (org-element-at-point))
(src-block
 (:language "ledger" :switches nil :parameters ":results silent" :begin 1643 :end 1922 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "2019/03/29 Richard Stallman's Gas Station\n    expenses:transport:fuel                     2000 PHP\n    expenses:food                                200 PHP\n    assets:cash                                -2200 PHP\n" :post-blank 1 :post-affiliated 1663 :name "Gas Receipt" :parent nil))

Yup, that looks right. If we go down one row, we should get both the date and payee.

(save-excursion
  (org-babel-goto-named-src-block "Gas Receipt")
  (next-line)
  (thing-at-point 'line t))
"2019/03/29 Richard Stallman's Gas Station\n"

Booyeah. Let’s wrap this all up into a function.

(defun goto-gas-receipt (line-offset)
  (org-babel-goto-named-src-block "Gas Receipt")
  (next-line line-offset))

Let’s test it.

(save-excursion
  (goto-gas-receipt 2)
  (thing-at-point 'line t))
"    expenses:transport:fuel                     2000 PHP\n"

We’re ready!

The xact

Conveniently, we have ledger-xact-payee and ledger-xact-date. Since both work with point, we get to use our cool utility function!

(save-excursion
  (goto-gas-receipt 1)
  (list (ledger-xact-payee)
        (ledger-xact-date)))
("Richard Stallman's Gas Station" "2019/03/29")

Sweet.

The Amount

There are always at least two amounts in every xact because of double-entry bookkeeping. [fn:bookkeeping] Because ledger entries can get more complicated than this, we can’t just assume the simplest case. Instead, we can let the user specify it for us!

(save-excursion
  (goto-gas-receipt 4)
  (let ((point-context (ledger-context-at-point)))
    (ledger-context-field-value point-context 'commoditized-amount)))
"-2200 PHP"

This is good, but we can do better. On its own, a string isn’t very usable. We can make a function that simply returns the value and discards the currency. For now, we don’t want to support multiple currencies.

(defun ledger-transaction-amount-value ()
  "Returns the value of the amount of a transaction without its attached currency."
  (let ((amount (ledger-context-field-value (ledger-context-at-point)
                                            'commoditized-amount)))
    (string-to-number (car (split-string amount)))))

(save-excursion
  (goto-gas-receipt 4)
  (ledger-transaction-amount-value))
-2200

We’ll get the absolute value of this number later on, because we never want to submit negative expenses.

Airtable

Setup

This section is tricky, because this involves secrets: the “project id” included in the URL, and the API key. Mine are… just kidding. Let’s load the encrypted secrets.

(load-file "../secrets/airtable-secrets.el")
(require 'airtable-secrets)

We’ll need the excellent emacs-request library.

(use-package request :ensure t)

Each Airtable base has its own tables and schema, so it will be up you to figure out the right table to interact with, and exact fields to use in the JSON payload. Let’s created a blank airtable base from the expense tracking template. We can figure out the api for this particular base by going to https://airtable.com/api and selecting the base we just made.

Experimentation

I’m wary about running POST requests right off the bat without being sure about our requests. We’ll use request.el for our http needs. Let’s try to get the list of receipts, and since we’re only doing this for confirmation, we can set (1) maxRecords to 1. We need to make the request synchronous (2) so that org mode can capture the returned value.

(request-response-data
 (request airtable-secrets-url
          :type "GET"
          :params '(("maxRecords" . 1) ;; (1)
                    ("view" . "Main View"))
          :sync t ;; (2)
          :parser 'json-read
          :headers `(("Content-Type" . "application/json")
                     ("Authorization" . ,(format "Bearer %s" airtable-secrets-auth-token)))))
((records .
          [((id . "recvM8nBwdDtki4vo")
            (fields
             (Receipt\ Photo .
                             [((id . "attRl2O8I67NQBQXo")
                               (url . "https://dl.airtable.com/0cWfA9hiTOWP5QfdUNij_cactuscastle.jpg")
                               (filename . "cactuscastle.jpg")
                               (size . 16064)
                               (type . "image/jpeg")
                               (thumbnails
                                (small
                                 (url . "https://dl.airtable.com/MBqpodIaQJWI62Q5t1iq_cactuscastle.jpg")
                                 (width . 48)
                                 (height . 36))
                                (large
                                 (url . "https://dl.airtable.com/WQVXLIuiQTGYIgDoAIxY_cactuscastle.jpg")
                                 (width . 256)
                                 (height . 191))))])
             (Category . "Interior Decor")
             (Short\ Description . "Cactus")
             (Total . 11.5)
             (Date\ &\ Time . "2015-11-06T14:22:00.000Z")
             (Notes . "A cute blue cactus with golden spines, will go great in the dining room.")
             (Who\ Paid\? . "Maritza"))
            (createdTime . "2015-08-03T23:10:03.000Z"))]))

Great, we got a response! Now let’s try to POST a new entry. We’re setting the payor as “Quinns” because that’s one of two values allowed by the template.

(request-response-data
 (request airtable-secrets-url
          :type "POST"
          :sync t
          :parser 'json-read
          :data (json-encode `(("fields" . (("Short Description" . "Testing")
                                            ("Who Paid?" . "Quinns") ;; (1)
                                            ("Date & Time" . "2019-04-09T14:22:00.000Z")
                                            ("Total" . 10)
                                            ))))
          :headers `(("Content-Type" . "application/json")
                     ("Authorization" . ,(format "Bearer %s" airtable-secrets-auth-token)))
          ))
((id . "recFyrCYKAAwjELUr")
 (fields
  (Short\ Description . "Testing")
  (Total . 10)
  (Date\ &\ Time . "2019-04-09T14:22:00.000Z")
  (Who\ Paid\? . "Quinns"))
 (createdTime . "2019-04-09T11:25:12.000Z"))

Packaging

We can now create a command to post expenses! Don’t forget to make it interactive, so we can invoke it from M-x.

(defun ledger-airtable-post-expense ()
  "Post an expense to airtable."
  (interactive)
  (let* ((xact-date (ledger-xact-date))
         (xact-payee (ledger-xact-payee))
         (xact-amount (ledger-transaction-amount-value))
         (amount (abs xact-amount))
         (date (replace-regexp-in-string (regexp-quote "/") "-" xact-date))
         (date-time (format "%sT12:00:00.000Z" date)))
    (request blog--ledger-airtable-secrets-base-url
             :type "POST"
             :sync t
             :parser 'json-read
             :data (json-encode `(("fields" . (
                                               ("Date & Time" . ,date-time)
                                               ("Who Paid?" . "Quinns")
                                               ("Total" . ,amount)
                                               ("Short Description" . ,xact-payee)
                                               ))))
             :headers `(("Content-Type" . "application/json")
                        ("Authorization" . ,(format "Bearer %s" blog--ledger-airtable-secrets-auth-token)))
             :success (cl-function
                       (lambda (&key data &allow-other-keys)
                         (print "Expense Posted!")))
             :error (cl-function
                     (lambda (&key error-thrown &allow-other-keys)
                       (print error-thrown))))))

(save-excursion
  (goto-gas-receipt 4)
  (ledger-airtable-post-expense)
  )

:

"Expense Posted!"

A quick trip to Airtable tells me that indeed, the entry has been posted. Now all I have to do is take a picture of the receipt, but that’s out of our scope because the Airtable mobile app makes that easy.

Notice that we have filled up the various callback functions in the request, because we want this function to run asynchronously.

Conclusion

By now, you probably realized why I didn’t turn this into a library: the implementation of ledger-airtable-post-expense is too dependent on the schema of the Airtable base it wants to talk to. I could factor this out to accept some function that allows one to build out the ~”fields”~ data structure, but at that point, we go back to the implementation we came up with here.

That being said, feel free to copy this function and modify it to suit your needs. Just remember, you do so at your own risk. I am not responsible for you messing up your whole company’s database. ;)

Lastly, you may look at the source for this entire blog here. Load it up in emacs. If you have org, babel, etc…, then you can probably hit C-c C-c and evaluate the src-blocks. Of course, you’ll have to modify the bits with secrets.

Footnotes

[fn:airtable] My company also uses Airtable for other administrative tasks like tracking leaves of absence and many other things. We’ve mostly switched over to Notion because it performs better as a knowledge base, but unfortnately Notion doesn’t have an API. [fn:bookkeeping] An ancient technique that Ledger is all about. This article greatly helped me understand this thing.

Parsing Credit Card Statements

Converting PDF to text

We need `pdftotext`, but it seems to have been deprecated. Let’s use poppler because it’s an updated version of xpdf, which itself contains pdftotext.

brew install poppler
(setq pdf-path "~/Downloads/statement.pdf"
      txt-path "~/Downloads/statement.txt")
(defun line-beginning-position-of-position (position)
  "Awkwardly named, I know. Returns what would be the start position
of the line containing `position`."
  (save-excursion
    (goto-char position)
    (line-beginning-position)))
(setq statement-string
      (shell-command-to-string
       (format "pdftotext -f 2 -layout %s %s && cat %s"
               pdf-path
               txt-path
               txt-path)))

(with-temp-buffer
  (insert statement-string)
  (goto-char (point-min))
  (buffer-substring (re-search-forward "Date[ \t]+Date")
                    (re-search-forward "We find ways"))
  (buffer-substring (re-search-forward "Date[ \t]+Date")
                    (re-search-forward "We find ways.*+Page [1-9] of [1-9]"))
  ;; (goto-char (point-min))
  ;; (let ((dates (split-string
  ;;               (buffer-substring-no-properties
  ;;                (search-forward "Post\nDate\n")
  ;;                (line-beginning-position-of-position
  ;;                 (search-forward "We find ways")))
  ;;               "\n")))
  ;;   (seq-partition dates (/ (length dates) 2)))
  )

Tmux

Github and Gitlab READMEs in Org Mode

Introduction

It goes without saying that READMEs are pretty important. They describe the project you’ve made, provide instructions on how to build/run/deploy/use it, etc… Fortunately, both github and gitlab support READMEs in org. Unfortunately, org support can get a little funky. The lingua franca of writing in the web is markdown, and it shows. I am stubborn though, and I will use org because org is better. This article is a collection of gotchas and bits of helpful information in getting your README.org to come out right. I expect this to become a living document that I will update from time to time as the environment changes. I hope one day for this article to become unnecessary.

Image Links

Status badges that tell you if your CI has passed, or how good your test coverage is, are really just images that link to some external URL showing more details. Of course, places like clojars provide convenient snippets for creating these things in markdown and nothing else.

/images/clojars-version-badge.png

[![Clojars Project](https://img.shields.io/clojars/v/com.levitanong/periods.svg)](https://clojars.org/com.levitanong/periods)
<p><a href="https://clojars.org/com.levitanong/periods"><img src="https://img.shields.io/clojars/v/com.levitanong/periods.svg" alt="Clojars Project" title="" /></a></p>

In orgmode, it’s actually easier. According to the Org documentation,

If the description is a file name or URL that points to an image, HTML export (see HTML export) will inline the image as a clickable button. If there is no description at all and the link points to an image, that image will be inlined into the exported HTML file.

[[https://clojars.org/com.levitanong/periods][https://img.shields.io/clojars/v/com.levitanong/periods.svg]]

https://img.shields.io/clojars/v/com.levitanong/periods.svg

Source Blocks

Chances are, if you’re writing a github/gitlab README, you should be showing some code, whether it’s to demonstrate your API, be it shell, language, or REST. If you’re showing code, you might as well actually evaluate that code and show the result. Org Babel has your back.

code block evaluation

results being funky

https://orgmode.org/manual/results.html

When dealing with emacs-request, need to use sync.

multiline headers for readability

https://orgmode.org/manual/Using-Header-Arguments.html

results not showing on github

https://github.com/wallyqs/org-ruby/issues/62

Troubleshooting

If something goes wrong, and your org mode file isn’t being rendered the way you expect, chances are it’s due to the parser. The same would be true with markdown. After all, something converts them to html.

It just so happens that github and gitlab both use the same parser for rendering org mode files: https://github.com/wallyqs/org-ruby.

Fish et al

clojars and credentials

git crypt

Document trying PHP out, maybe compare with lisp

Footnotes