Your finances as version-controlled input data
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
app
docs
example
src
test
.ghci
.gitignore
.gitmodules
ChangeLog.md
LICENSE
README.org
Setup.hs
TODO.org
package.yaml
stack.yaml

README.org

hledger: Make It So

Inspired by the structure and ideas of adept’s Full-fledged Hledger.

I’ve given a talk at Lambda Luminaries Johannesburg featuring hledger and hledger-makeitso.

After Cloning This Repository

This repository has some submodules included, mostly related to the examples in the documentation.

You need to initialise and update the submodules:

git submodule init
git submodule update

Who should use this project?

At the moment this software is in an alpha stage. I’m still making and changing decisions as I go along.

If you’re familiar with hledger and are willing to tolerate some annoying breaking changes from time to time, then please try it out and let me know what you think.

Build Instructions

You need a recent version of stack installed.

Then run:

stack test
stack install

Which should end with this:

Copied executables to ~/.local/bin:
- hledger-makeitso

Ensure that ${HOME}/.local/bin is in your PATH.

Usually this means adding this to your ~/.bashrc:

PATH="${HOME}/.local/bin:${PATH}"

How to Use It

Overview of the Basic Workflow

  1. Save an input CSV file to a specific directory.
  2. Add an hledger rules file. Include some classification rules if you want.
  3. Run hledger-makeitso import

Add all your files to your favourite version control system.

The generated journal that you most likely want to use as your LEDGER_FILE is called makeitso.journal. This has include directives to all the automatically imported journals, as well as includes for your own manually managed journal entries.

In a typical software project we don’t add generated files to version control, but in this case I think it is a good idea to add all the generated files to version control as well - when you inevitably change something, e.g. how you classify transactions in your rules file, then you can easily see if your change had the desired effect by looking at a diff.

Detailed Step-By-Step Guide

Have a look at the detailed step-by-step instructions and the files in the documentation directory.

For a visual overview, check out the slide show version of the same step-by-step instructions:

https://pauley.org.za/hledger-makeitso/

You can see the example imported financial transactions as it was generated by the step-by-step instructions here:

https://github.com/apauley/hledger-makeitso-example-finances

Feature Reference

Input Files

Your input files will probably be CSV files with a line for each transaction, although other file types will work fine if you use a preprocess or a construct script that can read them. These scripts are explained later.

We mostly use conventions based on a predefined directory structure for your input statements.

For example, assuming you have a savings account at mybank, you’ll put your first CSV statement here: import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv.

Some people may want to include accounts belonging to their spouse as part of the household finances: import/spouse/otherbank/checking/1-in/2018/987654321_2018-06-30.csv.

More About Input Files

All files and directories under the import directory is related to the automatic importing and classification of transactions.

The directory directly under import is meant to indicate the owner or custodian of the accounts below it. It mostly has an impact on reporting. You may want to have separate reports for import/mycompany and import/personal.

Below the directory for the owner we can indicate where an account is held. For a bank account you may choose to name it import/john/mybank.

If your underground bunker filled with gold has CSV statements linked to it, then you can absolutely create import/john/secret-treasure-room.

Under the directory for the financial institution, you’ll have a directory for each account at that institution, e.g. import/mycompany/bigbankinc/customer-deposits and import/mycompany/bigbankinc/expense-account.

Next you’ll create a directory named 1-in. This is to distinguish it from 2-preprocessed and 3-journal which will be auto-generated later.

Under 1-in you’ll create a directory for the year, e.g. 2018, and within that you can copy the statements for that year: import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv

Rules Files

If your input file is in CSV format, or converted to CSV by your preprocess script, then you’ll need an hledger rules file.

hledger-makeitso will try to find a rules file for each statement in a few places. The same rules file is typically used for all statements of a specific account, or even for all accounts of the same specific bank.

  • A global rules file for any mybank statement can be saved here: import/mybank.rules
  • A rules file for all statements of a specific account: import/spouse/bigbankinc/savings/bigbankinc-savings.rules

Statement-specific Rules Files

What happens if some of the statements for an account has a different format than the others?

This can happen if you normally get your statements directly from your bank, but some statements you had to download from somewhere else, like Mint, because your bank is being daft with older statements.

In order to tell hledger-makeitso that you want to override the rules file for a specific statement, you need to add a suffix, separated by an underscore (_) and starting with the letters rfo (rules file override) to the filename of that statement.

For example: assuming you’ve named your statement 99966633_20171223_1844_rfo-mint.csv.

hledger-makeitso will look for a rules file named rfo-mint.rules in the following places:

  • in the import directory, e.g. import/rfo-mint.rules
  • in the bank directory, e.g. import/john/mybank/rfo-mint.rules
  • in the account directory, e.g. import/john/mybank/savings/rfo-mint.rules

Example rules file usage

A common scenario is multiple accounts that share the same file format, but have different account1 directives.

One possible approach would be to include a shared rules file in your account-specific rules file.

If you are lucky enough that all statements at mybank share a common format across all accounts, then you can include a rules file that just defines the parts that are shared across accounts.

Two accounts at mybank may have rules files similar to these.

A checking account at mybank:

# Saved as: import/john/mybank/checking/mybank-checking.rules
include ../../../mybank-shared.rules
account1 Assets:Current:John:MyBank:Checking

Another account at mybank:

# Saved as: import/alice/mybank/savings/mybank-savings.rules
include ../../../mybank-shared.rules
account1 Assets:Current:Alice:MyBank:Savings

Where import/mybank-shared.rules may define some shared attributes:

skip 1

fields date, description, amount, balance

date-format %Y-%m-%d
currency $

Another possible approach could be to use your preprocess script to write out a CSV file that has extra fields for account1 and account2.

You could then create the above mentioned global import/mybank.rules with the fields defined more or less like this:

fields date, description, amount, balance, account1, account2

Opening and Closing Balances

Opening Balances

hledger-makeitso looks for a file named 3-journal/YEAR-opening.journal in each account directory, where YEAR corresponds to an actual year directory, eg. 1983 (if you have electronic statements dating back to 1983).

If it exists the file will automatically be included at the beginning of the generated journal include file for that year.

You need to edit this file for each account to specify the opening balance at the date of the first available transaction.

An opening balance may look something like this:

2018-06-01 Savings Account Opening Balance
assets:Current:MyBank:Savings               $102.01
equity:Opening Balances:MyBank:Savings

Closing Balances

Similar to opening balances, hledger-makeitso looks for an optional file named 3-journal/YEAR-closing.journal in each account directory.

If it exists the file will automatically be included at the end of the generated journal include file for that year.

A closing balance may look something like this:

2018-06-01 Savings Account Closing Balance
assets:Current:MyBank:Savings               $-234.56 = $0.00
equity:Closing Balances:MyBank:Savings

The preprocess Script

Sometimes the statements you get from your bank is less than suitable for automatic processing. Or maybe you just want to make it easier for the hledger rules file to do its thing by adding some useful columns.

If you put a script called preprocess in the account directory, e.g. import/john/mybank/savings/preprocess, then hledger-makeitso will call that script for each input statement.

The preprocess script will be called with 4 positional parameters:

  1. The path to the input statement, e.g. import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv
  2. The path to an output file that can be sent to hledger, e.g. import/john/mybank/savings/2-preprocessed/2018/123456789_2018-06-30.csv
  3. The name of the bank, e.g. mybank
  4. The name of the account, e.g. savings
  5. The name of the owner, e.g. john

Your preprocess script is expected to:

  • read the input file
  • write a new output file at the supplied path that works with your rules file
  • be idempotent. Running preprocess multiple times on the same files will produce the same result.

The construct Script

If you need even more power and flexibility than what you can get from the preprocess script and hledger’s CSV import functionality, then you can create your own custom script to construct transactions exactly as you need them.

At the expense of more construction work for you, of course.

As an example, hledger’s CSV import currently only supports two postings per transaction, even though hledger itself is perfectly happy with transactions containing more than two postings, e.g.:

2019-02-01 Mortgage Payment
Liabilities:Mortgage                                1,000.00
Expenses:Interest:Real Estate                         833.33
Assets:Cash                                         -1833.33

The construct script can be used in addition to the preprocess script, or on it’s own. But since the construct script is more powerful than the preprocess script, you could tell your construct script to do anything that the preprocess script would have done.

Save your construct script in the account directory, e.g. import/john/mybank/savings/construct.

hledger-makeitso will call your construct script with 4 positional parameters:

  1. The path to the input statement, e.g. import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv
  2. A “-” (indicating that output should be sent to stdout)
  3. The name of the bank, e.g. mybank
  4. The name of the account, e.g. savings
  5. The name of the owner, e.g. john

Your construct script is expected to:

  • read the input file
  • generate your own hledger journal transactions
  • be idempotent. Running construct multiple times on the same files should produce the same result.
  • send all output to stdout. hledger-makeitso will pipe your output into hledger which will format it and save it to an output file.

Manually Managed Journals

Not every transaction in your life comes with CSV statements.

Sometimes you just need to add a transaction for that time you loaned a friend some money.

hledger-makeitso looks for two files within each owner directory as part of the import:

  1. import/john/_manual_/pre-import.journal
  2. import/john/_manual_/post-import.journal

You can enter your own transactions manually into these files, or put the include files to your own transactions.

If one or more of these files exist, they will be included just before (the pre file) or just after (the post file) the journal containing all the automatic imports:

cat import/john-include.journal
### Generated by hledger-makeitso - DO NOT EDIT ###

!include _manual_/pre-import.journal
!include mybank/mybank.journal
!include _manual_/post-import.journal

Compatibility with Ledger

When writing out the journal include files, hledger-makeitso sorts the include statements by filename.

Ledger fails any balance assertions when the transactions aren’t included in chronological order.

An easy way around this is to name your input files so that March’s statement is listed before December’s statement.

Another option is to add --permissive to any ledger command.

So you should easily be able to use both ledger and hledger on these journals.

Project Goals

My hledger files started to collect a bunch of supporting code that weren’t really specific to my financial situation.

I want to extract and share as much as possible of that supporting code.

Adept’s goals also resonated with me:

  • Tracking expenses should take as little time, effort and manual work as possible
  • Eventual consistency should be achievable: even if I can’t record something precisely right now, maybe I would be able to do it later, so I should be able to leave things half-done and pick them up later
  • Ability to refactor is a must. I want to be able to go back and change the way I am doing things, with as little effort as possible and without fear of irrevocably breaking things.

Still To Be Done

I add ideas and thoughts in TODO.org

Let me know if you can think of some improvements.