Better UX around starting and ending files, close command ?

[simonmichael]

Here are a few thoughts aimed at better user experience around starting new files periodically, eg each year. Discussion welcome.

Some context

• https://hledger.org/1.32/hledger.html#close and latest https://hledger.org/dev/hledger.html#close
• https://groups.google.com/g/hledger/c/G0LLoV4PnBM/m/A2_5fXZQAgAJ (2023)
• https://groups.google.com/g/hledger/c/aZXoDSQW1xk/m/vV2T_HGDAgAJ (2023)
• https://www.reddit.com/r/plaintextaccounting/comments/18zxlbn/hledger_year_closing/ (2024)
• https://plaintextaccounting.org/FAQ#how-to-organise-files

Terminology

We need a short term for "multiple files used to cover different time periods" (as distinct from multiple files split up in some other way, like by account). I'll call them "periodic files" unless we find something better.

We need language to distinguish between

• opening and closing an account (beginning and ending its lifetime; like beancount's open and close directives)
• starting and ending a periodic journal file (eg setting year-start balances, asserting year-end balances, etc)

From here I'll try to use these four terms (open, close, start, end) as above, where possible. Though, "open"/"close" are pretty baked in to the close command right now.

Can we simplify the process of starting a new file ?

We want to represent correct asset/liability (and possibly equity) balances at all times:

• from the start of the earliest periodic file, reflecting the existing balances when we started bookkeeping
• through the transitions from one periodic file to the next
• when processing just one of the files
• or when processing several or all of them in sequence.

The current common method, start/end transactions (A)

1. assume all balances are initially zero
2. in a special starting balances transaction, post the exact starting balances
3. and immediately assert those same balances, verifying them (and 1).

If you combine multiple periodic files, all but the earliest starting balances are redundant and excessive. So we also

4. At the end of each periodic file except the latest, in a special ending balances transaction, post the negative of the ending balances. This cancels out the redundant starting balances transaction which will follow in the next file.

We balance these starting/ending balances with an equity account, call it equity:start. If migrating equity balances along from one file to the next: we must always exclude equity:start, otherwise it would accumulate redundantly and cause an incorrect equity balance.

We also need to

5. exclude the ending balances transaction when we want to see end of year balances in past years (otherwise they all appear zero)

and may want to

6. exclude starting/ending transactions from register reports.

An alternate method, balance assignments (B)

1. In a starting balances transaction, assign (set) the balances to the correct starting balances. This doesn't require initially zero balances; it posts whatever amounts are necessary to reach the correct starting balances.
2. At the end of each periodic file except the latest, assert the ending balances (without changing them), to provide some error checking. The asserted ending balances and the assigned starting balances in the next file could also be visually compared.

A possible future method, smarter balance assignments (C)

1. Use a new kind of balance assignment that succeeds only if the existing balance is either exactly zero or exactly the correct balance already. This would provide more error checking than B.

New directives (D)

1. A new directive, like start-balances, would be more semantically correct (in the real world, there are no transactions when you end a data file and start a new one) and could improve usability.
2. There could be a corresponding, usable end-balances directive.
3. These could perhaps detect more errors than the previous methods, eg enforcing "no postings are allowed after a file's end-balances or before a file's start-balances".
4. (Though, most likely these would mainly be syntactic sugar for transactions, since we understand how those work, and other semantics would require many things be resolved, such as how sequencing works.)
5. More on this below.

Which method is best ?

My rationale for B or C was that they would be easier to use than A, requiring less busywork when you need to change old balances/accounts/journal entries (to fix mistakes, or improve your bookkeeping, eg)  and propagate the effect of those changes through subsequent files. (This is especially noticeable if you track lots of accounts and gets worse over time as you accumulate more files.)

B/C also have this advantage over A: they let you select non-contiguous periodic files, so eg you could process just 2020 and 2024 and still see correct balances in those years.

But B/C have a problem. Assuming the starting balance assignments are balanced transactions (1.1), the balancing equity:start account accumulates excessively with each new file, because there's no corresponding ending balances transaction to prevent it.

To avoid this, we could use unbalanced postings to assign starting balances (1.2). But this violates the Accounting Equation so we would then no longer be able to achieve A+L+E = 0 in reports.

This could be avoided by having the earliest starting balance transaction be balanced, and the ones in later files unbalanced (1.3). But this does not fit the goal of being able to freely choose one, a subset, or all of the files.

Requirements

To review: We'd like to have

1. correct balances always
2. flexibility in choosing one or more files to process
3. good error checking
4. preservation of the accounting equation
5. minimal interference with reports, ideally

Method A (posting starting balances and zeroing out ending balances in each file) achieves 1-4. Is it the only way ? Is there really no other solution ?

What other improvements could be made to close ?

Proposals

• Try to move towards "start"/"end" terminology for file-start/file-end balances, leaving "open"/"close" more free for account opening/closing
• Change the defailt "equity:opening/closing balances" account name to "equity:start"
• Change the default close --close description to "end balances"
• Change the default close --open description to "start balances"
• By default, close just assets and liabilities, not equity
• If the user includes equity (eg with close type:ALE), still exclude equity:start (or their chosen --close-acct)
• Improve docs and examples
• Have close add the start: tags, with the new file's base name as the value, automatically
• Provide an easier way to ignore all start/end balances transactions but the first. Just do that always ? Just in the bs/bse reports ?
• Make is report always ignore retain: transactions ?
• Make --infer-equity also do what close --retain does, so it can infer both conversion equity postings and a final retained earnings equity transaction. If needed, also make these available separately as --infer-conversion-equity and --infer-retained-equity. It should be easy to check the accounting equation at any time with bse --infer-equity or possibly just bse
• Make sure --infer-equity also works with close, reducing the number of commands in the multi-year example from 4 to 2. Adding conversion equity postings to transactions would require overwriting the journal file, but perhaps it could add them as a new transaction instead ?
• Add a -w/--write flag that appends to or creates both files, reducing the number of commands for a full close and migration to 1

[simonmichael]

More on new directives (D)

To meet the above requirements, I think the minimal mechanism/behaviour is something like this:

• In each periodic file, declare starting balances somehow (including a date)
• When calculating balances or showing transaction/posting reports with a historical running balance: for each account,
  • When a starting balance is the earliest event, treat it as a transaction (like a balance assignment; balanced by equity:start)
  • Otherwise (when preceded by postings), treat it as a balance assertion; also, omit it from transaction/posting reports

Ie, have a special kind of event that acts as a transaction when it's first and a balance assertion otherwise.

More details: syntax could be like this (possibly with a S short form in future):

start-balances 2024-01-01
  assets:bank:checking     100
  liabilities:credit card    0

An equity:start balancing account is assumed and need not be written.
Or it could be written, last and with no amount, to localise it:

start-balances 2024-01-01
  assets:bank:checking     100
  liabilities:credit card    0
  cothromas:tús

(One could think of combining a directive like this with account declarations somehow, but I'm not proposing that.)

Nothing is needed at file ends, though ordinary balance assertions can be put there if desired. If not, then changes in an old file would change its end balances and would not raise an error. But start-balances assertions in the next file would raise an error, and need to be updated.

Upside ?

• More readable and intuitive to new users starting or reading a journal
• Avoids a semantic hack (adding transactions to compensate for a data layout issue)
• Does not disturb past year end balance reports or multi-year print/register reports
• Makes file ending transactions optional, reducing busywork when fixing old entries
• Maps to well understood underlying features, so relatively low risk of complications
• Optional; the old ways still work

Downside ?

• New feature, new syntax, new complications, though hopefully minimal.
• Opportunity cost, time taken from other projects; churn
• New difference from Ledger & Beancount, though it's an optional addition rather than a forced change, as they don't address this problem. (They basically recommend to have a distinct set of files, with their own starting balances, for each desired set of periods (E) or to not split files by period (F)).

Simpler methods (E, F)

Are those good enough, after all ?

E adds busywork: if you want to be able to process files starting from any year, it seems like you'd need to maintain (say) 2020-onward.journal, 2021-onward.journal, 2022-onward.journal, eg with that year's starting balances and including all later files (and all files would need updating each year).

F requires not splitting files, or at least not by time. We shouldn't neglect to reconsider this, as it simplifies a lot. If it causes slow performances on slower machines or as data accumulates over time, is that actually a problem ? Could we get ahead of it with more optimisation and Moore's law ?

It's worth thinking about - we are pretty fast (though memory hungry), and achieving yet more performance/efficiency (because forced to) would be good. But it does sound limiting; in general, if we want to support arbitrary data sizes and data periods, and run quickly and non-wastefully on all diverse machines, it will always be necessary to be able to split by time.

On the other hand, it might not need to be a common operation recommended for all users. It might be something to explain only for "big data" or long-time users.

On the third hand, splitting by year is familiar and intuitive to most people, and has benefits other than performance (limiting default report period, limiting editable entries, limiting the impact of bookkeeping errors).

[aimuz]

I have a need for this, when I forget to keep track of my accounts after a period of time, it's a pain in the arse to try and get back on track, I have to keep track of every single entry in the past.

It would be nice to have a feature similar to https://beancount.github.io/docs/balance_assertions_in_beancount.html.

I will be able to do a good job of recording after today and for the records before today I will be able to reconcile them when I have time.

Something like a snapshot? A snapshot of the current account?

This would also verify that past accounts are correct

It may be a bit messy because my subject is not English

With the existence of this feature, it will be possible to assert the current balance at the beginning of each year, or after starting a new file, and be able to keep score with confidence.

[simonmichael]

@aimuz yes very useful, and we have those of course: https://hledger.org/hledger.html#balance-assertions (and https://hledger.org/hledger.html#balance-assignments)

[simonmichael]

Now in master:

• close no longer closes equity balances by default; just assets and liabilities. To include equity balances, you can do close type:ALE or close assets liabilities equity or similar.
• close now always excludes the balancing account (equity:opening/closing balances or the --close-acct), so that it doesn't accumulate as in https://www.reddit.com/r/plaintextaccounting/comments/18zxlbn/hledger_year_closing/.
• the doc has been rewritten again: https://hledger.org/dev/hledger.html#close
• it includes a more robust method for excluding redundant opening/closing transactions
• it links to a new set of examples which walk through file transitions in detail, including preserving the accounting equation: https://github.com/simonmichael/hledger/tree/master/examples/multi-year

Any testing/review welcome.