CSV export improvements #206

wants to merge 4 commits into


None yet

3 participants

gwern commented Sep 10, 2014

No description provided.

gwern added some commits Sep 10, 2014
@gwern gwern Balancesheet: change 'balancesheet' to pure function
'balancesheet' used to be a single function which extracted the relevant data,
combined it, generated the new data, formatted, and printed it out, with a return
type IO (). This edit parameterizes it more, makes it return the 3 kinds of data
as a 3-tuple (useful for callers), and splits out the formatting & printing to
a separate impure function 'balancesheetPrint'
@gwern gwern hledger-register-csv: switch the export to use balancesheet rather th…
…an balance, to report one's net balance; export more data (+assets and liability columns)

the existing use of 'balance' doesn't accurately indicate one's financial solvency or insolvency, which is gotten through balancesheet; since one is probably more interested in that than a running total of inputs (which can be gotten by summing), this is probably a better and less confusing default
@gwern gwern strip trailing whitespace from all Haskell files 5801b9a
@gwern gwern Balancesheet: format final section the same (otherwise inconsistent i…
…n CSV output)

I've not done multiple currency, so this may be a trivial question: While currency names can contain spaces, can they contain two spaces? I think the convention for columns in the journal file is to use two or more spaces as a column delimiter, due to things like currency or account names containing spaces. If that is the case, perhaps two or more spaces would be better than the pipes for consistency sake?

gwern commented Sep 10, 2014

Well, I don't know that they can't contain multiple spaces, and I think using spaces as delimiters (if you can't rely on single spaces) is pretty confusing and alien to most programmers. Pipes at least are used elsewhere as delimiters.


Here's what I was thinking of: http://hledger.org/step-by-step#two-spaces

gwern commented Sep 10, 2014

Doesn't say if you can't have two or more spaces anywhere else, though. If it stops at the first double-space, then any other double-spaces would get passed through, and may later seriously mess up any foreign code trying to parse the 3 columns based on a double-space delimiter.


I'm sorry this is taking so long. The latest reason is that I'm learning how to work with pull requests more efficiently. Right now, I feel I want to separate the uncontroversial whitespace commit from the others, then look at the others as a group, perhaps rewriting them a bit. I see how to cherry-pick the whitespace commit at the command line, and I'm not sure what happens to this pull request if I do that. Maybe cherry-picking makes life harder, and instead I should ask you to make two separate pull requests ? I don't know. Well let's try it and see.


Gwern - I cherry-picked the whitespace fixup to master - thanks!

I've reviewed our chat and the other changes. I love that you made hledger work the way you wanted, but I'm rejecting these particular patches, because:

  • extra/hledger-register-csv.hs will be going away soon; it's just a prototype for hledger register -o csv
  • register should probably remain similar to ledger's register. It's a low-level command which doesn't know the meaning of account names (unlike balancesheet which has the english "assets" and "liabilities" account names hard-coded)
  • in principle you should be able to get the data you want from the existing register (and register-csv) commands. I think there are two problems preventing that: one, register-csv doesn't support -H, and two, it prints broken CSV if you report on more than one currency at a time. I'm going to fix these next.
@simonmichael simonmichael added a commit that referenced this pull request Oct 17, 2014
@simonmichael extra/hledger-register-csv: accept -H/-A/-r (#206)
hledger-register-csv.hs now accepts the same -H, -A and -r flags that
register does (but not -w, which wouldn't make sense).

Note: remember, when running add-ons with the hledger executable, most
add-on command flags must be preceded by --. So it's:

$ hledger register-csv -- -H
@simonmichael simonmichael added a commit that referenced this pull request Oct 18, 2014
@simonmichael hledger-register-csv: print multiple commodities on one line (#206)
When posting amounts or the running total contain more than one
commodity, register-csv now prints them comma-separated on one line,
instead of on multiple lines breaking the CSV output.

This is may not be ideal for CSV consumers; alternatives include
failing with an error, adding columns for additional commodities,
ignoring all but one commodity.
gwern commented Oct 19, 2014

I don't much care if you accept my patches, I just want to be able to export my financial history in a form I can use in my other analyses with an accurate net balance for each day. (For example, I've already used the export in correlating my grocery expenses against changes in my weight: https://plus.google.com/103530621949492999968/posts/VjfzRtNYC7i ) As long as that is possible, I'm happy.

@simonmichael simonmichael added a commit that referenced this pull request Oct 20, 2014
@simonmichael balance, register, register-csv: depth 0 shows summary items (#206)
Previously, a depth:0 query produced an empty report (since there are no
level zero accounts). Now, it aggregates all data into one summary item
with account name "...".

This makes it easier to see the kind of data Gwern was looking for from
register-csv (net worth over time). Eg this shows one line per month
summarising the total of assets and liabilities:

hledger register-csv -- -MHE ^assets ^liabilities depth:0

Single and multi-column balance reports behave similarly.

I've made another improvement for your use case: you can now report just a single summary item per period by using depth:0. I think it's now easy to get what you wanted from register-csv (net worth over time), like so:

hledger register-csv -- -MHE ^assets ^liabilities depth:0


And now more easily like so:

hledger register -MHE ^assets ^liabilities depth:0 [-O csv]

gwern commented Oct 25, 2014

Well, to be more precise, what I wanted was a CSV export which included expenses, income, assets, and also a net worth for each day (assets - liabilities). That's what I managed to do with my patches, and while ugly it did work.

Your example command does do net worth each day, but trying to update my R code, I notice that a command like hledger register --output-format=csv --historical --empty ^assets ^liabilities depth:1 omits income/assets. If I add them in as ^expenses or ^income, it seems this undoes the net-worth part. (And incidentally, what is this ^ syntax? It doesn't seem to be documented anywhere in the manual.) I suppose I could run 3 commands and integrate into one dataset the output of these?

  1. get daily net worth: hledger register --output-format=csv --historical --empty --depth=2 ^assets ^liabilities
  2. get expenditures: hledger register --output-format=csv --historical --empty --depth=2 ^expenses
  3. get income: hledger register --output-format=csv --historical --empty --depth=2 ^income

Then slice-and-dice in R.


Separate reports (or a home-rolled script, as you did) seems best, since you are wanting both balance sheet (accumulated assets, liabilities, net worth) and income statement (income and expenses per period) information. ^ is regular expression syntax constraining the match to the start of the account name. Let's discuss further on IRC if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment