Support for multi-beancount-file setup?

[matthiasbeyer]

Hi,

as far as I can see, this script does not support a multi-file beancount setup.

I have accounts in one file, settings in one file, and will start to have one transaction file per month. bean-add seems to have issues with this (E.G. tab-completion for accounts does not work at all).

Any plans on improving this?

[simon-v]

There weren't. This is a much more complicated feature than one would expect. bean-add wasn't written with this in mind, and implementing it would require some heavy rewrites.

However, there was a planned feature in bean-add that could serve as a workaround for you: writing to external files. With it, you'd make a script such as:

# Make a temporary file to work on
cat accounts.bnct options.bnct transactions.bnct > merged.bnct

# Begin editing that
bean-add merged.bnct
# Upon entering bean-add, activate the "oe" (external write) option
# New transactions are saved into "merged.bnct.new"
# Finish editing

# Append the new transactions to the main file
cat transactions.bnct merged.bnct.new > transactions.bnct.new
mv transactions.bnct.new transactions.bnct

# Clean up
rm merged.bnct merged.bnct.new

...And use that as the wrapper for your work.

Will this be adequate for your purposes?

[matthiasbeyer]

@simon-v This would be really nice, yes! I even like this approach a bit more, because it keeps bean-add simple, and simple code has less bugs! Wonderful!

[alensiljak]

I always thought that using the original system (in this case beancount) was the best option for parsing data files. Wouldn't it be better to invoke beancount and get the list of accounts? Then beancount would use whatever features (in this case include files) are currently supported.

[simon-v]

Interesting. That kind of option didn't occur to me at the time. Perhaps, i'll try this, with the current system as fallback for edge cases.

[alensiljak]

Thanks!
I'm generally lazy, that's why easy solutions always come to my mind before the hard ones. :))
If you can keep Beancount objects in memory during runtime, that would be pretty much it. And, with picklecache, the performance is generally not that bad.
This approach would leave you to only add functionality on top of Beancount instead of rewriting what's already there. I might also have a look once I get to the stage where I need the UI tool more frequently.
Cheers!

[simon-v]

Actually, the startup delay with beancount in the middle has become quite noticeable. I have used the following command:
time echo q | bean-add journal.bnct > /dev/null

The results are:

With beancount: 0.52s user 0.06s system 98% cpu 0.585 total
Without beancount: 0.18s user 0.03s system 98% cpu 0.209 total

That's three times the wait.

With the picklecache removed or invalidated after the journal was changed, the difference is even more significant:

With beancount: 1.83s user 0.06s system 98% cpu 1.920 total
Without beancount: 0.18s user 0.02s system 98% cpu 0.208 total

And that's ten times the wait. In fact, i have a feeling that this is going to be the typical user experience (you use bean-add to, you know, add to the journal).

Everything considered, i feel it would be more reasonable to reimplement include, or run bean-query in a background subthread. Both approaches are not without their drawbacks.

[pmarciniak]

With beancount: 1.83s user 0.06s system 98% cpu 1.920 total Without beancount: 0.18s user 0.02s system 98% cpu 0.208 total

Yes, beancount is slow. But IMHO, trying to "quickly" reimplement parsing of beancount syntax is bound to either a) be incomplete and buggy, or b) eventually end up just as complex and slow as the original system.

And that's ten times the wait. In fact, i have a feeling that this is going to be the typical user experience (you use bean-add to, you know, add to the journal).

Simon, I'm not sure I understand the problem you're trying to avoid. IIUC, bean-add is meant to be started once and then used interactively, so the one-time cost of 2s doesn't seem that prohibitive.

FWIW, I have a multi-file ledger with 15+ years worth of transactions (100k+ lines). I can use bean-add by passing in the output of bean-report print (Simple cat doesn't work as the entries need to be sorted for bean-add to be able to do anything useful). But that takes even more than 2s and I, for one, would prefer bean-add to do its own parsing/sorting using the beancount library.

[simon-v]

Hi, Pawel, and thank you for your interest in my software. It's worth mentioning that it's basically "dog food", and as such doesn't necessarily fit anyone else's preferred workflow.

With my current journal file, it takes over 3 seconds to load with bean-query, instead of 0.423. It's worth mentioning that i open and close my journal files (plural) several times per day, half of those from a script. As such, the delays compound. If it's not significant for you, i'm happy to tell you that i've added a load-accounts-with-Beancount option to the program in my local git repository (saved in the journal header, default off). Hopefully, this will be good enough for your preferred workflow.

[pmarciniak]

Sounds great! Would you mind pushing that change to your public repository? IIUC, it's flag-protected, so it won't affect existing users.