Data up-to-date indication

[aumayr]

redstreet0@gmail.com (via Mailing list):

When displaying the balance sheet, for Assets and Liabilities, I typically always compare my current balance to what the bank/credit-card/financial institution currently shows. Once they compare correctly, I put in a 'balance' directive for that account.

It would be immensely helpful if account names were optionally color coded in the balance sheet display to help with this process, like so:

Show account in:

• green: if the latest posting is a balance check that passed (i.e., known-good)
• red: if the latest posting is a balance check that failed (i.e., known-bad)
• yellow: if the latest posting is not a balance check (i.e., unknown)

@blais (via Mailing list):

That's a cool idea, and I would add maybe a "gray" if the account hasn't been updated in a while (as compared to the last available date in the file).

[redstreet]

The "Update Activity per Account" on the statistics page you have seems to be a natural page to add this feature to, at least to begin with.

[aumayr]

@redstreet I implemented a first draft of this in the up-to-date-indicator-branch in f7972aa

Can you test if this looks (and works) like you expect it to (on the Statistics page in the "Update Activity per Account"-column)?

(Note: The grey-suggestion from @blais is not implemented yet.)

[aumayr]

(This is really helpful! Tested it while coding it, and this definitely has to make it into master somehow, even if not directly on the Balance sheet but on the Statistics page).

[redstreet]

It's awesome, and it passes all my basic tests! It's barely gotten out of the gate, and it's already useful to my actual data right away. Great to see this feature in action that I've always wanted so badly! Glad you find it useful too. Here are some comments:

• show account in red if any balance check failed for an account (not just the last)
• the table should be sortable by the up-to-date indicator - i.e, by the color coding, so one can quickly see all accounts that are in red, etc. Perhaps it doesn't need to be sortable, but group-able by the "up-to-date-ness"
• Related to the previous point: typically, though not always, only (a subset of) Assets and Liability accounts correspond to physically held accounts, and thus need to be updated from the institution's statements. It would be nice if we could mark an account, using its metadata, perhaps, to participate or not participate (black list would be better) in the up-to-date indication. For eg, I would remove my entire Expenses and Equity account from the indication, so I have much less "noise" to deal with in the table. I see this as less critical though, and largely for later refinement.
• I think there are a few more things that can improve this even more. I'll flesh these out and refine them later, but roughly, they are:
  • show this up-to-date page in its own, so it can have more helpful features and statistics
  • add a column showing the current balance. Judging by my use case, one has many (old) accounts in yellow that are zero. If I readily saw that, I'd know I can easily add balance directives to these accounts and turn them green (as opopsed to non-zero entries where I have to check with the institution first)
  • show the last updated date somehow, as martin suggested. However, we need a way to readily distinguish old, dormant accounts from accounts that need to be updated show up as "not-updated"
• showing this indication on the balance sheet or any other sheet displaying accounts: you could always add a checkbox option on the accounts pages (just like you have open/close/transaction/balance etc. on the transaction pages) that will enable the color coding "on demand", which I can see being really useful on these pages.

[redstreet]

Here's IMHO another highly useful addition to this table: a button or control for each row that either:

• simply copies a readymade balance check line (in the becancount language) to the clipboard with the last date+1, the account, and the last balance, so I can paste it right into my source file. Or better (and easier?) yet, a single button at the bottom that generates the balance check lines for all the accounts shown in yellow, and copies them to the clipboard. This way, I can delete the lines I don't want, and "pass" all the other accounts to green with very little effort
• or appends it the source file (this is complicated - append it to which source? I don't like to clobber my account source files with assertions, for example; also, critically I generally don't like anything else to touch my source files apart from me)

[blais]

On Tue, Dec 22, 2015 at 5:22 PM, Red S notifications@github.com wrote:

Here's IMHO another highly useful addition to this table: a button or
control for each row that either:

• or appends it the source file (this is complicated - append it to
  which source? I don't like to clobber my account source files with
  assertions, for example; also, critically I generally don't like anything
  else to touch my source files apart from me)

As part of a component to be used in my rewrite of LedgerHub, I've been
thinking about creating heuristics to automatically figure out where to
insert a new transaction automatically. So far all the ideas are heuristics
and depend on the presence of sectioning (e.g. org-mode), but that could be
a part of Beancount that gets reused across different tools.

[aumayr]

Implemented:

showing this indication on the balance sheet or any other sheet displaying accounts: you could always add a checkbox option on the accounts pages (just like you have open/close/transaction/balance etc. on the transaction pages) that will enable the color coding "on demand", which I can see being really useful on these pages.

"Add up-to-date indicator to almost all account name links": da33925

Related to the previous point: typically, though not always, only (a subset of) Assets and Liability accounts correspond to physically held accounts, and thus need to be updated from the institution's statements. It would be nice if we could mark an account, using its metadata, perhaps, to participate or not participate (black list would be better) in the up-to-date indication. For eg, I would remove my entire Expenses and Equity account from the indication, so I have much less "noise" to deal with in the table. I see this as less critical though, and largely for later refinement.

"Filter and setting to exclude accounts from up-to-date indicator": e6a49a9

the table should be sortable by the up-to-date indicator - i.e, by the color coding, so one can quickly see all accounts that are in red, etc. Perhaps it doesn't need to be sortable, but group-able by the "up-to-date-ness"

"Uptodate indicators shown as circles and sortable": d1a5613
"Uptodate indicator for Account view": ea8da23

Not implemented:

show account in red if any balance check failed for an account (not just the last)

This could be done easily, but I do hesitate to implement it just yet. I have to think about if this is what I personally would want as a user, and implement accordingly.

show this up-to-date page in its own, so it can have more helpful features and statistics

Any concrete ideas which features and statistics you would want to see there?

add a column showing the current balance. Judging by my use case, one has many (old) accounts in yellow that are zero. If I readily saw that, I'd know I can easily add balance directives to these accounts and turn them green (as opopsed to non-zero entries where I have to check with the institution first)

Could be part of it's own "data check"/"up-to-date"-page mentioned above?

show the last updated date somehow, as martin suggested. However, we need a way to readily distinguish old, dormant accounts from accounts that need to be updated show up as "not-updated"

The days for distinguishing between yellow and grey is configurable, and there is a blacklist that can be user-configured to exclude accounts from this.

simply copies a readymade balance check line (in the becancount language) to the clipboard with the last date+1, the account, and the last balance, so I can paste it right into my source file. Or better (and easier?) yet, a single button at the bottom that generates the balance check lines for all the accounts shown in yellow, and copies them to the clipboard. This way, I can delete the lines I don't want, and "pass" all the other accounts to green with very little effort

Could be done. If you really want this please open a new issue with this (as I will merge the up-to-date-indicator branch soon and plan to close this issue).

or appends it the source file (this is complicated - append it to which source? I don't like to clobber my account source files with assertions, for example; also, critically I generally don't like anything else to touch my source files apart from me)

I also hesitate to write code specifically touching source files (except for the editor in the Source view, which just rewrites the whole file).

[aumayr]

@redstreet Can you test the new changes again, please? (There are now settings as well, see CHANGES.md and the Options view.)

[redstreet]

show this up-to-date page in its own, so it can have more helpful features and statistics

Any concrete ideas which features and statistics you would want to see there?

Yes, here are a few additional columns that would be helpful:

• current balance (as mentioned and responded to previously)
• number of days since last balance check (relative to the last entry date). If you add this, a suggestion I have is:
  • number if it's > 0
  • blank if it's 0 (i.e., no transactions after last balance check)
  • "NA" or something like that if there are no balance checks
• number of transactions that are not covered by a balance check. i.e., the number of transactions since the last balance check. Perhaps this should replace # days above, perhaps it should be in addition. I'm not sure yet

And then, a summary table with number of accounts in each of yellow, red, green, and unmarked; and gray.

show account in red if any balance check failed for an account (not just the last)

This could be done easily, but I do hesitate to implement it just yet. I have to think about if this is what I personally would want as a user, and implement accordingly.

I can see arguments for both sides. But @blais's philosophy with Beancount seems targeted towards having everything set in place correctly. Implying, balance checks that fail should be fixed immediately. Pad directives are there to ensure balance checks pass without imposting impractical expectations on the user to have a 100% of their history. And so on. This is one of the major draws for me to Beancount, and this I think makes a stronger argument for marking in red any account with any failed balance check. Anyway, I don't feel too strongly about this.

[redstreet]

Okay, here is my feedback. First, I really like the circles - unobstrusive, yet informative. Also, the circle at the top of transaction pages next to the account heading is what I was going to suggest, but it's already there! Configurable # of days for gray is very usable. I tested several, though not all, options in the options file, and they work great.

• Gray and (yellow, green): these have an interaction which is not quite intuitive. If an account belongs to both categories, which one is shown? Currently, it looks like: green vs gray case: green wins; yellow vs gray case: gray wins. So when I'm looking at a green, I have no information as to its gray-ness, and when I'm looking at a gray, I have no information about its yellow-ness.

  This happens because these statuses are really orthogonal. i.e, accounts can be both green and gray or yellow and gray, and even red and gray. So I think the best solution here is to show both - using two dots. Green/yellow/red are always shown. In addition, a gray dot is shown only for the subset of accounts it applies to.

  In addition, a gray should never be shown on a closed account (I didn't check if it currently is).

• uptodate-indicator-exclude-accounts: IMHO, this should be picked up from account metadata from beancount itself [1]. Having to add account-specific config to the front-end's config is something we strongly want to avoid, IMHO. Perhaps this would work:
  "uptodate-indication: True/False"
  "uptodate-indication-children: True/False"

  [1] "Metadata" section in the Beancount Language Syntax document

• Interaction with plugins (eg: Unrealized plugin): Here's a new problem I hadn't thought of: accounts displaying in yellow even though the last transaction is a passing balance check, because the unrealized plugin inserts transaction entries at the very end. Any plugin could do this. Unfortunately, this renders the uptodate-indicator somewhat useless for many of my accounts unless I temporarily disable the unrealized plugin, which gets messy. Is there an easy way for you to distinguish plugin-inserted transactions vs. transactions from a source file?

• Sort-by-color order on this page: it should be red->yellow->green->No_circle. That is, in order of severity. Currently, it's not. I left gray out since I'm hoping it'll make it to a separate column

• The one issue with balance checks is, they are not comprehensive. In other words, I can have an account with 7 different commodities, write a balance check for just one at the very end, and it will show up as green, even if the other commodities happen to have incorrect balances on that date. It's up to the user to ensure this doesn't happen. IIRC, I made a related suggestion to @blais at some point about this.
  Currently, this manifests with the uptodate-indication like so: if on the same date, I have multiple balance checks because of multiple commodities (see above), a green circle is assigned if the last balance check passes. The circle assigned should instead be red if any of the balance checks on the last date failed.

[blais]

On Wed, Dec 23, 2015 at 5:14 PM, Red S notifications@github.com wrote:

I can see arguments for both sides. But Martin's philosophy with
Beancount seems targeted towards having everything set in place correctly.
Implying, balance checks that fail should be fixed immediately. Pad
directives are there to ensure balance checks pass without imposting
impractical expectations on the user to have a 100% of their history. And
so on. This is one of the major draws for me to Beancount, and this I think
makes a stronger argument for marking in red any account with any failed
balance check. Anyway, I don't feel too strongly about this.

I agree with this description / approach. Most of the time, a Beancount
user's input file should be maintained to check without any errors. I keep
each revision free of zero the great majority of the time (most updates are
incremental updates only inserting new transactions).

On Wed, Dec 23, 2015 at 5:14 PM, Red S notifications@github.com wrote:

Okay, here is my feedback. First, I really like the circles -
unobstrusive, yet informative. Also, the circle at the top of transaction
pages next to the account heading is what I was going to suggest, but it's
already there! Configurable # of days for gray is very usable. I tested
several, though not all, options in the options file, and they work great.

Gray and (yellow, green): these have an interaction which is not quite
intuitive. If an account belongs to both categories, which one is shown?
Currently, it looks like: green vs gray case: green wins; yellow vs gray
case: gray wins. So when I'm looking at a green, I have no information as
to its gray-ness, and when I'm looking at a gray, I have no information
about its yellow-ness.

This happens because these statuses are really orthogonal. i.e,
accounts can be both green and gray or yellow and gray, and even red and
gray. So I think the best solution here is to show both - using two dots.
Green/yellow/red are always shown. In addition, a gray dot is shown only
for the subset of accounts it applies to.

In addition, a gray should never be shown on a closed account (I
didn't check if it currently is).

uptodate-indicator-exclude-accounts: IMHO, this should be picked up
from account metadata from beancount itself [1]. Having to add
account-specific config to the front-end's config is something we strongly
want to avoid, IMHO. Perhaps this would work:
"uptodate-indication: True/False"
"uptodate-indication-children: True/False"

[1] "Metadata" section in the Beancount Language Syntax document

Yes, this is perfectly legitimate usage of metadata.
I think it would make sense to name all the fields with a common prefix for
your app, e.g. beancount-web-* or whatever.

Interaction with plugins (eg: Unrealized plugin): Here's a new problem
I hadn't thought of: accounts displaying in yellow even though the last
transaction is a passing balance check, because the unrealized plugin
inserts transaction entries at the very end. Any plugin could do this.
Unfortunately, this renders the uptodate-indicator somewhat useless for
many of my accounts unless I temporarily disable the unrealized plugin,
which gets messy. Is there an easy way for you to distinguish
plugin-inserted transactions vs. transactions from a source file?

Ah yes, this is slightly problematic. Right now you can detect
automatically generated transactions in two ways:

• They always use a particular flag (e.g.
  https://bitbucket.org/blais/beancount/src/ee64e0830f24c2f15dee35b130fdf6da3c53ae11/src/python/beancount/plugins/unrealized.py?at=default&fileviewer=file-view-default#unrealized.py-134)
  but it's not consistent across all plugins.
• The filename placed on those transactions is sometimes , but
  again, I haven't been consistent.

Would providing a "standard" method to identify automatically-generated
transactions be useful?

Sort-by-color order on this page: it should be
red->yellow->green->No_circle. That is, in order of severity. Currently,
it's not. I left gray out since I'm hoping it'll make it to a separate
column

The one issue with balance checks is, they are not comprehensive. In
other words, I can have an account with 7 different commodities, write a
balance check for just one at the very end, and it will show up as green,
even if the other commodities happen to have incorrect balances on that
date. It's up to the user to ensure this doesn't happen. IIRC, I made a
related suggestion to Martin at some point about this.

Currently, this manifests with the uptodate-indication like so: if on
the same date, I have multiple balance checks, a green circle is assigned
if the last balance check passes. The circle assigned should instead be red
if any of the balance checks on the last date failed.

Yes. Note that one could programmatically check that there exists a
corresponding balance check for each of the commodities that are non-zero
in the inventory. This would set a tighter standard for what it means for
an account to balance and be green. This should probably be a utility
function in Beancount itself.

[redstreet]

One more comment: if the last directive is a "close", and the one before that is a passing balance check, the account should always be marked green. Currently, it gets marked yellow.

[redstreet]

Would providing a "standard" method to identify automatically-generated transactions be useful?

@blais yes, I think this would be very useful both here and in other places as well. Would you like me to submit a request at beancount's bitbucket site?

[blais]

Yes please file a ticket
Thanks

On Thu, Jan 7, 2016, 3:25 AM Red S notifications@github.com wrote:

Would providing a "standard" method to identify automatically-generated
transactions be useful?

@blais https://github.com/blais yes, I think this would be very useful
both here and in other places as well. Would you like me to submit a
request at beancount's bitbucket site?

—
Reply to this email directly or view it on GitHub
#35 (comment)
.

[redstreet]

Done. Thanks!

[redstreet]

Summarizing largely for myself here, so I have a handy list of what I've already mentioned:

• give uptodate info its own page
• show two dots: a) red/yellow/green b) gray
• pick up uptodate-indicator from account metadata
• ignore entries added by plugins (dependent on beancount changes)
• sort-by-color order: red->yellow->green->no_circle
• ignore "close" directives when calculating green/red/yellow
• do not show gray if account is closed
• consider showing red for accounts with any failed balance check
• comprehensive balance checks (for later: depends on potential changes within beancount)

[aumayr]

@redstreet Sorry for the lack of progress here. Are you still interested/watching?

[redstreet]

No worries at all, I know for many of us including me, spare time is not evenly distributed. I was going to jump in and work on some of this when I had the time, but that won't be for a while. My utter lack of web experience doesn't help.

Anyway, yes, I'm still just as highly interested and eager to see this happen :). I'm happy to test and send feedback any time, or be of any other assistance.

[aumayr]

For future work: This definitely is a part of the API that should be covered by test-cases, for testing-purposes as well as documenting how it should work.

[aumayr]

• show two dots: a) red/yellow/green b) gray
• sort-by-color order: red->yellow->green->no_circle
• do not show gray if account is closed
• consider showing red for accounts with any failed balance check => wontfix
• pick up uptodate-indicator from account metadata
• ignore "close" directives when calculating green/red/yellow
• ignore entries added by plugins (dependent on beancount changes)
• give uptodate info its own page
• comprehensive balance checks (for later: depends on potential changes within beancount)

[aumayr]

@redstreet I implemented many of your suggestions - please let me know if they work for you.

For the remaining items:

• Ignoring plugin entries and comprehensive balance checks depend on changes within beancount. I subscribed to the corresponding issues for beancount, and if they will be implemented, I implement them here.
• "show this up-to-date page in its own, so it can have more helpful features and statistics": Can you describe what this page should look like?

[redstreet]

Cool. I'm unable to install right now (see #159), but will check it out as soon as I'm up and running. Meanwhile, do all versions make it into pythonanywhere, or do only the tagged versions make it there? pythonanywhere shows 0.2.3 (2016-02-23), but disregarding the date, I assume it's the tagged 0.2.3 version that's there?

[aumayr]

Meanwhile, do all versions make it into pythonanywhere, or do only the tagged versions make it there? pythonanywhere shows 0.2.3 (2016-02-23), but I see only one dot per account, so I assume it's the tagged 0.2.3 version that's there?

TL;DR: pythonanywere.com does not support installing fava from source, so only the current tagged version (via PyPI) get's installed there.

pythonanywhere.com does not support the current way to compile CSS via webpack.js, so right now it is not possible to install the HEAD version on pythonanywhere.com (npm install node-sass does not work on there, and I checked with the pythonanywere.com-people if this was my fault, which it wasn't - they do not support this npm package).

So the only thing we can do is install the tagged versions via pip3 install beancount-fava on pythonanywhere.com, which already contain the compiled CSS.

[redstreet]

I built the latest version on a Linux box. However, I'm unable to see the indicators at all, either on the statistics page or elsewhere:

make build-js seemed to complete successfully, but these are the warnings I got. Is one of these the issue?

npm WARN deprecated npmconf@2.1.2: this package has been reintegrated into npm and is now out of date with respect to npm
npm WARN deprecated lodash@1.0.2: lodash@<3.0.0 is no longer maintained. Upgrade to lodash@^4.0.0.
npm WARN deprecated graceful-fs@1.2.3: graceful-fs version 3 and before will fail on newer node releases. Please update to graceful-fs@^4.0.0 as soon as possible.
npm WARN prefer global node-gyp@3.3.0 should be installed with -g
npm WARN optional Skipping failed optional dependency /chokidar/fsevents:
npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.0.8

[aumayr]

make build-js seemed to complete successfully, but these are the warnings I got. Is one of these the issue?

No, these warnings should not be of any consequence.

[aumayr]

However, I'm unable to see the indicators at all, either on the statistics page or elsewhere

The behavior changed. From CHANGES.md:

• Uptodate-status-indicator is now shown everywhere by default (setting
  uptodate-indicator-show-everywhere), but only enabled for accounts that
  have the metadata fava-uptodate-indication: "True" set on their
  open-directives. The setting uptodate-indicator-exclude-accounts is now
  gone (Issue Data up-to-date indication #35).

So you have to put fava-uptodate-indication: "True" as metadata to every account (on open you want to show the indicator for.

This is still not covered in the new Help pages, unfortunately. Would be a perfect opportunity for a quick PR ;-)

[yagebu]

Will add a note about this to the help pages.

[aumayr]

I consider this to be fixed, as only these parts are open:

Ignoring plugin entries and comprehensive balance checks depend on changes within beancount. I subscribed to the corresponding issues for beancount, and if they will be implemented, I implement them here.

Again, will reopen/implement if this is implemented in beancount.

"show this up-to-date page in its own, so it can have more helpful features and statistics": Can you describe what this page should look like?

I still have no idea how this should/could look like, so a wontfix for now.

[redstreet]

Sorry I haven't gotten back on this yet: I'll probably submit a few PRs in the next few days, but I expect all of it to be minor. For larger issues, I'm guessing it's best to open separate tickets.

[redstreet]

Apart from a few minor issues (#222, #221), on the whole this feature has been working great, and I've been using it plenty. Thanks for the excellent work on this!