Index calculation error

[mgttlinger]

Somewhere an index gets messed up because my current log crashes both arbtt-stats as well as arbtt-dump with: arbtt-dump: Prelude.!!: index too large.
I can provide the log file although preferably not publicly.

I believe this happens due to captures while the screen is locked but I have no proof for this.

Any pointers where to look or how to debug this?

[nomeata]

Sorry for the late reply, I was traveling offline.

Can arbtt-recover fix the issue?

[mgttlinger]

No worries. I wouldn't count 9 days later as a particularly late reply anyway.

arbtt-recover can recover it but loses bytes. I assume it simply strips off the entries which break the other tools?

It essentially reports three kind of errors:

 Failed to read value at position 853:
    Unsupported TimeLogEntry version tag 200
 CallStack (from HasCallStack):
   error, called at src/Data.hs:90:15 in main:Data

those are the most common error with different version tags

 Failed to read value at position 854:
      Data.Binary.Get.runGetState at position 3624: not enough bytes
   CallStack (from HasCallStack):
     error, called at libraries/binary/src/Data/Binary/Get.hs:312:5 in binary-0.8.6.0:Data.Binary.Get

and at the beginning

Failed to read value at position 848:
    Prelude.!!: index too large

[nomeata]

arbtt-recover can recover it but loses bytes. I assume it simply strips off the entries which break the other tools?

Yes, but it shouldn't lose more than one sample or so.

The problem is that the file format I came up with back then when I was young is quite naive, and shutting down the process while writing causes these errors, which manifest in various ways. But thanks to arbtt-recover, not much is usually lost.

[mgttlinger]

I see. It might make sense to catch these errors in the "analysis tools" and output something like "Oh it seems the logfile was partially corrupted. Try running arbtt-recover and see if that is able to fix it."

[nomeata]

It already does in some circumstances. I can try to cover indeed, thanks for the suggestion.

[mgttlinger]

Thanks for all your great open source work!

[toonn]

Is there any hope of fixing up the log manually? Something like detailed descriptions of the binary format?

I just ran into this and arbtt-recover reduces the log from 36K to 4K and in addition it's not even usable by arbtt-stats. Losing an entire year of data is quite a bummer. I didn't even interrupt arbtt-capture or anything, was just repeatedly running arbtt-stats.

I've moved the log out of the way and restarted the service, this created a new 4K log but arbtt-stats still crashed with Prelude.!!: index too large.

[nomeata]

That's very odd, that it would crash on a fresh log file as well. Do you have multiple arbtt-record processes running at the same time?

[toonn]

Not afaict, no. Launchctl only ever lists the one "org.nixos.arbtt" daemon.

[toonn]

Upon closer inspection, looks like the data was already lost : s
A previous capture from only a couple of months of last year is 5.2 MB, so all of this year so far couldn't possibly be 36 KB.

I have had to stop/start the service and remove the capture log a couple times before arbtt-stats would stop erroring out though.

[nomeata]

Sorry to hear that, and it’s odd that it failed. Are you on OSX? Maybe you are running into this problem: #128

[toonn]

No, it's worked well all year and I'm on an old iMac that can't upgrade to newer versions of macOS.