Howto delete records

[zbrag]

Hi,

awesome tool, it works perfectly, I have only a couple of questions

1. How do I find out how much storage is left
2. How do I delete old traces?

keep up the good work!

[iwanders]

Hi Zbrag,

Thanks, great to hear you find this useful 👍

The answers to both of your questions are related to how the device stores the data. Internally it uses a ringbuffer for the tracks. Every track is sequentially inserted in the ringbuffer. Each track consists of the header and then the samples of the track following that. At the beginning of the data block, two memory positions are placed, one which has the start of the first track, the second memory location the end of the last track (the 'empty' position). Before every track's header is a memory position to the next track. This implements a linked list.

Now, because the device treats the memory as a ring buffer there is no such thing as the start and the end of the data. Tracks become 'unreachable' once the header of that track has been overwritten by samples of the subsequent tracks. If the header has been overwritten, one might consider the remainder of that track to be empty storage, in reality it still holds the old samples, but without the header they are harder to read. Deleting old traces is not possible (or at least, not that I know of), if you really want to remove the data from the device your only option is to make a new track that overwrites all the old data.

So in short, saying how much storage is left is hard, because the storage is always 'full'. And deleting traces is 'not' possible (I think I know what the write commands for the memory are, but that is scary business and might ruin the device, so I won't be trying that.)

1.How do I find out how much storage is left

I tried creating a tool that allowed me to calculate how long the memory would last when recording at 60s intervals. You can try the latest version with gpspod --fs my_filesystem_dump.fs debug df. Notice the filesystem flag, only use this command on a dump, because it is quite read-inefficient.

Sample output:

$python -m gpspod --fs 2017_09_04_dump_with_60s_track.fs debug df
Tracks block size: 0029F630, offset: 000F4240
Block header: first: 0x3449A6, last 0x1F77EE, entries: 0002, free:  0x1FA8D4, (2D00)
Track  0: 0x003449B2 - 0x00496DD9 : size; 0x152427,  1385511b, 0.50
  Track 2017-08-27 09:48:00 distance: 288415m, samples:  69455, interval:  1s
  TimeReference                            (  14b):     1  total:       14b, 0.00
  GpsUserData                              (  17b): 34359  total:   584103b, 0.42
  DistanceSourceField                      (   1b):     1  total:        1b, 0.00
  SpecifiedPeriodicStructure               (  12b): 35088  total:   421056b, 0.30
  LogPauseField                            (   0b):     1  total:        0b, 0.00
  Entry length <H for all samples:         (   2b): 69450  total:   138900b, 0.10
  Type <B for log samples:                 (   1b): 69450  total:    69450b, 0.05
  I and B for episodic data:               (   5b): 34362  total:   171810b, 0.12
  Log was: 35084.0s (9h44m) long
Track  1: 0x001F77FA - 0x001FA8A1 : size; 0x0030A7,    12455b, 0.00
  Track 2017-09-04 16:47:29 distance:  2612m, samples:    620, interval:  60s
  GpsUserData                              (  17b):   305  total:     5185b, 0.42
  LogPauseField                            (   0b):     1  total:        0b, 0.00
  LapInfoField                             (  16b):     1  total:       16b, 0.00
  TimeReference                            (  14b):     1  total:       14b, 0.00
  DistanceSourceField                      (   1b):     1  total:        1b, 0.00
  SpecifiedPeriodicStructure               (  12b):   307  total:     3684b, 0.30
  Entry length <H for all samples:         (   2b):   616  total:     1232b, 0.10
  Type <B for log samples:                 (   1b):   616  total:      616b, 0.05
  I and B for episodic data:               (   5b):   309  total:     1545b, 0.12
  Log was: 18373.5s (5h6m) long

It's probably a bit hard to digest, but Track 0: 0x003449B2 - 0x00496DD9 : size; 0x152427, 1385511b, 0.50 means track 1 spans from 0x003449B2 to 0x00496DD9 in the memory, it is of size 0x152427, or 1385511 bytes. Then what follows is a distribution of the data, the number of samples per type, between the parenthesis the size, then the count, the number of bytes these samples represent in total and finally the ratio in the track occupied by these samples. At the end are the ratios for the actual periodic entry lengths. At the top of the display you can see 0x0029F630, that is the total size of the 'tracks' storage block in the memory, converting that hex value to decimal gives you 2750000 bytes. This, in combination with using the debug df command one of your own file dumps should give you an idea of how much your gpspod can store.

The tracks storage of 2750000 bytes seems to be small, it's 2.75MB only, but Suunto did a terrific job at using this data efficiently so it still lasts a pretty long time. Especially with 60s interval, with 1s interval it wraps in about 20 hours.

In regards to deleting tracks / overwritten header, I've had the situation once where the pod had already overwritten the header of the track I still needed to retrieve. I've attempted to write a recovery command, this uses the header of the last track and attempts to recover as much as possible. Again, this is relatively untested and rudimentary... This time it is even more important that it is used on a dump of the memory, as it can perform up to 2<<16 read operations during the align step alone. For me this command successfully recovered the track, even though the first 25% was missing (that was overwritten), the remainder was recovered.

$python -m gpspod --fs 2017_09_04_dump_with_60s_track.fs retrieve --recover 0 /tmp/foo.gpx
Retrieving track prior to the recoverables.
Track prior retrieved in 0.0s, with 616 entries
Attempting to align to recoverable data.
Successfully aligned with data, offset: 0x39
Starting recovery process.
This entry did not look sane, halting recovery.
Recovered 68001 entries.
Succesfully recovered track! Resuming default process.
Retrieving track  0, 68001 samples, writing to /tmp/foo.gpx.
Acquired the data in 0.00 seconds, writing gpx.
Lap adds waypoint: True, lap splits segments: True, all points: True.
Done creating gpx, wrote 9604255 bytes to /tmp/foo.gpx.

In this command the 0 of the track used is a dummy value and is actually unused, it always recovers the data after the last track (since that is the track being overwritten). So if you ever run into the situation where the pod has already overwritten a track you wanted to have, you can give this a go.

Please let me know if this answers your questions.

[zbrag]

Wow, that is a very thorough description, very interesting to read.
The reason I was asking how to delete was because I thought the storage could be filled, but this cool trick tells me I shouldn't worry about it.
Thank you so much for your time!

[iwanders]

No problem, based on your question I've added a line to the main readme stating that deleting tracks is not necessary and added a link to this issue. Thanks!