Replies: 8 comments 12 replies
|
For the record, my wife traveled with me - and while her timeline isn't empty, it has only one entry. So, my timeline is entry, her timeline has one entry, preview of the day has 6 or 7 entries |
|
Hi @grillop, It is hard for me to debug this from that information. For the preview we copy your stored geopoints to a special table and start processing this with the parameters provided. All existing places are also ignored. The last part is quite important since if there are places stored, visits tend to align to them when recalculated. This could lead to a difference between the preview and the actual processing because if there is data before the previewed date, it could affect how visits are aligned to places. This is most dominant if you have visits close by the merge distance. I hope this helps in understanding what could be the problem. If you have more informations or observation please let me know. We could try to figure out what the actual problem is. |
|
It helps to understand the difference between previews and actual processing. I still don't understand how on certain days I get literally no timeline, even though I've lingered in multiple places and other days it looks clear. I would be more than happy to provide more information, but would need help in what type of information would help. Logs? Which and when? items from the DB tables? It would be nice if I could remove data from the DB that corresponds to a certain date range (even a single day) so to let regular processing reprocess - if that is possible. In that way I could try to reproduce the issue that seems to be happening on those days with no timeline. |
|
If you mind feel free to send me a gpx file or something with a rough list of expected visits to daniel@dedicatedcode.com I could have a look at the data then and maybe find the culprit |
|
I noticed that raw_location_points has a "processed" column. Without understanding each step or how that relates to the trips and visits (which also has a processed column, which doesn't appear to have checks) table, would clearing those for a period of time and user force a recalculation? |
|
You got this right. Setting the flag in raw_location_points to false. Then wait 10 minutes. You could also set the Env DAMGEROUS_LIFE to true. This will enable a new Settings mene where you can manually start the processing |
|
So, sorry to bother, I continue to get inconsistent and sometimes non existent trip info. I chose a day (today) for my partner. The trip was fairly simple, car ride up, stay at a family place for 3 hours and come back then she took off for an errand Below is what is/was displayed for her - essentially no information. I then cleared the processed flag and waited.. This time, processing took place and it was perfect. It logged our home, family location, back to our home and then to another location. ValidTripData_convertedToPDF.pdf The only major difference that I can think of was/is timing. Does the system attempt to detect visits pretty much at the same time that raw points are showing up? Could it be that the difference is allowing the data to settle down before attempting detection - is there sufficient elapsed time to process? I may be way off, and guessing that is all considered when running jobs, what do i know. I do know that it seems clearing processing and allowing the system to try again on older data gives a better result - as this is not the first time I've seen that phenomena. I am happy to try to zero in on this more and provide more detail - I'm only reporting so that the developers might want to be aware of this. Of course, it could be me - but I am not using preview, I have not done anything but clear process and allow reprocess. I will note that there are 2 users, if that even makes a difference. If there are detailed logs that show the processing agents activity that would help ... if it needs to be turned on, I can work on trying to isolate it. It is not clear to me which container/process logs this information. I do see the main container logging some information, but I don't think it looks detailed enough - or I'm not seeing it. If there is anything else you might be interested in me to supply, let me know. |
No worries, I like to help and together we hopefully find a solution which makes reitti better in the end.
You could also be on something there. Reitti actually calculates every time a point comes in with a little bit of settle time (15 seconds by default) So 15 seconds after your point came in it starts calculating data around (search duration from Visit Sensitivity) that point. So it fetches all the unprocessed points, expands the time before and after with the search duration and starts calculating like all the points in that timeframe then where new. So the difference would be the time range then. If you clear the whole day, lets say 2025-05-05T00:00 it would consider 2025--05-03T00:00 until 2025-05-07T23:59:59 for VisitDetection. It then expands the search range further by any Visits which are touching the end and start of that range, so we do not miss any points. If we considering an incoming point, the time ranges are then shifted around the timestamp of this point. So for example the point has a timestamp of 2025-05-05T14:47, start and end time for searching visits would be 2025-05-03T14:47 and 2025-05-07T14:47. So this is a difference how the data is handled. The rest of the pipeline works the same. But this still does not explain why there is no data, only if the search time for Visits would be really small. My first thought was that there are wrong Visits. Normally we observe something like the Visits are to long and are getting merged together. Back to times, it could also be a timezone issue. The idea is that reitti normalizes incoming data into UTC.
Having multiple users should not affect the calculation but I will double check the code.
You are right, they are not really helpful since there is just to much going on. They only provide me a hint where t look when developing.
You could go to Thank you very much for your writeup. We will sort this out :) |
|
*You could go to Settings > Export Data and choose the day. Then you can
export the GPX file for that day. I would also need a screenshot of the
VisitSensitivity. Since you mentioned GPSLogger, this also stores the GPX
file for the day on your phone. This could also be helpful if you could get
a hold on it. So we would have the raw data from GPSLogger and the
data reitti saw*
So, yes - it happened again. Took a short trip up 4th st in St Pete and
back. 2 visits. When i got back and all was processed, no trips showed
other than my home base. When I cleared processed flags, it then
reprocessed correctly.
I've attached data from gpslogger and from reitti for the day. Let me know
if there is anything else I can do to help.
…On Tue, Nov 25, 2025 at 12:45 AM Daniel Graf ***@***.***> wrote:
So, sorry to bother, I continue to get inconsistent and sometimes non
existent trip info. I chose a day (today) for my partner. The trip was
fairly simple, car ride up, stay at a family place for 3 hours and come
back then she took off for an errand
No worries, I like to help and together we hopefully find a solution which
makes reitti better in the end.
Below is what is/was displayed for her - essentially no information.
I then cleared the processed flag and waited.. update
public.raw_location_points set processed =false where "timestamp" >
'2025-11-24' and user_id = 3; Double checked to ensure column cleared and
waited ...
This time, processing took place and it was perfect. It logged our home,
family location, back to our home and then to another location. Below is a
display after allowing it to try processing again ...
The only major difference that I can think of was/is timing. Does the
system attempt to detect visits pretty much at the same time that raw
points are showing up? Could it be that the difference is allowing the data
to settle down before attempting detection - is there sufficient elapsed
time to process? I may be way off, and guessing that is all considered when
running jobs, what do i know. I do know that it seems clearing processing
and allowing the system to try again on older data gives a better result -
as this is not the first time I've seen that phenomena.
You could also be on something there. Reitti actually calculates every
time a point comes in with a little bit of settle time (15 seconds by
default) So 15 seconds after your point came in it starts calculating data
around (search duration from Visit Sensitivity) that point. So it fetches
all the unprocessed points, expands the time before and after with the
search duration and starts calculating like all the points in that
timeframe then where new. So the difference would be the time range then.
If you clear the whole day, lets say 2025-05-05T00:00 it would consider
2025--05-03T00:00 until 2025-05-07T23:59:59 for VisitDetection. It then
expands the search range further by any Visits which are touching the end
and start of that range, so we do not miss any points. If we considering an
incoming point, the time ranges are then shifted around the timestamp of
this point. So for example the point has a timestamp of 2025-05-05T14:47,
start and end time for searching visits would be 2025-05-03T14:47 and
2025-05-07T14:47.
So this is a difference how the data is handled. The rest of the pipeline
works the same. But this still does not explain why there is no data, only
if the search time for Visits would be really small. My first thought was
that there are wrong Visits. Normally we observe something like the Visits
are to long and are getting merged together.
Back to times, it could also be a timezone issue. The idea is that reitti
normalizes incoming data into UTC.
I am happy to try to zero in on this more and provide more detail - I'm
only reporting so that the developers might want to be aware of this. Of
course, it could be me - but I am not using preview, I have not done
anything but clear process and allow reprocess. I will note that there are
2 users, if that even makes a difference.
Having multiple users should not affect the calculation but I will double
check the code.
If there are detailed logs that show the processing agents activity that
would help ... if it needs to be turned on, I can work on trying to isolate
it. It is not clear to me which container/process logs this information. I
do see the main container logging some information, but I don't think it
looks detailed enough - or I'm not seeing it.
You are right, they are not really helpful since there is just to much
going on. They only provide me a hint where t look when developing.
If there is anything else you might be interested in me to supply, let me
know. You has asked "If you mind feel free to send me a gpx file or
something with a rough list of expected visits" but, i'm not sure how to
produce that and what it might look like.
You could go to Settings > Export Data and choose the day. Then you can
export the GPX file for that day. I would also need a screenshot of the
VisitSensitivity. Since you mentioned GPSLogger, this also stores the GPX
file for the day on your phone. This could also be helpful if you could get
a hold on it. So we would have the raw data from GPSLogger and the data
reitti saw
Thank you very much for your writeup. We will sort this out :)
—
Reply to this email directly, view it on GitHub
<#496 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABW3PZ725VL6U3HKIII5M6D36PUI3AVCNFSM6AAAAACM43F2NOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKMBXGAZDSOA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
I do not see the files but don´t worry. At the end of the day (it just started in Germany) i will have changed the whole ingestion pipeline to have a single threaded approach instead of the multiple layers of different event handling methods. This should dramatically simplify the pipeline and improve first the repeatability and speed. Further, I hope this will help in debugging that more easy. After I merged #489 I would like that you try out the develop version of reitti. I believe this will also fix your problem. If not we will investigate further. Thank your for your patience and will to help :) |
|
Hmm, they were attached. Nonetheless, I'll be happy to get the developer
version and test it out
…On Wed, Nov 26, 2025, 11:27 PM Daniel Graf ***@***.***> wrote:
I do not see the files but don´t worry. At the end of the day (it just
started in Germany) i will have changed the whole ingestion pipeline to
have a single threaded approach instead of the multiple layers of different
event handling methods.
This should dramatically simplify the pipeline and improve first the
repeatability and speed. Further, I hope this will help in debugging that
more easy.
After I merged #489 <#489>
I would like that you try out the develop version of reitti. I believe this
will also fix your problem. If not we will investigate further. Thank your
for your patience and will to help :)
—
Reply to this email directly, view it on GitHub
<#496 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABW3PZ4RL4DRAMNYWVECWUD36Z4RLAVCNFSM6AAAAACM43F2NOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKMBZGI3DENA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
HI @grillop, yesterday I pushed the develop version with the changes. I have found that after the update, i will recommend adding a new Visit Sensitivity Configuration starting at that day of the update. That worked the best for me. The day of the update, reitti will insert or ignore geo points to the incoming data to get to a desired points per minute and this simplifies the whole configuration there. I would like to only have the minimum duration of a stay in the end. But we are not yet there. 
Let me know if this worked :) |
|
brought up the new version, i didn't see a change. today, shows yesterday. I click on Nov 29th and the url shows the 28th, the map "text" shows the 29th, but it has a duration of 20 hours ... and visually the contents of the map represents what happened on the 28th. |
|
This seems like another problem from #506 |
|
I think it relates to an overnight trip, which is really me being home. The 30th shows the 29th Not sure now to manage a trip where you're home for a day and 1/2. |
|
Is there a trip shown or some long visit and there should be a trip from home to home overnight? Anyway, booth situations should not break anything. |
|
Yeah, just not making sense. Showing tried of 50 miles. Keeps showing trips
across several days even though I attempt to choose a single day. But then
ranges are confusing to me, ui wise
…On Sun, Nov 30, 2025, 1:14 PM Daniel Graf ***@***.***> wrote:
Is there a trip shown or some long visit and there should be a trip from
home to home overnight?
Anyway, booth situations should not break anything.
—
Reply to this email directly, view it on GitHub
<#496 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABW3PZYMPU5MS5X6EVTKKBT37MXZPAVCNFSM6AAAAACM43F2NOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKMJRG44TGOA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Ok, we will sort this out. First i will fix the time issue with the datepicker. I think this will at least help in pinning the issue further down. |
|
Hi @grillop, i fixed the date picker issues. Can you have a look at the latest develop version if it makes more sense now. |
|
No need to check, I finally saw that behaviour also on my machine. Will try to isolate the reason and come up with a fix |
Uh oh!
There was an error while loading. Please reload this page.
I have been using and tweaking my installation for a few weeks now - mostly trying to get a good consistent timeline. It has been a bit inconsistent - and along with tweaks I have had the system recalculate all data with increasing good results. I'm using geoapify. I am using gpslogger to on my pixel..
For a previous day I do not see any timeline. If i go into the settings, and with no changes, ask to preview the previous day, I get a good set of places, etc. It hasn't picked up in the past, hence my several attempts at running it through entirely again.
I have/am seeing the following in the log - many instances ...
Invalid trip time range detected for user xxxxxxxxxxxxx: 2025-11-17T20:23:28Z to 2025-11-17T19:56:29Z
Could this relate? Is this something anyone has seen? perhaps it is a bad configuration somewhere.
All reactions