Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

QZ FIT file start timestamp inconsistent with Zwift #1311

Open
victorypoint opened this issue Feb 24, 2023 · 69 comments · Fixed by #1312 · May be fixed by #1978
Open

QZ FIT file start timestamp inconsistent with Zwift #1311

victorypoint opened this issue Feb 24, 2023 · 69 comments · Fixed by #1312 · May be fixed by #1978
Assignees
@cagnulein
Labels
bug Something isn't working pinned

Comments

@victorypoint
Copy link
Contributor

victorypoint commented Feb 24, 2023 •
edited
Loading

@cagnulein, now that we have a consistent way to synchronize that start of activities between Zwift and QZ (ensure speed is 0 then increase to start FIT file recording at the same time), I'm finding a strange inconsistency with the start timestamp in the QZ FIT file. This is affecting the ability to accurately merge data between the two FIT files (e.g. latitude, longitude, elevation, power, etc).

For example, here's results of a recent test using QZ fake treadmill mode:

  • Launch QZ in paused state - 2:06pm
  • Launch Zwift and begin a manual run - 2:07pm - avatar stands still waiting for speed
  • Hit QZ Start button - 2:11pm - avatar stands still waiting for speed
  • Hit QZ preset speed button 10kph - 2:14pm - avatar begins to move
  • Hit QZ Stop button, end Zwift run, exit both QZ and Zwift - 2:22pm
  • Examine Zwift and QZ FIT files:
    • Zwift starting timestamp - 2:14:05pm - this appears to be when QZ speed preset button was pressed
    • QZ starting timestamp - 2:17:01pm - this is strange and shows QZ is ahead by 2 minutes 56 seconds
    • 0QZ starting timestamp - 2:17:01pm
    • 1QZ starting timestamp - 2:17:01pm

I've retested this several times and always arrive at a strange delayed QZ starting timestamp. Any help resolving this is greatly appreciated.

FIT files attached.
test4.zip
@victorypoint victorypoint added the bug Something isn't working label Feb 24, 2023
@victorypoint
Copy link
Contributor Author

A few more notes. Using https://www.fitfileviewer.com/, the Session and Event tables of QZ FIT file show a start timestamp of 2:14:03pm. That could possibly indicate there is missing data from 2:14:03 - 2:17:01pm in the QZ file.

@cagnulein
Copy link
Owner

seems a bug in the garmin fit library.
If you look in the fitfileviewer website, the session record has a good timestamp, that's because, instead of using the date object from the fit library, i'm calculating it myself.
also the lap timestamp is wrong, and it's using the fit object too...so i have to use my method only.

Question: if I create a windows version with the patch, can you help me doing a test and verify that it's all good?

@cagnulein
Copy link
Owner

@victorypoint when it's ready you will see a new windows version here https://github.com/cagnulein/qdomyos-zwift/actions/runs/4267692357 (bottom of the page) it will be ready in 30 minutes

@victorypoint
Copy link
Contributor Author

@cagnulein, thank-you! Very excited to test this out. That's interesting about the timestamp bug in the Garmin FIT SDK. I'm curious if you're using the latest 21.105 SDK, Feb. 2, 2023? - https://developer.garmin.com/fit/download/

@cagnulein
Copy link
Owner

cagnulein commented Feb 25, 2023 via email

@cagnulein
Copy link
Owner

@victorypoint ready https://github.com/cagnulein/qdomyos-zwift/suites/11196536723/artifacts/572412013

@victorypoint
Copy link
Contributor Author

@victorypoint ready https://github.com/cagnulein/qdomyos-zwift/suites/11196536723/artifacts/572412013

Thank-you. I started doing some testing. Go to bed! 😴🛌

@victorypoint
Copy link
Contributor Author

@cagnulein, using new Windows QZ build v2.12.73, I repeated my test as before using QZ fake treadmill mode::

  • Launch QZ in paused state - 10:27pm
  • Launch Zwift and begin a manual run - 10:29pm - avatar stands still waiting for speed
  • Hit QZ Start button - 10:30pm - avatar stands still waiting for speed
  • Hit QZ preset speed button 10kph - 10:33pm - avatar begins to move
  • Hit QZ Stop button, end Zwift run, exit both QZ and Zwift - 10:40pm
  • Examine Zwift and QZ FIT files:
    • Zwift starting timestamp - 10:33:06pm - this appears to be when QZ speed preset button was pressed
    • QZ starting timestamp - 10:35:58pm - still a problem here - this shows QZ is ahead by 2 minutes 52 seconds

Here are detailed timestamps per FIT table:

  • Zwift timestamps:

    • File_id - time created - 10:29:21pm, 1 record
    • Event - start timestamp - 10:33:05pm, stop timestamp - 10:40:17pm, 2 records
    • Lap - timestamp - 10:40:16pm, start_time - 10:33:05pm, 1 record
    • Session - timestamp - 10:40:17pm, start_time - 10:33:05pm, 1 record
    • Activity - timestamp - 10:40:17pm, 1 record
    • Record - first timestamp - 10:33:06pm, 432 records - this appears to be when QZ speed preset button was pressed

  • QZ timestamps:

    • File_id - time created - 10:33:04pm, 1 record
    • Event - timestamp - 10:33:04pm, 1 record, no stop timestamp
    • Lap - timestamp - 10:33:04pm, start_time - 10:33:04pm, 1 record
    • Session - timestamp - 10:33:04pm, start_time - 10:33:04pm, 1 record
    • Activity - timestamp - 10:33:04pm, 1 record
    • Record - first timestamp - 10:35:58pm, 419 records

FIT files attached.
test1.zip

@cagnulein cagnulein linked a pull request Feb 25, 2023 that will close this issue
QZ FIT file start timestamp inconsistent with Zwift (Issue #1311) #1312
Merged
@cagnulein
Copy link
Owner

sorry @victorypoint I found now the original issue, it's not fault of the fit library, it's just my fault :(
New build here in the bottom https://github.com/cagnulein/qdomyos-zwift/actions/runs/4268708107

@cagnulein
Copy link
Owner

@victorypoint in case you missed, new artifact is ready

@victorypoint
Copy link
Contributor Author

victorypoint commented Feb 25, 2023 •
edited
Loading

@cagnulein, this update is great! I retested as before. Here's the results:

  • Launch QZ in paused state - 2:38pm
  • Launch Zwift and begin a manual run - 2:42pm - avatar stands still waiting for speed
  • Hit QZ Start button - 2:43pm - avatar stands still waiting for speed
  • Hit QZ preset speed button 10kph - 2:45pm - avatar begins to move
  • Hit QZ Stop button, end Zwift run, exit both QZ and Zwift - 2:55pm
  • Examine Zwift and QZ FIT files:
    • Zwift starting timestamp - 2:45:07pm - this appears to be very close when QZ speed preset button was pressed
    • QZ starting timestamp - 2:45:04pm - this also appears to be very close QZ speed preset button was pressed

Looking at the FIT tables using https://www.fitfileviewer.com/, I noticed both FIT files have consistent data structure, however the QZ Event table has no stop timestamp record whereas the Zwift table does. Is this important?

I will continue to do more testing.

@cagnulein
Copy link
Owner

cagnulein commented Feb 26, 2023 via email

@victorypoint
Copy link
Contributor Author

@cagnulein, my next test discovered a problem with how QZ calculates elevation gain. I'll post the issue here but let me know if you want a new ticket for this.

For this test, I wanted to compare how QZ calculated elevation compares with the terrain elevation for a given course. To get a terrain elevation graph, I used bike mode and mapped the entire 23K Volcano Climb course in a FIT file. Next, I switched to run mode (using the bike pairing meetup trick) and repeated the course using auto-inclination, then merged the QZ elevation into the resulting Zwift FIT file. Here's the 2 graphs overlayed. You can see the Zwift bike elevation in blue and QZ run elevation in green. From what I can tell, the ascent elevations are very close, however QZ appears to be shallowing the decent.

volcano climb - graph comparison

Note that I used fake treadmill for this test with 'Zwift inclination gain' = 2. For comparison, here is an elevation graph of Zwift bike elevation overlayed with QZ bike calculated elevation.

GPXSee 2023-02-27 10_31_36 AM

So, it appears that QZ is treating decent (negative elevation) with a different calculation than ascent (positive elevation).

@cagnulein
Copy link
Owner

yes because zwift send the negative inclination to half :) you have to enable the zwift negative incliantion double setting. There are some youtube videos about this zwift issue too :)

@victorypoint
Copy link
Contributor Author

victorypoint commented Feb 27, 2023 •
edited
Loading

@cagnulein, I repeated this test using the longer Mega Pretzel course just to confirm it wasn't isolated to a single course. Again I'm finding that QZ is treating decent (negative elevation) with a different calculation than ascent (positive elevation). Here's the 2 graphs overlayed - Zwift bike elevation in blue and QZ run elevation in green. Again, I'm using fake treadmill with 'Zwift inclination gain' = 2.

uber pretzel - graph comparison

Also, here is the elevation graph of Zwift bike elevation overlayed with QZ bike calculated elevation.

GPXSee 2023-02-26 2_27_00 PM

@cagnulein
Copy link
Owner

check my above comment @victorypoint

@victorypoint
Copy link
Contributor Author

yes because zwift send the negative inclination to half :) you have to enable the zwift negative incliantion double setting. There are some youtube videos about this zwift issue too :)

Thanks! .. desperately poking around in QZ looking for this setting now...

@victorypoint
Copy link
Contributor Author

yes because zwift send the negative inclination to half :) you have to enable the zwift negative incliantion double setting. There are some youtube videos about this zwift issue too :)

Sorry, I can't find this setting. Why exactly is Zwift sending negative inclination to half and positive inclination to full?

@cagnulein
Copy link
Owner

bike options, double negative inclination. zwift does this because, IMHO, the bikes with negative inclination will be uncontrollable (because you will have super high cadence during downhill)

@victorypoint
Copy link
Contributor Author

@cagnulein, found it under Bike options - 'Double Negative Inclination'. So it's a Zwift bug?

@cagnulein
Copy link
Owner

yes confirmed :)

@victorypoint
Copy link
Contributor Author

yes confirmed :)

Ok thanks. Will retest.

@victorypoint
Copy link
Contributor Author

victorypoint commented Feb 28, 2023 •
edited
Loading

@cagnulein, so with QZ 'Double Negative Inclination' turned on this time, unfortunately I'm getting the same elevation graphs as before. I'm thinking maybe there's another setting that needs to be set in QZ. I'm using fake treadmill with 'Zwift inclination gain' = 2 as before with Wahoo Treadmill device.

GPXSee 2023-02-27 10_33_45 PM

Logs and FIT files attached.
bike vs run mode.zip

@cagnulein
Copy link
Owner

cagnulein commented Feb 28, 2023 via email

@cagnulein
Copy link
Owner

@victorypoint i'm an idiot, i'm applying the zwift negative inclination only to the bikes. That's why you're seeing this.
Ok so now I'm doing 2 modifications

  1. using this settings for all the devices
  2. moving the setting to the advanced settings

cagnulein added a commit that referenced this issue Feb 28, 2023
@cagnulein
QZ FIT file start timestamp inconsistent with Zwift #1311
55ad2db
cagnulein added a commit that referenced this issue Feb 28, 2023
@cagnulein
Merge branch 'master' into QZ-FIT-file-start-timestamp-inconsistent-w…
2847abd 
…ith-Zwift-#1311
@victorypoint
Copy link
Contributor Author

the only thing is that how handle the pause event :)

Is this what you mean? We will likely need Settings-Treadmill Options-"Pause when App Starts" turned on. First Zwift activity, we press Start button start then Stop button when done. Then reset QZ for next activity by pressing Back and then Pause. Or we leave "Pause when App Starts" turned off. Then we press Pause then Start at beginning of activity.

btw can i merge the main modification of this ticket ?
i guess it's working and also it's safe

Yes, it's working well!

cagnulein added a commit that referenced this issue Mar 3, 2023
@cagnulein
QZ FIT file start timestamp inconsistent with Zwift (Issue #1311) (#1312
9e8a8a1 
)
@cagnulein
Copy link
Owner

Yes, it's working well!

merged!

We will likely need Settings-Treadmill Options-"Pause when App Starts" turned on. First Zwift activity, we press Start button start then Stop button when done. Then reset QZ for next activity by pressing Back and then Pause. Or we leave "Pause when App Starts" turned off. Then we press Pause then Start at beginning of activity.

yeah I mean I was looking to find a way to achieve this without adding any new settings. If I'm right, QZ elapsed timer is not counting if the speed is == 0, if this is true the start trigger is ok for the fit file. For the stop instead it's tricky because there is no automatic way to stop it. But I was thinking: is it really necesseary? I mean since the Zwift activity has an end, we can simple discard all the remaining qz activity from the zwift one (the qz activity will be probably longer than the zwift one, but we can simply cut the remaining off, isn't it?)

@victorypoint
Copy link
Contributor Author

@cagnulein, let me test this idea - single QZ FIT file for multiple Zwift activities and see if FIT merge works.

@victorypoint
Copy link
Contributor Author

victorypoint commented Mar 4, 2023 •
edited
Loading

But I was thinking: is it really necesseary? I mean since the Zwift activity has an end, we can simple discard all the remaining qz activity from the zwift one (the qz activity will be probably longer than the zwift one, but we can simply cut the remaining off, isn't it?)

@cagnulein, ok I tested this theory and you're a genius! Have I ever mentioned that before? 🤣🤣

Here's what I did to test multiple Zwift activities/single QZ session. A typical scenario is to start on May Field, jump to a run event, then exit to main Zwift screen and perhaps do other runs, workouts, meetups while staying in Zwift with QZ running in the background.

  • Use QZ fake treadmill mode
  • Launch QZ and Zwift on PC
  • Start in bike mode for auto-incline and bike for a short distance to test
  • Create and join run meetup. Run for a km or more in run meetup
  • Exit and go to May Field for a km or more
  • Jump to run meetup again and run for a km or more
  • Exit and go to London Classique and run for a bit
  • Exit and join Watopia Chili Pepper pacer bot and run for a bit
  • Jump to run meetup again and run for a bit
  • End and exit Zwift and QZ
  • There may have been a few more activities in there - I can't remember 😂
  • Collect single QZ FIT file and all Zwift activity FIT files and run merge process - it worked!! The only downside as we noted earlier is that QZ metrics transfer over to each workout including elevation gain, elapsed time, etc.

Here are elevation graphs of resulting FIT file merges for each Zwift activity. Blue is Zwift FIT, green is Zwift/QZ/elevation merge FIT. Note that the first graph is the bike ride where Zwift FIT file always has correct terrain elevation. The remaining graphs are runs where Zwift elevation is incorrect (in blue) and need to be corrected with QZ elevation (green):

2023-03-01-19-20-13
2023-03-01-19-32-01
2023-03-01-19-40-23
2023-03-01-19-43-13
2023-03-01-19-43-33
2023-03-01-19-48-49
2023-03-01-19-57-53
2023-03-01-20-04-07
2023-03-01-20-04-28

@cagnulein
Copy link
Owner

so your java skills are still strong! 💪

@cagnulein
Copy link
Owner

so I guess we can reopen this
#1369 (comment)

@cagnulein cagnulein reopened this Mar 20, 2023
@cagnulein
Copy link
Owner

my idea is adding a command line argument to specify the zwift fit file directories. So qz can get the lastest one and opening it each seconds or so? what do you think?

@victorypoint
Copy link
Contributor Author

my idea is adding a command line argument to specify the zwift fit file directories. So qz can get the lastest one and opening it each seconds or so? what do you think?

Incredible idea! This just after I came up with a Windows batch solution using the Java option 😂 - https://github.com/victorypoint/RepairZwiftFIT

@stale
Copy link

stale bot commented Apr 4, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix This will not be worked on label Apr 4, 2023
@victorypoint
Copy link
Contributor Author

Keep this alive please!

@stale stale bot removed the wontfix This will not be worked on label Apr 4, 2023
@cagnulein
Copy link
Owner

about this @victorypoint should we proceed with opening the fit file in real time? do you think it's doable?

@victorypoint
Copy link
Contributor Author

about this @victorypoint should we proceed with opening the fit file in real time? do you think it's doable?

@cagnulein, yes, I believe it's doable. The active FIT file is readable in realtime which is what we want to obtain coordinates for each timestamp.

@cagnulein
Copy link
Owner

@victorypoint can you attach a fit file from a workout here? I mean getting it while you're running, so i can check that the file is closed. Also how often this file is updated from zwift?

@stale
Copy link

stale bot commented Apr 20, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix This will not be worked on label Apr 20, 2023
@stale stale bot removed the wontfix This will not be worked on label Apr 20, 2023
@victorypoint
Copy link
Contributor Author

@victorypoint can you attach a fit file from a workout here? I mean getting it while you're running, so i can check that the file is closed. Also how often this file is updated from zwift?

@cagnulein, so sorry for the delay on this! I've been so busy with the python stuff! 😮. So I just did a workout as follows:

  • Zwift Windows with QZ fake treadmill. FIT file path is - C:\Users\alanu\Documents\Zwift\Activities\
  • I grabbed all in progress FIT files for analysis at 3 minute intervals - inProgressActivity.fit and 2023-04-27-11-54-03.fit
  • I used FIT file viewer to inspect files - https://www.fitfileviewer.com/
  • Grabbed inProgressActivity.fit snapshots every 3 minutes
    • inProgressActivity0.fit, 1, 2, 3 - only one record - 8/25/1992, 2:08:16 PM
    • inProgressActivity4.fit - 600 records with timestamp every second - 4/27/2023, 11:56:17 AM - 4/27/2023, 12:06:16 PM
  • Grabbed 2023-04-27-11-54-03.fit snapshots every 3 minutes as well
    • 2023-04-27-11-54-03-1.fit - 109 records every second - 4/27/2023, 11:56:17 AM - 4/27/2023, 11:58:05 AM
    • 2023-04-27-11-54-03-2.fit - 331 records every second - 4/27/2023, 11:56:17 AM- 4/27/2023, 12:01:47 PM
    • 2023-04-27-11-54-03-3.fit - 441 records every second - 4/27/2023, 11:56:17 AM - 4/27/2023, 12:03:37 PM
    • 2023-04-27-11-54-03-4.fit - 710 records every second - 4/27/2023, 11:56:17 AM - 4/27/2023, 12:08:05 PM

So it looks like my previous theory that the inProgressActivity.fit file updates in real-time is wrong - it appears to update every 10 minutes or so? However, the timestamped FIT file does update in real-time I believe every second and is readable during workouts.

@cagnulein
Copy link
Owner

@victorypoint are you still interested on this? now with the zwift api we can collect gps data!

@cagnulein
Copy link
Owner

the only issue is that for now the windows build doesn't support the zwift api (i have to find the time to add them), but if you want I can add this on the android or ios build

@victorypoint
Copy link
Contributor Author

@cagnulein, yes! Absolutely! I can test on all platforms so good to go 👍

@cagnulein
Copy link
Owner

ok i will try to do first on ios, i need to convert the X and Y from zwift to gps coordinates, i found that there is a conversion to do for the different worlds

@cagnulein cagnulein linked a pull request Jan 9, 2024 that will close this issue
Zwift Coordinates from API #1978
Open
@cagnulein
Copy link
Owner

@victorypoint i thought it could be easy instead i'm struggling to find information about how to translate coordinates from zwift system to the world system. https://github.com/sandermvanvliet/RoadCaptain/blob/d8ec891349212d2a8ef2691925376712680e0bc4/src/RoadCaptain/TrackPoint.cs#L256 seems to be the most update resource but his conversion doesn't match at all. Also I don't understand if this conversion is actually working on RoadCaptain. Do you have some information about this?

I have X and Y zwift coordinates in realtime but I can't transform them to real GPS coordinates

@victorypoint
Copy link
Contributor Author

@cagnulein, ok I'll check it out in the next few days. I haven't used roadcaptain yet so need to study it a bit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cagnulein cagnulein

Labels
bug Something isn't working pinned
Projects
None yet
Milestone
No milestone
2 participants
@cagnulein @victorypoint