-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Huge offset just before entering HOLDOVER state #20
Comments
@vvfedorenko Could you rerun the test and send me logs with debug set to 1 in oscillatord.conf ? I will do tests on my side |
@vvfedorenko I think I know the problem comes: When the antenna is unplugged, the GNSS receiver takes some time to discard its value and to set its fix to false. during this period the time pulse is not to be trusted and may vary a lot because it relies on its internal OCXO. Fortunately the GNSS receiver sends us data about the antenna status. Here is an example with antenna status accepting OPEN as a valid state: It takes 20 seconds to enter holdover:
You can see the disconnection of the antenna when the antenna status goes from 2 to 4 Here is another example where the only valid state for antenna is ok: It takes only 2 seconds
|
@vvfedorenko Could you please try with branch fix-holdover-enter-condition ? |
@CharlesParent We use Huber-Suhner splitter to provide GPS signal to servers in rack, and we have normal operation status as following:
That's why we cannot use this fix. And the problem actually is somehow different. If you take closer look at logs I provided you can find that there were 2 consequent phase jumps in different directions (negative and then positive):
These jump look like some problem of providing PPS signal in ptp device that is monitored by phasemeter. |
@vvfedorenko could you rerun the test with debug set to 1 in oscillatord.conf and send me the logs ? |
@CharlesParent I started test with debug information, but the actual problem is that ART card lost GNSS when other servers showed good amount of satellites, there was no physical disconnection of antenna. This situation happens with all 4 ART card we have in prod right now occasionally and I will wait for another occurrence of it with debug log on. |
@CharlesParent I was able to reproduce this problem in case of low-quality GNSS signal with The things that I would pay attention to are:
Here we can see that with 2 PPS signals and 0 phase_error, this data point is treated as not valid, because GNSS fix state is false. At the same time, GNSS library thinks that GNSS point is valid. And the next cut:
Here we can see that phase_error is really high, but parsed GNSS data has Fix status True, even though GNSS library says this point is not valid. If this point is the latest in 6 seconds loop, OD_PROCESS will treat it as absolutely valid signal and will produce phase shift to absolutely crazy 33672640ns ahead. |
I got even better log
First we got pahse_error: 35666365ns with Fix: true, but value is not parsed as valid by GNSS. At the same time was used to correct phase. Then couple of changes to other direction and the actual time goes bananas in the end. |
Hi @vvfedorenko ! Thanks for the logs, do not hesitate to post more logs in you have ! I will investigate why Fix Ok is returning false but valid still equals one. Where do these logs come from ? Are you on your home setup on in a close-to-production setup ? Because you only have 4/5 satellites, this is not ideal |
Hi @CharlesParent! Yes, it's my home setup and I specially made such conditions to have low-quality GNSS signal to reproduce the issue. According to our production setup monitoring we had the same conditions (small amount of visible satellites) during issues before |
# This is the 1st commit message: WIP # The commit message #2 will be skipped: # WIP # The commit message #3 will be skipped: # WIP # The commit message #4 will be skipped: # WIP # The commit message #5 will be skipped: # WIP # The commit message #6 will be skipped: # WIP # The commit message #7 will be skipped: # WIP # The commit message #8 will be skipped: # WIP # The commit message #9 will be skipped: # WIP # The commit message #10 will be skipped: # WIP # The commit message #11 will be skipped: # WIP # The commit message #12 will be skipped: # WIP # The commit message #13 will be skipped: # WIP # The commit message #14 will be skipped: # WIP # The commit message #15 will be skipped: # WIP # The commit message #16 will be skipped: # WIP # The commit message #17 will be skipped: # WIP # The commit message #18 will be skipped: # WIP # The commit message #19 will be skipped: # WIP # The commit message #20 will be skipped: # WIP # The commit message #21 will be skipped: # WIP # The commit message #22 will be skipped: # WIP # The commit message #23 will be skipped: # WIP # The commit message #24 will be skipped: # WIP # The commit message #24 will be skipped: # WIP # The commit message #25 will be skipped: # WIP
We constantly observe a problem with oscillatord - it show huge offset just before entering Holdover, which creates phase offset and provides wrong timestamp on ptp device. You can find this behaviour in logs, and the diff is visible in ts2phc logs
oscillatord.log.txt
ts2phc.log.txt
The text was updated successfully, but these errors were encountered: