Near-duplicate epochs in some mission data

[ludwigus]

Hi,
the altimeter data in RADS is supposed to be at a rate of 1Hz. However, for CryoSat2, Geosat, Jason1-3 and Saral there are some observations which differ by far less than a second and are likely duplicates or near-duplicates. As an example here are some points of c2p0377c004.nc:

#          tstr               lon           lat       alt_gdre   range_ku 
2010-07-16T00:17:11.352256   19.6174756  -73.8137340 740403.663 737129.734 
2010-07-16T00:17:12.295694   19.5878533  -73.7574717 740389.087 737143.169 
2010-07-16T00:17:13.239130   19.5584034  -73.7012060 740374.478      nan 
2010-07-16T00:17:14.182567   19.5291240  -73.6449367 740359.837      nan 
2010-07-16T00:17:15.126005   19.5000135  -73.5886638 740345.164 737149.569 
2010-07-16T00:17:16.069441   19.4710701  -73.5323877 740330.459      nan 
2010-07-16T00:17:16.069442   19.4710701  -73.5323877 740330.459      nan 
2010-07-16T00:17:17.012878   19.4422922  -73.4761081 740315.721 737192.444 
2010-07-16T00:17:17.956315   19.4136780  -73.4198252 740300.951      nan 
2010-07-16T00:17:18.899753   19.3852258  -73.3635388 740286.149 737185.700 

The epoch 2010-07-16T00:17:16.069442 differs only by 1e-6 seconds from the previous point and the remaining parameters are exactly the same. For the other missions (Geosat, the Jasons and Saral) they don't seem to be exact duplicates as also the position differs by some hundreds of meters. I have checked all the missions for epochs with dt<0.1s and attached a list with the result. This list contains a row with a summary and an example for each year of each mission. The columns are "mission", "year", "number of near-duplicates" and "time stamps of first near duplicate of this year". For all the other missions I didn't find any epochs with dt<0.1s.

It seems that there is either a bug in the original data or in the scripts that translates these file to the RADS format.
near_duplicate.txt

[leuliett]

Ludwig,
Thanks for the question. Most of the near-duplicates are explained by leap seconds in UTC in the original data. In those cases, independent measurements were made, but the time-stamps are very close. There can be errors in the data either because of an error in the timing used for determining range or because errors when the orbits are generated. This can affect the data for several seconds to minutes before and after the leap second.

[ludwigus]

Hi Eric,
thanks for the fast response. I can understand that this is the first guess according to what I wrote but I don't think that explains this bug. I have reorganized the data in yearly database tables. Thats why I report the results per year and the first example I have shown is usually at the beginning of a year. However, the near duplicates occur all over the year, not only around January 1st or June 30th. Clearly, the first thing I have checked is if the near duplicates occur in the RADS NetCDFs as well and they do.
In c2p0377c004.nc I found 11 near duplicates:

2010-07-16T00:17:16.069441 2010-07-16T00:17:16.069442
2010-07-16T00:18:36.261586 2010-07-16T00:18:36.261587
2010-07-16T00:21:46.835860 2010-07-16T00:21:46.835861
2010-07-16T00:29:15.911872 2010-07-16T00:29:15.911873
2010-07-16T00:34:56.492629 2010-07-16T00:34:56.492630
2010-07-16T00:47:35.959414 2010-07-16T00:47:35.959415
2010-07-16T00:51:12.949924 2010-07-16T00:51:12.949925
2010-07-16T00:51:56.348026 2010-07-16T00:51:56.348027
2010-07-16T00:53:08.049239 2010-07-16T00:53:08.049240
2010-07-16T00:57:52.023776 2010-07-16T00:57:52.023777
2010-07-16T00:58:24.100634 2010-07-16T00:58:24.100635

However, in 2010 there was no leap second. Furthermore, these are no consecutive epochs. There are randomly scattered along the orbit. I fear the problem is somewhere else.

[remkos]

I finally took some time to look at this, and traced it down to the original CryoSat (GOP version C) products from ESA.
This is hopefully solved in the next release of there products, so that it also gets solved in RADS.

So I suggest we just live with the limitation and close this issue.