Weird behavior between Python 3 and Python 2.7 for timestamps

[Djailla]

Hello,

When parsing a GPX where precision is on milliseconds, using Python 2 and gpxpy, no issue to get the correct timestamp value per point. But if I use Python, I end up with a 1s precision and then got duplicate timestamps when recording is less than every seconds and fail to calculate speed compared to previous point.

Are you aware of any issue with Python version and timestamp ?

Regards

In attachement the GPX I use to test.
Move_2017_02_07_12_21_36_Course.gpx.zip

[Djailla]

Up ?

[tkrajina]

Yes, I'm here but unfortunately I don't have the time to investigate. In this specific case I'm not 100% sure what you mean "got duplicate timestamps".

It would help a lot if you could provide me a code which fails (along with the gpx file). Ideally, it would be a test in tests.py but a standalone script is also OK.

[Djailla]

I'll do that

[philshem]

I'm not sure if this is the same issue, because I've not tested in Python 3, but I am seeing milliseconds truncated to seconds.

The reason may be because my GPX file starts with no (null) milliseconds value. Here are the first two timestamps of my file. The first one has only seconds, the second has 200 milliseconds.

• 2017-08-10T15:59:28Z
• 2017-08-10T15:59:28.200Z

Sample valid GPX file

My guess at the cause is the I think the issue is due to the DATE_FORMATS lookup, which may be only checked once per GPX file.

In gpxpy/gpxpy/gpxfield.py, the function parse_time() is called in the TimeConverter() class and contains a call to mod_gpx.DATE_FORMATS, which loops over potential datetime formats. I suspect that the first matching format is returned, which in my case is a timestamp without milliseconds.

From gpx.py

# GPX date format(s) used for parsing. The T between date and time and Z after
# time are allowed, too:
DATE_FORMATS = [
    '%Y-%m-%d %H:%M:%S',
    '%Y-%m-%d %H:%M:%S.%f',
    #'%Y-%m-%d %H:%M:%S%z',
    #'%Y-%m-%d %H:%M:%S.%f%z',
]

From gpxfield.py (parse_field function)

    for date_format in mod_gpx.DATE_FORMATS:
            try:
                return mod_datetime.datetime.strptime(string, date_format)
            except ValueError:
                pass

---

Here's a sample code, using the pastebin GPX file linked above

import gpxpy
import gpxpy.gpx
with open('simple.gpx','rb') as gpx_file:
	g = gpxpy.parse(gpx_file)
	for track in g.tracks:
		for segment in track.segments:
			for point in segment.points:
				print type(point.time), point.time

prints the second timestamp without milliseconds

<type 'datetime.datetime'> 2017-08-10 15:59:28
<type 'datetime.datetime'> 2017-08-10 15:59:28

I'm using Python 2.7.12 (64 bit) and gpxpy==1.1.2

[tkrajina]

Yes, you're right @philshem

The problem is that we can never be sure which format are the timestamps. But, again, if we implement the fix from #92 then we can:

• reverse the order of parsers ('%Y-%m-%d %H:%M:%S.%f first). This will solve this issue.
• change the existing DATE_FORMATS to be functions. Those function can do some basic sanity checks before parsing the timestamp the usual way. For example, if input string has no dots . then it makes no sense to continue parsing with '%Y-%m-%d %H:%M:%S.%f. That way we can avoid multiple timestamp parsers in most cases.

[tkrajina]

One more thing, as I said, I'm pretty busy for any complex changes in gpxpy at the moment. But I think this one should be less than an hour of work (including tests). I think I could find some time this or next weekend. (of course, I'm accepting pull requests if somebody needs this sooner).

[philshem]

Reversing the order is something I can do, but I guess you are referring to the DATE_FORMAT ;)

But as for the function for DATE_FORMAT, I'm more than happy to wait. At the moment I use a local dev version of gpxpy, but it's really custom and not good for a patch. Many thanks!