Support for GPX1.1 #20

Closed
fvezzoli opened this Issue Aug 19, 2013 · 15 comments

Comments

Projects
None yet
7 participants
@fvezzoli

I read all GPX standard and i found the difference between version.
The first difference is the moving of metadata in a "metadata" child. The difference in the metadata child is:

  • the author/mail in the 1.0 become the author in 1.1 with personType
  • the mail is divided in id and domain attribute
  • the url/urlname in 1.0 become linktype in 1.1 and add the possibility to add more link
  • added the copyrigth
    In the other part of GPX standard the only difference is the url → link modify.
    I try to adjust the parsing code to add the possibility for parsing both version.
    For creation gpx code start at end of parsing code modification.
@tkrajina

This comment has been minimized.

Show comment
Hide comment
@tkrajina

tkrajina Mar 10, 2014

Owner

Regarding the GPX 1.0 and 1.1. I finally found some time to create a script to pretty-print the structures of 1.0 and 1.1.

The results are here:

https://github.com/tkrajina/gpxpy/blob/gpx-1.1/xsd/gpx1.0.txt
https://github.com/tkrajina/gpxpy/blob/gpx-1.1/xsd/gpx1.1.txt

The problem is that I copied the object model from the sample GPX files I had at the time. Now with 1.1 the object model is different, and that will break existing code.

Possible solutions:

  1. Keep the existing object-model but transform it to/from GPX1.1 when needed.
  2. Let the user decide if to use GPX1.0 or GPX1.1 object model in the library
  3. Keep both, depending on the GPX version, but that way users would need to have two different versions of the code to handle 1.0 and 1.1.

Any suggestions between 1. and 2. ?

Owner

tkrajina commented Mar 10, 2014

Regarding the GPX 1.0 and 1.1. I finally found some time to create a script to pretty-print the structures of 1.0 and 1.1.

The results are here:

https://github.com/tkrajina/gpxpy/blob/gpx-1.1/xsd/gpx1.0.txt
https://github.com/tkrajina/gpxpy/blob/gpx-1.1/xsd/gpx1.1.txt

The problem is that I copied the object model from the sample GPX files I had at the time. Now with 1.1 the object model is different, and that will break existing code.

Possible solutions:

  1. Keep the existing object-model but transform it to/from GPX1.1 when needed.
  2. Let the user decide if to use GPX1.0 or GPX1.1 object model in the library
  3. Keep both, depending on the GPX version, but that way users would need to have two different versions of the code to handle 1.0 and 1.1.

Any suggestions between 1. and 2. ?

@b-jazz

This comment has been minimized.

Show comment
Hide comment
@b-jazz

b-jazz Mar 15, 2014

Can you do some sort of superset object model that could handle either format? Reading in a gpx1.0 format file would set a flag for 1.0, gpx1.1 would set it to 1.1. Output would be whatever was already set, or forced by the caller, or default to the more descriptive of the two (1.1, I'm guessing).
If you read in a 1.0 format file and then start setting attributes that are found in 1.1, you'd automatically promote the flag to 1.1. If the caller wants to output 1.0 after using 1.1 features, you'd make a best-effort attempt to output everything that can be represented in 1.0.

If you have to break the API to make improvements, I think it would be worth it. More abstraction and features for those that want to move forward. Anyone that doesn't want to change code can be happy with an older release that keeps on working as is.

b-jazz commented Mar 15, 2014

Can you do some sort of superset object model that could handle either format? Reading in a gpx1.0 format file would set a flag for 1.0, gpx1.1 would set it to 1.1. Output would be whatever was already set, or forced by the caller, or default to the more descriptive of the two (1.1, I'm guessing).
If you read in a 1.0 format file and then start setting attributes that are found in 1.1, you'd automatically promote the flag to 1.1. If the caller wants to output 1.0 after using 1.1 features, you'd make a best-effort attempt to output everything that can be represented in 1.0.

If you have to break the API to make improvements, I think it would be worth it. More abstraction and features for those that want to move forward. Anyone that doesn't want to change code can be happy with an older release that keeps on working as is.

@tkrajina

This comment has been minimized.

Show comment
Hide comment
@tkrajina

tkrajina Mar 18, 2014

Owner

I think I'll stick with the GPX1.0 object model for the library, but transparently parse/serialize to 1.0 or 1.1 depending on the specific track. There will probably be some minor API changes.

The 1.1. is more descriptive but I don't think it was worth the change (for example the metadata field in the gpx instead of all those field directly under ...).

Anyway, I'm now rewriting the parser/serializer parts in order to be easier to handle 1.1.

Owner

tkrajina commented Mar 18, 2014

I think I'll stick with the GPX1.0 object model for the library, but transparently parse/serialize to 1.0 or 1.1 depending on the specific track. There will probably be some minor API changes.

The 1.1. is more descriptive but I don't think it was worth the change (for example the metadata field in the gpx instead of all those field directly under ...).

Anyway, I'm now rewriting the parser/serializer parts in order to be easier to handle 1.1.

@tkrajina

This comment has been minimized.

Show comment
Hide comment
@tkrajina

tkrajina Apr 13, 2014

Owner

My first 1.1 implementation is here: https://github.com/tkrajina/gpxpy/tree/gpx-1.1

Owner

tkrajina commented Apr 13, 2014

My first 1.1 implementation is here: https://github.com/tkrajina/gpxpy/tree/gpx-1.1

@OldMac

This comment has been minimized.

Show comment
Hide comment
@OldMac

OldMac Aug 6, 2014

Hi.

I've tried using gpx-1.1 but not sure how to extract the heart rate data from a file.
In GPX file the HR extension looks like......

<extensions>
 <gpxtpx:TrackPointExtension>
  <gpxtpx:hr>146</gpxtpx:hr>
 </gpxtpx:TrackPointExtension>
</extensions>

When I run

for track in gpx.tracks:
for segment in track.segments:
for point in segment.points:
print(point.time, point.latitude, point.longitude, point.elevation, point.extensions)

The result is...
2014-07-16 20:40:13 50.997989 -1.381288 51.0 {'gpxtpx:TrackPointExtension': '\n '}

Any thoughts? I've just started using python after a long break and I'm not a programmer so any help appreciated.

Thanks

OldMac commented Aug 6, 2014

Hi.

I've tried using gpx-1.1 but not sure how to extract the heart rate data from a file.
In GPX file the HR extension looks like......

<extensions>
 <gpxtpx:TrackPointExtension>
  <gpxtpx:hr>146</gpxtpx:hr>
 </gpxtpx:TrackPointExtension>
</extensions>

When I run

for track in gpx.tracks:
for segment in track.segments:
for point in segment.points:
print(point.time, point.latitude, point.longitude, point.elevation, point.extensions)

The result is...
2014-07-16 20:40:13 50.997989 -1.381288 51.0 {'gpxtpx:TrackPointExtension': '\n '}

Any thoughts? I've just started using python after a long break and I'm not a programmer so any help appreciated.

Thanks

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Aug 11, 2014

My Garmin GPX generates extensions like this:

<trkpt lat="1.234" lon="-1.234">
  <ele>14.200000</ele>
  <time>2014-08-08T12:20:38Z</time>
  <extensions>
    <gpxx:Temperature>32.000000</gpxx:Temperature>
  </extensions>
</trkpt>

Which are parsed correctly and result in the following usage:

In [31]: print p
[trkpt:1.234,-1.234@14.6@2014-08-08 12:20:38]

In [32]: print p.extensions
{u'gpxx:Temperature': u'32.000000'}

In [33]: print p.extensions['gpxx:Temperature']
32.000000

I'm guessing there is a bug in the new parser and how it deals with an arbitrarily long chunk of custom xml. It might be assuming that it would only find a single line. You should file it as a bug, and/or submit a code patch pull request.

ghost commented Aug 11, 2014

My Garmin GPX generates extensions like this:

<trkpt lat="1.234" lon="-1.234">
  <ele>14.200000</ele>
  <time>2014-08-08T12:20:38Z</time>
  <extensions>
    <gpxx:Temperature>32.000000</gpxx:Temperature>
  </extensions>
</trkpt>

Which are parsed correctly and result in the following usage:

In [31]: print p
[trkpt:1.234,-1.234@14.6@2014-08-08 12:20:38]

In [32]: print p.extensions
{u'gpxx:Temperature': u'32.000000'}

In [33]: print p.extensions['gpxx:Temperature']
32.000000

I'm guessing there is a bug in the new parser and how it deals with an arbitrarily long chunk of custom xml. It might be assuming that it would only find a single line. You should file it as a bug, and/or submit a code patch pull request.

@tkrajina

This comment has been minimized.

Show comment
Hide comment
@tkrajina

tkrajina Aug 11, 2014

Owner

This is one of the reasons why I still haven't merged the gpx-1.1 branch into master. The "extensions" thing works only for extensions with (xml tree) depth 1.

For example:

<extensions>
   <aaa>bbb</aaa>
   <ccc>ddd</ccc>
</extensions>

...will be parsed into:

{ 'aaa': 'bbb', 'ccc': 'ddd' }

But it won't work for:

<extensions>
   <aaa>
      <ccc>ddd</ccc>
   </aaa>
</extensions>

My (unfinished) idea was to parse this as:

{ 'aaa/ccc': 'ddd' }

But then there is a problem of how to store attributes for nested xml tags, like:

<extensions>
   <aaa>
      <ccc yyy="qqq">ddd</ccc>
   </aaa>
</extensions>

A possible solution for nested tags is:

{ 'aaa/ccc': 'ddd' }

...and for attrubtes use something like:

{ 'aaa/ccc.yyy': 'ddd' }

But even the dict for extensions is probably not the best solution, because tags don't need to be unique. For example, what to do when the extension is:

<extensions>
   <aaa>bbb</aaa>
   <aaa>ccc</aaa>
</extensions>

The seconds "aaa" tag will overwrite the first one in the dict.

The "extension" thing should be refactored in order to work as a dict for simple extensions and have some API for more complicated extension constructs.

Owner

tkrajina commented Aug 11, 2014

This is one of the reasons why I still haven't merged the gpx-1.1 branch into master. The "extensions" thing works only for extensions with (xml tree) depth 1.

For example:

<extensions>
   <aaa>bbb</aaa>
   <ccc>ddd</ccc>
</extensions>

...will be parsed into:

{ 'aaa': 'bbb', 'ccc': 'ddd' }

But it won't work for:

<extensions>
   <aaa>
      <ccc>ddd</ccc>
   </aaa>
</extensions>

My (unfinished) idea was to parse this as:

{ 'aaa/ccc': 'ddd' }

But then there is a problem of how to store attributes for nested xml tags, like:

<extensions>
   <aaa>
      <ccc yyy="qqq">ddd</ccc>
   </aaa>
</extensions>

A possible solution for nested tags is:

{ 'aaa/ccc': 'ddd' }

...and for attrubtes use something like:

{ 'aaa/ccc.yyy': 'ddd' }

But even the dict for extensions is probably not the best solution, because tags don't need to be unique. For example, what to do when the extension is:

<extensions>
   <aaa>bbb</aaa>
   <aaa>ccc</aaa>
</extensions>

The seconds "aaa" tag will overwrite the first one in the dict.

The "extension" thing should be refactored in order to work as a dict for simple extensions and have some API for more complicated extension constructs.

@b-jazz

This comment has been minimized.

Show comment
Hide comment
@b-jazz

b-jazz Jan 7, 2015

When you are talking about 'aaa/ccc' for nested tags, that would just be for the object model, right? When you output a new gpx file with gpxpy (after doing whatever processing you do), it will still convert it into the original form (assuming it is untouched), right? I think that is very crucial and just want to make sure that would be the case.

Also, wouldn't putting a slash in the name cause problems since it is a valid, non-reserved character and you might already have an xml element with the name "aaa/ccc"?

b-jazz commented Jan 7, 2015

When you are talking about 'aaa/ccc' for nested tags, that would just be for the object model, right? When you output a new gpx file with gpxpy (after doing whatever processing you do), it will still convert it into the original form (assuming it is untouched), right? I think that is very crucial and just want to make sure that would be the case.

Also, wouldn't putting a slash in the name cause problems since it is a valid, non-reserved character and you might already have an xml element with the name "aaa/ccc"?

@alkino

This comment has been minimized.

Show comment
Hide comment
@alkino

alkino May 23, 2015

What do you think of?

<extensions>
   <aaa>bbb</aaa>
   <aaa>ccc</aaa>
</extensions>
{'aaa': ['bbb', 'ccc']}
<extensions>
   <aaa>
      <ccc yyy="qqq">ddd</ccc>
   </aaa>
</extensions>
{'aaa': {'ccc': 'ddd'}} #don't know about 'yyy' :s

alkino commented May 23, 2015

What do you think of?

<extensions>
   <aaa>bbb</aaa>
   <aaa>ccc</aaa>
</extensions>
{'aaa': ['bbb', 'ccc']}
<extensions>
   <aaa>
      <ccc yyy="qqq">ddd</ccc>
   </aaa>
</extensions>
{'aaa': {'ccc': 'ddd'}} #don't know about 'yyy' :s
@jaap3

This comment has been minimized.

Show comment
Hide comment
@jaap3

jaap3 Aug 5, 2015

My (admittingly naive) idea would be to allow users to register a handler for extensions. gpxpy would then just pass on the extension nodes to the handler responsible for that extension type.

jaap3 commented Aug 5, 2015

My (admittingly naive) idea would be to allow users to register a handler for extensions. gpxpy would then just pass on the extension nodes to the handler responsible for that extension type.

@JaniszM

This comment has been minimized.

Show comment
Hide comment
@JaniszM

JaniszM Sep 22, 2015

Will this branch be merged into master? (https://github.com/tkrajina/gpxpy/tree/gpx-1.1) I would like to have support for 1.1 because of tag :)

JaniszM commented Sep 22, 2015

Will this branch be merged into master? (https://github.com/tkrajina/gpxpy/tree/gpx-1.1) I would like to have support for 1.1 because of tag :)

@tkrajina

This comment has been minimized.

Show comment
Hide comment
@tkrajina

tkrajina Sep 23, 2015

Owner

@alkino Yes, you found the problem :) what with yyy? I was thinking of something like

<aaa>www</aaa>
translates to
{'aaa': 'www'}

and

<aaa yyy="qqq"><ccc>ddd</ccc></aaa>
translates to
{'aaa': {'$yyy':'qqq', 'ccc': 'ddd'}}

But then what to do with something here: <aaa>something<ccc yyy="qqq">ddd</ccc></aaa>? Maybe we can quietly ignore this scenario :)

Also note that in the XML the order of subelements is preserved, but translating it into a dict the order is gone (i.e. one more XML information is lost, this can be a problem).

@JaniszM If you all guys agree I can merge it in master and release a new version immediately but with one important exception -- the exception API will change in the future. Because the way it is now it works only for the most trivial situations. Something like @jaap3 proposed might help, but at the moment I don't have time to finish this :(

Ignoring exceptions, I think the gpx-1.1 branch should be ready for merging.

Owner

tkrajina commented Sep 23, 2015

@alkino Yes, you found the problem :) what with yyy? I was thinking of something like

<aaa>www</aaa>
translates to
{'aaa': 'www'}

and

<aaa yyy="qqq"><ccc>ddd</ccc></aaa>
translates to
{'aaa': {'$yyy':'qqq', 'ccc': 'ddd'}}

But then what to do with something here: <aaa>something<ccc yyy="qqq">ddd</ccc></aaa>? Maybe we can quietly ignore this scenario :)

Also note that in the XML the order of subelements is preserved, but translating it into a dict the order is gone (i.e. one more XML information is lost, this can be a problem).

@JaniszM If you all guys agree I can merge it in master and release a new version immediately but with one important exception -- the exception API will change in the future. Because the way it is now it works only for the most trivial situations. Something like @jaap3 proposed might help, but at the moment I don't have time to finish this :(

Ignoring exceptions, I think the gpx-1.1 branch should be ready for merging.

@jaap3

This comment has been minimized.

Show comment
Hide comment
@jaap3

jaap3 Sep 23, 2015

Sorry did you mean extensions instead of exceptions?

jaap3 commented Sep 23, 2015

Sorry did you mean extensions instead of exceptions?

@JaniszM

This comment has been minimized.

Show comment
Hide comment
@JaniszM

JaniszM Sep 23, 2015

I don't need it right now, so If you can work on exception/extension it's ok for me to wait a bit :) It's not critical point right now to have this <speed> tag from <extensions>. So feel free to do what you guys think is right or you need :)

Edit: I haven't time before to read all comment but I just did it.
If you want to use simple translation without attributes you can simply do it using combination of tabs and dict. Then you can do as much levels as you want.
But If you want to store also attribute I think there is a moment when you need class for it where members can store attributes and function can get those attributes from dict/tab structure. This is only my idea :)

JaniszM commented Sep 23, 2015

I don't need it right now, so If you can work on exception/extension it's ok for me to wait a bit :) It's not critical point right now to have this <speed> tag from <extensions>. So feel free to do what you guys think is right or you need :)

Edit: I haven't time before to read all comment but I just did it.
If you want to use simple translation without attributes you can simply do it using combination of tabs and dict. Then you can do as much levels as you want.
But If you want to store also attribute I think there is a moment when you need class for it where members can store attributes and function can get those attributes from dict/tab structure. This is only my idea :)

@tkrajina

This comment has been minimized.

Show comment
Hide comment
@tkrajina

tkrajina Oct 4, 2015

Owner

OK, gpx-1.1 is now merged in master and a new version (1.0) is deployed on pypi. Thanks you all all for your suggestions and feedback.

Owner

tkrajina commented Oct 4, 2015

OK, gpx-1.1 is now merged in master and a new version (1.0) is deployed on pypi. Thanks you all all for your suggestions and feedback.

@tkrajina tkrajina closed this Oct 4, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment