Merge clear_GPS.lua and reset_GPS.lua into geoToolbox.lua

[wpferguson]

When I submitted a PR to add reset_GPS.lua @supertobi noted that it might be a good fit in geoToolbox.lua. I had just read through the geoToolbox.lua code and thought the same thing, so I closed the PR and decided to merge the code. clear_GPS.lua also manipulates image GPS information so that is also probably a good fit in geoToolbox.lua.

I opened this issue so that there would be a place to discuss the merge and how to arrive at a best fit.

[wpferguson]

clear_GPS and reset_GPS both add buttons (actions) to the selected images module. Should they do that or put the buttons in geoToolbox or both? I'm fine with any of the solutions.

[wpferguson]

And the GUI could look like this

[supertobi]

I would put the buttons to the geoToolbox, but the selected images module is fine too. I'm against putting it in both.
Perhaps change the label "reset GPS data" to something like "reset GPS to original image data". You could even show the original and the new GPS data above the button.
And if you are at it, the "reverse geocode" function needs some love. 😃

[wpferguson]

@supertobi I understand pick the first image, pick the second, calc_in_between, but I don't understand what the slider is for.

[supertobi]

Example:
Image A Time 10:00 Position X
Image B Time 11:00 Position Y
Image C Time 10:15 Position ?

Now the calc in between should position Image C closer to A.
A-----C-------------------B

The idea of the slider was to move the image location closer to A or B.

A----------------C---------B
0               60        100

The use case is, that the GPS attached to my camera sometimes does not save the GPS data. (Hiking in a forest.) Or sometimes it safes wrong data. (Hiking in the mountains.) The nice thing when hiking is that you move move or less with linear speed.

[supertobi]

Btw. I have an old description of the geoToolbox here (But without the calc in between):
https://dablogter.blogspot.com/2017/04/darktable-geotoolbox-script.html

[wpferguson]

Should we move kml_export, gpx_export and geoJSON_export in geoToolbox also?

[supertobi]

I'm not sure. I think it would be to big to add everything into one Lua file and if we have more then one file we can even keep it as individual scrips. Perhaps it would be a nice to have the same prefix for all geo related scrips. What do you think?

[wpferguson]

I didn't think about how big the script would grow. I'm already going to merge 2 other scripts with it, so adding another 3 would make it fairly sizeable. Probably not a good idea.

[wpferguson]

I've thought about

Example:
Image A Time 10:00 Position X
Image B Time 11:00 Position Y
Image C Time 10:15 Position ?

and I have an idea.

If I understand this correctly then Image A had good GPS data and Image B has good gps and all the images between A and B don't have or have bad GPS data. So, if I select all the images from A to B and call a routine, I can use the first image for the starting GPS info and the last image for the ending GPS info. As far as slewing the data I thought about 2 sliders, or a drop down an a slider. The first slider or drop down picks the image where the speed changed and the second slider adjusts where, location wise, it really was. So if you went 5 km between A and B but the 1st km was up a steep hill and the last 4 were downhill, you would select the image taken at or near the top of the hill and move the slider to the 1 km point, so we would assume constant but slow progress for the first km and quicker but constant progress for the last 4 km.

In order to test this, I think I might actually have to get some exercise. My camera can record GPS so I can take a sequence and save the GPS info, then select the first and last image and do a calculation and compare the results.

I just had another thought. Back to the A B example. If you had another sequence between the two points where your speed varied, you could probably sub select those images and slew them and replace the calculated GPS with a new calculated GPS.

Okay, one last thought. Do we need a set GPS to set an image to a known coordinate? Back to the A B example, if there is a known landmark and I took a picture there, then I know the coordinates so I could set that location and narrow down the calculation.

[supertobi]

If I understand this correctly then Image A had good GPS data and Image B has good gps and all the images between A and B don't have or have bad GPS data. So, if I select all the images from A to B and call a routine, I can use the first image for the starting GPS info and the last image for the ending GPS info. As far as slewing the data I thought about 2 sliders, or a drop down an a slider. The first slider or drop down picks the image where the speed changed and the second slider adjusts where, location wise, it really was. So if you went 5 km between A and B but the 1st km was up a steep hill and the last 4 were downhill, you would select the image taken at or near the top of the hill and move the slider to the 1 km point, so we would assume constant but slow progress for the first km and quicker but constant progress for the last 4 km.

Yes, that's what I was thinking about.

In order to test this, I think I might actually have to get some exercise. My camera can record GPS so I can take a sequence and save the GPS info, then select the first and last image and do a calculation and compare the results.

Absolutely, this is not about hard math and we will never get it 100% right. Perhaps don't invest to much time into it.

I just had another thought. Back to the A B example. If you had another sequence between the two points where your speed varied, you could probably sub select those images and slew them and replace the calculated GPS with a new calculated GPS.

It sounds interesting, but I'm not sure how you'd like to implement that. But Id be happy to test a prototype.

Okay, one last thought. Do we need a set GPS to set an image to a known coordinate? Back to the A B example, if there is a known landmark and I took a picture there, then I know the coordinates so I could set that location and narrow down the calculation.

We can already place images in the map mode, I'm not sure we need more

[wpferguson]

We can already place images in the map mode, I'm not sure we need more

Agreed

It sounds interesting, but I'm not sure how you'd like to implement that. But Id be happy to test a prototype.

     B
    / \
   /     \ 
  /         \
 /             \
A                \                 D
                     \            /
                        \       /
                           \  /
                             C

The first cut is this

A------------B-----------C-------------D

adjusted to be

A----B---------------C-----------------D

then select B to D and adjust C

B---------------C-----------------D

to

B-------------------------C-------D

In other words, once we do the first interpolation, all of the images have GPS information, so we can select any subset of them and fine tune the distribution. And, if we totally screw it up, we just do clear_gps() and start over :-D.

[supertobi]

Sounds good.

[wpferguson]

I'm looking at the altitude plot export. I notice that you're separating the fields with a semi-colon instead of a comma. Is this a locale thing, since the decimal separator is a comma? I guess what I'm trying to figure out is should this be a locale aware/specific export?

[wpferguson]

Learned something new. dt.gui.slection() is sorted by image ID. My Play Raw collection had 8 images with GPS so I selected them and the distance got computed. I selected the first image, and the last image and the distance was different. I had sorted them by time, so it turned out the 4th image had the lowest ID and that was the first image and the last image really was the last. I'm not sure about how dt.gui.action_images sorts, but I wouldn't be surprised if it wasn't the same.

[supertobi]

I'm looking at the altitude plot export. I notice that you're separating the fields with a semi-colon instead of a comma. Is this a locale thing, since the decimal separator is a comma?

Yes, in Germany we are using semi-colons in CSV files. I've never thought about this, but you could be right:
https://en.wikipedia.org/wiki/Comma-separated_values

I guess what I'm trying to figure out is should this be a locale aware/specific export?

According to Wikipedia there are even countries that use both "." and ",". (may vary by location or other factors):
https://en.wikipedia.org/wiki/Decimal_separator#Influence_of_calculators_and_computers
Perhaps we should just add a preference to define the separator.

[wpferguson]

Then I guess we treat csv as character separated values and offer a choice of comma, semi-colon, or tab.

Progress update: Everything is implemented except interpolate and reverse-geocode. I'm 2/3 done with interpolate. Once I finish those 2, I'll go back and do a final refactor to get all the redundant code out and get the code size down as far as I can. After that a final cleanup to make sure all the strings are translatable and comments are in place, then I'll send it to you for testing. I've tested most of the simple stuff, but the interpolation might be difficult. I can make up a test and make sure the locations are distributed properly, but you are the one with the valid use case so your testing may show something I didn't think of.

The UI is big. I've used side by side buttons where I could to reduce the size. I fixed side by side buttons in the API so that they expand and fill all the space, which got merged today and will be in 3.4. I have a couple of ideas, so I'll include them both in the code and you can check them, or maybe you'll have a better idea...

[wpferguson]

And if you are at it, the "reverse geocode" function needs some love.

cat mapbox.out.json | jq '.features | .[] | '.text''
"Lake Mangamahoe"
"New Plymouth"
"Taranaki"
"New Zealand"

from a play raw that had GPS info.

Mapbox's terms of use say that you can't save the information. There's a request to tag from the information. I'm thinking of adding Nominatim from openstreetmap. It doesn't require a key, but it is rate limited. Since reverse_geocode only does one lookup at a time, I don't think it will be a problem and the results can be saved (i.e. used as tags).

[wpferguson]

'/usr/bin/curl' --silent "https://nominatim.openstreetmap.org/reverse?format=geojson&zoom=10&lat=51.076038883333&lon=-115.37340278333" | '/usr/bin/jq' '.features | .[0] | '.properties' | '.address' | .[]'

[supertobi]

Nominatim sounds better then MapBox.

[wpferguson]

@supertobi here you go. I think some of the functions can maybe move to libraries, but see what you think. Test it and let me know. Thanks
geoToolbox.zip

[supertobi]

At the moment we have all distances in meter or km is that a problem for users in US? And should we change that?

[wpferguson]

I'll make it a preference. I'll still do the calculations in kilometers and meters and just convert to miles and feet

[supertobi]

@wpferguson
I just found geocode-glib, but I think we can't use it. What do you think?
https://gitlab.gnome.org/GNOME/geocode-glib

[wpferguson]

It might be useful if we wrote an external executable in c that we called to do the geocoding. Unfortunately it only uses 1 geocoder, nominatim, instead of offering a choice of geocoders. I'm concerned about swamping nominatim with requests.

[wpferguson]

Also, I added the choice of distance units

[supertobi]

Nice, I'll try to test it over the holidays.

[wpferguson]

It seems that @phweyland has outsmarted us both. You can copy/paste, clear, and reset GPS info from the metadata menu of selected images.

[supertobi]

@wpferguson Then lets remove the copy/past stuff.
Btw. I've tested a little bit and was no able to export the CSV file.
The first problem is, that the button tries to call "altitude_export_plot" but the function is "altitude_plot_export".
Then there is a typo in the function, you wrote "job.vallid" not "job.valid".
And then I was not able to create the CSV when image1.elevation is nil.

Perhaps you can create an PR. That's easier for reviewing.

[wpferguson]

Thanks @supertobi for checking it. I'll fix those few and then do a PR.