Add GPS location and time sync support

[gkoh]

Discussed in #42

^{Originally posted by stulevine January 20, 2023}
Hi @gkoh. Just wanted to let the you all know that I just installed furble to a new M5stack M5Stick C Plus ESP32-PICO-D4 Mini IoT Development Kit device purchased on Amazon.com and it works just perfectly with my new Fujifilm X-T5 with firmware v1.0.2 (just released this morning). Nice alternative to using my iPhone. The only other thing that would be great is if the M5Stick supported GPS locations sourcing. The X-T5 can use the CamRemote app on the iPhone when connected view BT to derive the GPS location and add it to the photo meta data. I'd be curious if this could be accomplished using the WiFi module of the M5Stick? Curious if anyone has any thoughts on this. I guess, if the M5Stick does not have GPS support, that I could purchase a GPS module and connect it to the GPIO pins and see how difficult it would be to support location transmission to the camera.

Stuart

Update: M5Stack does make a GPS unit that plugs into the M5Stick Grove connector interface and works with M5Core. Here is the product link I might order it and see if it's possible to send GPS coordinates to the camera. I think the camera will poll the device for that info if enabled in the setup menu on the camera. Any insights would be greatly appreciated.

[gkoh]

Initial investigation shows the following:

• geotag data (and GPS time) is definitely transferred over bluetooth only
• characteristic 0x8012 is used to send this
  • all the data is packed into a single multi-byte packet
• latitude and longitude are sent as signed 32-bit integers multiplied by 10,000,000
• altitude is sent as metres
  • data width is unclear, I'm too close to sea level and see lots of zeros
• time sync data also here, using the same format as the 'normal' time sync data

[gkoh]

The data packet structure for characteristic 0x8012 is 23 bytes, and looks like:

struct {
  uint16_t year;
  uint8_t month;
  uint8_t day;
  uint8_t hour;
  uint8_t minute;
  uint8_t second;
} time_t;

struct {
  int32_t latitude;
  int32_t longitude;
  int32_t altitude;
  uint8_t pad[4];
  time_t time;
} geotag_t;

The camera appears to notify the remote of the geotag configuration on characteristics 0x5032 and 0x5034. Further digging is needed to clarify this.

[stulevine]

Hi @gkoh Just received my GPS unit. Any progress on your last message regarding the characteristics for geotagging?

[gkoh]

Hi @gkoh Just received my GPS unit. Any progress on your last message regarding the characteristics for geotagging?

Hi @stulevine, I've got it mostly figured out with snippets of code, but nothing concrete yet.
My GPS unit is still in transit, shipping was delayed due to Lunar New Year (but it's on its way!).
I should be able to make fairly rapid progress once the unit arrives.

[gkoh]

Some further protocol information.
When the camera is configured for no geotag, we get no notifications on the NOT2 characteristic.
If geotag is set to 'Time', 'Location' or 'Time & Location', the camera sends 0x0100 to the NOT2 characteristic.
The format and contents of the return packet do not appear to change either.
It seems the camera will either ask for:

• nothing OR
• all geotag information

and then decides what it wants to use.

Furthermore, the camera needs to be powered off and on for that NOT2 notification to be sent.
If I disconnect and reconnect without a power off, we don't get the notification at all.

[stulevine]

Some further protocol information. When the camera is configured for no geotag, we get no notifications on the NOT2 characteristic. If geotag is set to 'Time', 'Location' or 'Time & Location', the camera sends 0x0100 to the NOT2 characteristic. The format and contents of the return packet do not appear to change either. It seems the camera will either ask for:

• nothing OR
• all geotag information

and then decides what it wants to use.

Furthermore, the camera needs to be powered off and on for that NOT2 notification to be sent. If I disconnect and reconnect without a power off, we don't get the notification at all.

Yes, that sound about right when I use the CamRemote app on my iPhone. Pretty much works the same. Interestingly, once it does connect via BT, and if I quit the app, allowing said app to work in the background, the camera still received Geotagging info updates. I guess the camera, once connected, does a periodic query to the phone to get updated GPS info.

[gkoh]

@stulevine Thanks for the information. That's something I haven't confirmed yet, and there are conflicting reports around geotag update interval.
At the moment I can't figure out:

• if the camera asks for a geotag update
• the phone auto-sends a geotag update at some interval
  • if this, what is 'some interval'?

When my unit arrives I'll try sending stuff to it.
Need to be careful though, have managed to hang my camera with the shutter open at least once 😐

[stulevine]

@stulevine Thanks for the information. That's something I haven't confirmed yet, and there are conflicting reports around geotag update interval. At the moment I can't figure out:

• if the camera asks for a geotag update

I'm pretty sure that's the way it works. Mainly because you turn on geotagging in the camera. With that said, I'd expect it not to poll for location info if that setting is off. I noticed that when I have it set to on in the camera, as soon as I open the CamRemote app, and it connects, the geolocation icon appears on the camera screen.

• the phone auto-sends a geotag update at some interval

I think it's the opposite way around. I would assume the camera does the polling. Let's assume this flow for the initial testing. Is there a way for me to debug the CamRemote connection to the camera? I think that if my hunch is correct, you won't need to worry about the interval just do something when the camera requests the geo location info. Make sense? I'm an iOS developer and I think this is the most logical approach.

• if this, what is 'some interval'?

When my unit arrives I'll try sending stuff to it. Need to be careful though, have managed to hang my camera with the shutter open at least once 😐

[stulevine]

Hi @gkoh,

How is it going with the implementation? Not sure if you saw my reply above. Just curious if you had any more questions or issues.

Thanks,

Stuart

[gkoh]

Hi @stulevine,
Sadly, my GPS unit is in postal limbo, its status hasn't moved for weeks 🙁
However, I have had a good go at the core GPS handling work, please see the branch:
https://github.com/gkoh/furble/tree/43-add-gps-location-and-time-sync-support

This seems to work for my X-T30, if you're able, please test with your camera.
The coordinates are hardcoded to Montevideo, Uruguay, to test negative lat/long.
Additionally, the time reference is Christmas day, 2024 (yes, the future), at 12:34:56.

Assuming this part works, just need to modify src/furble.ino to open the GPS unit and feed in the numbers.
I was looking at this library:
https://github.com/mikalhart/TinyGPSPlus

[gkoh]

As part of this feature, I'm also thinking about a setting to disable the screen during the remote menu.
If the M5StickC needs to stay on for feeding GPS, it will currently last little more than an hour, perhaps disabling the screen will extend that lifetime. We could instead flash the LED to indicate operation?

Alternately, we could also support this unit:
https://shop.m5stack.com/products/m5stack-esp32-core-ink-development-kit1-54-elnk-display

It's almost double the price, but the three-way switch and more than triple the battery life along with much lower screen power consumption make it very interesting.
Assuming it works as advertised, we should be able to easily get multiple hours of runtime from such a device.

[gkoh]

Despite being quite sure about the geotag message format being correct, the camera does not always take it.
In fact, it only occasionally accepts the message and displays the 'geo sync' icon.
Not sure what's going on here, need to investigate further.

[stulevine]

As part of this feature, I'm also thinking about a setting to disable the screen during the remote menu. If the M5StickC needs to stay on for feeding GPS, it will currently last little more than an hour, perhaps disabling the screen will extend that lifetime. We could instead flash the LED to indicate operation?

Yes, I too noticed the battery drain with the GPS unit attached. I think adding a setting to turn off the screen instead of just dimming it would help. Of course it should wake up when a button is pressed.

Alternately, we could also support this unit: https://shop.m5stack.com/products/m5stack-esp32-core-ink-development-kit1-54-elnk-display

That's an interesting option too. Funny thing, this is what I paid for my M5StickC on Amazon. I did so just to get the product in my hand quickly with free shipping (in the US). I would consider this option too. However, I'd like to see how the aforementioned option to turn off the screen works as well before taking the plunge.

It's almost double the price, but the three-way switch and more than triple the battery life along with much lower screen power consumption make it very interesting. Assuming it works as advertised, we should be able to easily get multiple hours of runtime from such a device.

[stulevine]

Hi @stulevine, Sadly, my GPS unit is in postal limbo, its status hasn't moved for weeks 🙁 However, I have had a good go at the core GPS handling work, please see the branch: https://github.com/gkoh/furble/tree/43-add-gps-location-and-time-sync-support

This seems to work for my X-T30, if you're able, please test with your camera. The coordinates are hardcoded to Montevideo, Uruguay, to test negative lat/long. Additionally, the time reference is Christmas day, 2024 (yes, the future), at 12:34:56.

Assuming this part works, just need to modify src/furble.ino to open the GPS unit and feed in the numbers. I was looking at this library: https://github.com/mikalhart/TinyGPSPlus

I'll give this branch a try and let you know how it works with my X-T5.

[gkoh]

I'll give this branch a try and let you know how it works with my X-T5.

I've just pushed updates to the branch, which seem to make the geotag updates be accepted by the camera.
Let me know how it goes.

[gkoh]

As part of this feature, I'm also thinking about a setting to disable the screen during the remote menu. If the M5StickC needs to stay on for feeding GPS, it will currently last little more than an hour, perhaps disabling the screen will extend that lifetime. We could instead flash the LED to indicate operation?

Yes, I too noticed the battery drain with the GPS unit attached. I think adding a setting to turn off the screen instead of just dimming it would help. Of course it should wake up when a button is pressed.

OK, perhaps a 3 way option for the remote control screen:

• dim screen
• screen off
• LED blink/pulse/something

Alternately, we could also support this unit: https://shop.m5stack.com/products/m5stack-esp32-core-ink-development-kit1-54-elnk-display

That's an interesting option too. Funny thing, this is what I paid for my M5StickC on Amazon. I did so just to get the product in my hand quickly with free shipping (in the US). I would consider this option too. However, I'd like to see how the aforementioned option to turn off the screen works as well before taking the plunge.

Certainly. It appears my GPS unit has finally moved (after being stuck in an airport for a month).
Hopefully we can characterise the power consumption soon and make better informed decisions.

[gkoh]

I'll give this branch a try and let you know how it works with my X-T5.

I've just pushed updates to the branch, which seem to make the geotag updates be accepted by the camera. Let me know how it goes.

Some further testing and analysis.
It looks like the camera is caching and comparing timestamps on the GPS geodata.
If I wait long enough (days) the camera accepts the geodata sync, but anytime prior to that, it will not show me the 'geosync' icon.
When I incremented the hardcoded GPS timestamp by one second, the sync was accepted.
Thus, I conclude having a real GPS unit should 'just work' and the oddities I'm seeing are all related to fake data.

[gkoh]

@stulevine Apologies for the long silence.
My GPS unit finally arrived and I have managed to find enough time to get the proof-of-concept working on the branch.
Please give it a try when you have an opportunity.
The major things to know are:

• LED blinks slow for "no fix", faster for "fix"
  • this is just a quick visual indicator for now, not considered final in any way
• the GPS fix timeout is 60s, if we receive no GPS updates in 60s, we assume "no fix"
• GPS is always active right now
  • need to add a setting to control this
• added quick hacky settings menu item to display GPS data
  • press B to exit
• need to be in 'Shutter' control loop for the GPS geotag data to be updated properly
  • could add a 'GPS' mode menu entry to handle this specially
• battery life super sucks and needs work

There's lots of things we can do to improve this, but I think actually works 🙂