Missing Nominatim data

[IzzySoft]

I've noticed for a while now that Nominatim data are often missing. Having returned from Portugal, out of ~1k photos only ~300 have any Nominatim data at all (country set to PRT), none of them have state or city information (while all of them have GPS data).

Now my 2 questions on this:

• What can I do to improve that? Some kind of rescan?
• Could I easily add at least country information to all apps in a given album, or even city information to multiple selected images?

A third question would of course be how this could be solved in a way not requiring manual editing. I've noticed a similar thing in Spain as well, where many photos just had ESP as country. Seems like I didn't have such issues with e.g. UK (where it even labeled "Scotland" as state and also had city information correct) or DE (cities set, though no states) or Cyprus (cities set).

[deckerst]

Aves finds countries/addresses in 2 distinct steps:

1. first a quick pass, completely offline and autonomously, using rough topographic boundaries to find the country, but not the full address. Because of the nature of the data, this is imprecise and it can easily miss locations (close to a coastline for example). Also, this give unlocalized country codes (e.g. PRT).
2. then another pass, requiring mobile services installed on the device (typically GMS, but in your case I suppose you plugged Nominatim), to do reverse geocoding. This is more precise and yields a full address, including the city and the country name in your local (e.g. Portugal). As far as I know, the mobile services require internet access to do reverse geocoding.

What can I do to improve that? Some kind of rescan?

You could try to select the media items and in the top right menu select the Rescan action. But as I remember, I make Aves go through pass 2 on launch for media items that only got the country from pass 1, if mobile services are detected on the device.

Could you paste here the system info you get in About > Bug Report?

Could I easily add at least country information to all apps in a given album, or even city information to multiple selected images?

If you do have a country code but no city, then I suppose the mobile services are not installed or they fail to do reverse geocoding. You could look into that.

If with your current setup even the country is missing, the only option would be to change the GPS coordinates in the metadata (you can do that in Aves). Or you could forgo GPS coordinates and derived address, and simply use tags for the country and/or city.

[IzzySoft]

Thanks for the details, Thibault!

first a quick pass … then another pass

That explains numbers having got up (seems like there's some "delayed event" performing those scans?); meanwhile PRT has 467 entries (with no new photos/videos added in the last 2 days).

You could try to select the media items and in the top right menu select the Rescan action.

Tried that with 2 example photos selected, details shown didn't change (just date, size and GPS displayed on top). Numbers in stats didn't change. Comparing with one of the photos from the PRT filter, it has PRT instead of the GPS coordinates.

if mobile services are detected on the device.

As you correctly remember, this is the device we've talked about in the past and where the vendor provided a quick fix to make Nominatim working a little over a year ago. Setup didn't change since then, so Nominatim per se should still be working. Funny thing is, too, that some media from Portugal correctly get the PRT while others do not, and that with some other countries it seems to even work fully (e.g. during my stay in Aberdeen 9 month ago I get the full country as "United Kingdom", the state as "Scotland" and even the cities as Aberdeen, Stonehaven etc; some places in Spain work as well, where I have 1k correctly marked "Spain" (plus 354 just ESP) with cities like Valencia, Segovia, Toledo – Spain I visited in March/April; Germany seems to work entirely fine).

Could you paste here the system info you get in About > Bug Report?

Sure: aves_report.zip contains the about.txt (copy-paste from step 2) and 2 log files (from step 1) where you'll be mostly interested in the second, which should cover the rescan run.

If you do have a country code but no city, then I suppose the mobile services are not installed or they fail to do reverse geocoding. You could look into that.

That's what puzzles me: I didn't change a thing with them for at least half a year, and they worked fine for the media from Spain 1.5 months ago – at least partly (some still have ESP instead of Spain). If you give me a hint / steps to follow to narrow that down, I'll of course do that and provide you with the details – be it via Aves directly, via some FOSS helper app, or via ADB.

Or you could forgo GPS coordinates and derived address, and simply use tags for the country and/or city.

You mean, select the media, then add the e.g. PRT tag to them? I couldn't find a related menu entry for that. And doing that for hundreds of media manually is quite cumbersome (tried that for one but couldn't find "existing labels" listed, just the option to create a new tag or use any of the 3 placeholders Country, State, Place – I must have missed something there).

I've checked with the FAQ (here in the wiki), but it is not clear how to use any of the existing tags one did not manually create oneself).

[deckerst]

Lots to reply to, but I must go soon, so I'll just point out that in your system info I see Mobile services: not available.
From the point of view of Aves, there's no mobile services so it won't even try to do the reverse geocoding. Do you have MicroG installed? (Nominatim is just the reverse geocoder, right?)
Edit: My bad, actually mobile services are just checked for google map layers. Geocoding is separate.

[IzzySoft]

actually mobile services are just checked for google map layers. Geocoding is separate.

Just to point out I'm running the "libre" variant here, that comes entirely without the Google Map layers even 😉 The device has microG installed, but all its Google connections (sign in, FCM etc) disabled.

[deckerst]

Geocoders typically requires being connected to the internet, so Aves checks connectivity before calling the geocoder (otherwise unconnected users would see these attempts take a long time before timeouts kick in).

Do you think a lack of connectivity on Aves launch could explain things in your case?

[deckerst]

couldn't find "existing labels" listed

This is strange. In the tag editor, you should see a "Top Tags" section showing your existing tags (found in your collection, as they are listed in the Tags page) ordered by usage.

If you go to the Tags page, is it empty?

[IzzySoft]

Do you think a lack of connectivity on Aves launch could explain things in your case?

I don't think that's likely. The device Aves runs on here has a good data plan, and the rescan was done while on WiFi. Nor can I remember any connectivity issues on that device (my other one, an old FP2, needs a reboot every 2 weeks or so as it starts missing its SIM, or gets its WiFi "flaky" – but the SHIFT6mq runs stable as a rock).

In the tag editor, you should see a "Top Tags" section

That's the section I'm missing! Hm, maybe something specific with the flavor? I'm running the "libre" one from F-Droid. Oh, and the device is still on Android 10, in case that matters.

Lemme cross-check, I've got the "normal" version on the FP2 (rarely use it there as it seems too heavy for that 32bit device – but that's a different issue)… Nope, no "Top Tags" there either. Ugh, but that's missing some updates, wow – still on 1.7.1, why? What's more strange: Aves does not show up there in the F-Droid client as "installed app", now I'm baffled. Querying the FP2 via ADB:

$ adb shell "pm list packages -3" |grep aves
package:deckers.thibault.aves.profile

Oof. That looks like some test version where we might have tried things in the past…

If you go to the Tags page, is it empty?

Menu already says "Tags 0". So yes: "No tags".

[deckerst]

The device Aves runs on here has a good data plan, and the rescan was done while on WiFi.

I see, thanks. I'll need to think of ways to debug this further.

deckers.thibault.aves.profile

Yes, this is a test package. You can remove it.

Menu already says "Tags 0". So yes: "No tags".

If you don't have existing tags, then it makes sense that you don't have the Top Tags section in the editor. I thought you were already using tags in your collection. In any case, tags are free text, you can add any text as a tag.

[IzzySoft]

If you don't have existing tags

No manual ones, no. I was hoping to be able using the location tags there when they're missing, so they also show up in the stats. I make heavy use of albums, though, so this is less about finding images 😉

Yes, this is a test package. You can remove it.

I just uninstalled that test version from the FP2 and installed the full version from my repo. Starts up fine, but stays empty as if there were no images to show (~200 photos on that device). Rebooted the device, ran `adb logcat |egrep '(aves|flutter)', started Aves again – this is the aves.log created by that. Screen says "Collection loading", nothing happens. So I unfortunately cannot cross-check there (I can make a separate issue for that if you want – but the FP2 isn't really suited to run large Flutter apps, and gets replaced here soon™ with a SHIFTphone8 as soon as that's ready.

But back to the issue at hand: Let me know if I can do anything to help finding the culprit. If you e.g. have a way I could try to manually query the system's Nominatim (e.g. via ADB, or a small sample app), or a version producing the relevant debug output, I could do that.

[deckerst]

I was hoping to be able using the location tags there when they're missing

Ah, I see. This is actually a good idea. I'll try and think how I can make that possible in the future.

But back to the issue at hand: Let me know if I can do anything to help finding the culprit. If you e.g. have a way I could try to manually query the system's Nominatim (e.g. via ADB, or a small sample app), or a version producing the relevant debug output, I could do that.

Sure. I'll try when I find the time.

[deckerst]

I can make a separate issue for that if you want

Yes please, with the additional info from the instructions in About > Bug Report. I see from your logs that it goes through most steps in the initialization process but the final initialization log line is missing.

[IzzySoft]

Yes please, with the additional info from the instructions in About > Bug Report.

Will do. A wish at that place: would be good if step 2 could also be saved directly to a file instead of just to the clipboard. If one doesn't have a text editor on the device, it's a bit tricky to get those details to the PC one does the "browser stuff" in (I don't use Github etc. from my mobile devices). On the FP2 I have no editor… OK, managed to install one (had to reboot for that for some reason, time my new device arrives). Created #1022 for this.

Thanks a lot!