New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GeoLite2 Deprecated Coordinates in 2019 #12411

Closed
m-cameron opened this Issue Jan 9, 2018 · 7 comments

Comments

Projects
None yet
5 participants
@m-cameron

m-cameron commented Jan 9, 2018

Maxmind has announced that they will remove coordinates from GeoLite2 in 2019.

https://dev.maxmind.com/geoip/geoip2/geolite2/

How can Piwik works without the coordinates? Must we subscribe to the commercial GeoIP2? Can we use IP2Location LITE DB5?

https://lite.ip2location.com/database/ip-country-region-city-latitude-longitude

@fdellwing

This comment has been minimized.

Show comment
Hide comment
@fdellwing

fdellwing Jan 9, 2018

Contributor

Why you need coordinates? City should be precise enough for 99% of the users that don't use a commercial solution anyway.

Contributor

fdellwing commented Jan 9, 2018

Why you need coordinates? City should be precise enough for 99% of the users that don't use a commercial solution anyway.

@Findus23

This comment has been minimized.

Show comment
Hide comment
@Findus23

Findus23 Jan 9, 2018

Member

I'm not sure if Piwik uses the coordinates to display the cities on the map.

Member

Findus23 commented Jan 9, 2018

I'm not sure if Piwik uses the coordinates to display the cities on the map.

@fdellwing

This comment has been minimized.

Show comment
Hide comment
@fdellwing

fdellwing Jan 9, 2018

Contributor

Well, the map works if you use other methods of geolocation instead of GeoLite DB. So there should be no problem with that?

Contributor

fdellwing commented Jan 9, 2018

Well, the map works if you use other methods of geolocation instead of GeoLite DB. So there should be no problem with that?

@m-cameron

This comment has been minimized.

Show comment
Hide comment
@m-cameron

m-cameron Jan 9, 2018

Need to check the codes if latitude and longitude are being used. Any developers here?

m-cameron commented Jan 9, 2018

Need to check the codes if latitude and longitude are being used. Any developers here?

@sgiehl sgiehl referenced this issue Feb 12, 2018

Closed

[WIP] GeoIP 2 integration #12553

15 of 19 tasks complete

@mattab mattab added this to the 4.0.0 milestone Mar 20, 2018

@mattab

This comment has been minimized.

Show comment
Hide comment
@mattab

mattab Apr 7, 2018

Member

As maxmind itself says "Latitude and Longitude are often near the center of population. These values are not precise and should not be used to identify a particular address or household."

from internal discussions: we do not really need lat/long values for each user but the lat/long are currently used for the following:

One solution would be to have, in Matomo, a static mapping of cities -> lat,long and use this instead, so we know for sure lat/long are not more precise than the closest city. Then we could use this static mapping to fill in the lat/long info for each visit (even in the DB raw data) and keep our APIs and Maps working as before.

Whether to keep or delete the fields lat/long in the DB, because the values can be set manually via the Tracking API, many people use and rely on this valuable info (and may have full consent to track locations), so we can keep the fields in the tables.

Member

mattab commented Apr 7, 2018

As maxmind itself says "Latitude and Longitude are often near the center of population. These values are not precise and should not be used to identify a particular address or household."

from internal discussions: we do not really need lat/long values for each user but the lat/long are currently used for the following:

One solution would be to have, in Matomo, a static mapping of cities -> lat,long and use this instead, so we know for sure lat/long are not more precise than the closest city. Then we could use this static mapping to fill in the lat/long info for each visit (even in the DB raw data) and keep our APIs and Maps working as before.

Whether to keep or delete the fields lat/long in the DB, because the values can be set manually via the Tracking API, many people use and rely on this valuable info (and may have full consent to track locations), so we can keep the fields in the tables.

@oschwald

This comment has been minimized.

Show comment
Hide comment
@oschwald

oschwald Apr 11, 2018

MaxMind is no longer discontinuing coordinates in GeoLite2:

After careful consideration, taking into account customer feedback, we have decided against removing latitude and longitude coordinates from the GeoLite2 databases. We are in the process of reviewing coordinates used in all of our GeoLite2 and GeoIP databases to ensure there is no risk of misuse.

oschwald commented Apr 11, 2018

MaxMind is no longer discontinuing coordinates in GeoLite2:

After careful consideration, taking into account customer feedback, we have decided against removing latitude and longitude coordinates from the GeoLite2 databases. We are in the process of reviewing coordinates used in all of our GeoLite2 and GeoIP databases to ensure there is no risk of misuse.

@mattab

This comment has been minimized.

Show comment
Hide comment
@mattab

mattab May 2, 2018

Member

Thanks for letting us know @oschwald - closing this as won't fix then.

Member

mattab commented May 2, 2018

Thanks for letting us know @oschwald - closing this as won't fix then.

@mattab mattab closed this May 2, 2018

@mattab mattab added the wontfix label May 2, 2018

@mattab mattab modified the milestones: 4.0.0, 3.5.0 May 2, 2018

mattab added a commit that referenced this issue May 2, 2018

sgiehl added a commit that referenced this issue May 3, 2018

sgiehl added a commit that referenced this issue May 3, 2018

mattab added a commit that referenced this issue May 4, 2018

GeoIP2 implementation as a plugin (#12699)
* require geoip2/geoip2 composer package

* Determine region name based on Location Provider

* Adds empty GeoIp2 plugin

* move location_region column definition to GeoIp2 plugin
so it's get changed as soon as the plugin is activated

* Adds GeoIP2 location providers

* ignore GeoIP2 mmdb files

* Adds script to generate GeoIP2 test databases

* Adds Command to convert region codes from FIPS to ISO for old log table records

* Adds GeoIP2 AutoUpdater

* Use GeoIP2 in tests

* update test files

* code fixes

* adds tests

* rename old GeoIP providers to Legacy

* Let GeoIP autoupdater UI handle GeoIp2 as well

* convert region codes to ISO in API after switch to GeoIP2

* do not show GeoIP providers if GeoIP2 plugin is enabled an no GeoIP Legacy provider is still in use

* small fixes

* review changes

* Use correct region translations

* Show correct message if no database can be found

* if log tables have been converted, use archive date to check if region codes still need to be converted to iso

* fix tests

* Improves extracting GeoIP2 databases

* Adjust GeoLocation diagnostics

* readds old taiwan fixes

* Assume all third party location providers as 'recommended'

* Download database over HTTPS

* remove outdated comment (see #12411)

* Remove indication that Geoip2 may be slow, since we found it should be quite fast and this should not be an issue in theory

* skip detection if IP is empty & do not try convert IP to IPv4

* remove downloaded file if an error occurs while extracting

* command should be runnable multiple times

* use ISO codes for suggested region codes

* reload after wizard success

* Drop table if exists.

* Fix two translation keys.

* add special region handling for UK

* update system test

* update UI files

* submodule update

* update test files

InfinityVoid added a commit to InfinityVoid/matomo that referenced this issue Oct 11, 2018

GeoIP2 implementation as a plugin (matomo-org#12699)
* require geoip2/geoip2 composer package

* Determine region name based on Location Provider

* Adds empty GeoIp2 plugin

* move location_region column definition to GeoIp2 plugin
so it's get changed as soon as the plugin is activated

* Adds GeoIP2 location providers

* ignore GeoIP2 mmdb files

* Adds script to generate GeoIP2 test databases

* Adds Command to convert region codes from FIPS to ISO for old log table records

* Adds GeoIP2 AutoUpdater

* Use GeoIP2 in tests

* update test files

* code fixes

* adds tests

* rename old GeoIP providers to Legacy

* Let GeoIP autoupdater UI handle GeoIp2 as well

* convert region codes to ISO in API after switch to GeoIP2

* do not show GeoIP providers if GeoIP2 plugin is enabled an no GeoIP Legacy provider is still in use

* small fixes

* review changes

* Use correct region translations

* Show correct message if no database can be found

* if log tables have been converted, use archive date to check if region codes still need to be converted to iso

* fix tests

* Improves extracting GeoIP2 databases

* Adjust GeoLocation diagnostics

* readds old taiwan fixes

* Assume all third party location providers as 'recommended'

* Download database over HTTPS

* remove outdated comment (see matomo-org#12411)

* Remove indication that Geoip2 may be slow, since we found it should be quite fast and this should not be an issue in theory

* skip detection if IP is empty & do not try convert IP to IPv4

* remove downloaded file if an error occurs while extracting

* command should be runnable multiple times

* use ISO codes for suggested region codes

* reload after wizard success

* Drop table if exists.

* Fix two translation keys.

* add special region handling for UK

* update system test

* update UI files

* submodule update

* update test files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment