feat: updated dataset (2020-10-13) #80
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This updates the IPv4 City geoip dataset to the latest version provided on https://dev.maxmind.com/geoip/geoip2/geolite2/ The b-tree format and lookup logic remain unchanged, however the code responsible for building the b-tree had to be changed to work with the new CSV format of the source dataset. Closes #68 Closes #63 BREAKING CHANGE: `area_code` and `metro_code` are no longer provided due to upstream changes License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>
lidel
requested review from
jessicaschilling,
hacdias,
rafaelramalho19,
Gozala and
dignifiedquire
lidel
changed the title
jessicaschilling
approved these changes
Oct 19, 2020
rafaelramalho19
approved these changes
Oct 19, 2020
| // |- locationsCsv | ||
| // |- blocksCsv | ||
| const DATA_HASH = 'bafybeid3munsqqt36qhoumn3kvgwmft6dsswzgl3wiohsanlyqemczcsvi' // GeoLite2-City-CSV_20201013 | ||
| const locationsCsv = 'GeoLite2-City-Locations-en.csv' |
There was a problem hiding this comment.
❓ Are we going to use other languages in the near future? I see that we have available de, en, es, fr, ja, pt-BR, ru and zh-CN
There was a problem hiding this comment.
Yes, this is certainly a possibility now, but not in the scope of this PR.
Created #81 to track that.
Gozala
approved these changes
Oct 20, 2020
| geonameData.push( | ||
| String(row.postal_code), | ||
| Number(row.latitude), | ||
| Number(row.longitude) |
There was a problem hiding this comment.
📝Last two could end up as NaNs if inputs aren't valid numbers. If that is not valid output it would be worth to uphold invariant here.
hacdias
approved these changes
Oct 20, 2020
Co-authored-by: Irakli Gozalishvili <contact@gozala.io> License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>
lidel
force-pushed
the
feat/GeoLite2-City-CSV_20201013
branch
from
f628658
to
a1eba19
Compare
lidel
added a commit
to ipfs/public-gateway-checker
that referenced
this pull request
Oct 20, 2020
Details: ipfs-shipyard/ipfs-geoip#80 License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>
lidel
added a commit
to ipfs/ipfs-webui
that referenced
this pull request
Oct 21, 2020
lidel
added a commit
to ipfs/ipfs-webui
that referenced
this pull request
Oct 21, 2020
Closes #1249 See ipfs-shipyard/ipfs-geoip#80 for details
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR updates the IPv4 City geoip dataset to the latest free version provided on https://dev.maxmind.com/geoip/geoip2/geolite2/ (
GeoLite2-City-CSV_20201013)Closes #68
Closes #63
Changes
npm run generateto be compatible with new input format without changing output format too mucharea_codeandmetro_codeare no longer provided due to upstream changesGeoLite2-City-CSV_20201013datasetbut I've kept our overrides just to protect against future regressions.
Not in this PR
ipfs.objectAPI instead ofipfs.daganddag-cborTest/Preview: ipfs-geoip on WebUI's Peers screen
I've run this PR against ipfs-webui and Peers screen looks good.
My node is located in Poland. I no longer see "USA" nodes with 30ms ping – faster-than-light networking is no more ;)
With this updated dataset, we now see much better distribution across all continents: