Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
[Detached] Make node-geolite2 a redistribution of MaxMind's databases #19
Following the discussion in #17.
Roadmap before mergeable:
I'll keep working on this but I wanted to share my progress so far and get @runk's suggestions if any.
So I implemented a self-updating mechanism to automatically update the local databases copies from the git repo. It uses SHA-384 checksums to determine whether a newer version is available. It's capable of keeping database copies up to date transparently, even when the library is used in a long-running script like a server or something, which should help ensure that nobody keeps using old versions of the databases past the allowed upgrade period of 30 days (see Do Not Sell requests and relevant sections in MaxMind's EULA).
I had to make a major API change however: the
var geolite2 = require('geolite2'); var maxmind = require('maxmind'); var lookup = maxmind.openSync(geolite2.paths.city); // or geolite2.paths.country or geolite2.paths.asn var city = lookup.get('126.96.36.199');
Proposed API (this PR):
var geolite2 = require('geolite2'); var lookup = geolite2.openSync('GeoLite2-City'); // or -Country or -ASN // an asynchronous geolite2.open() method is also available and returns a Promise var city = lookup.get('188.8.131.52');
Implementation detailsgeolite2 returns a Proxy wrapper around a maxmind Reader instance. When a property of the reader is accessed, a background routine starts (either once every 48 hours or the first time the reader is accessed) to fetch the latest databases checksums from the git mirror.
If any of the checksums differ from the local copy, fresh copies of the databases are downloaded, and their checksums matched again. Normally they should now be correct, so new maxmind Reader instances are spun up, loading up in memory the updated data, and they replace the old ones.
If the checksums don't match after a fresh download, or for any reason anything fucks up, a warning is printed to stdout and the whole self-update shebang will run again with the usual trigger (48 hours or program restart).
CI is broken again because the tests are now irrelevant and need to be rewritten.
@runk I'm going to need your guidance/help for this part; as this involves maintainer secrets and CI configuration that I cannot change within the scope of a PR.
Once this is done we just have to verify the redistribution terms with MaxMind and, assuming all parties involved are OK, this will be mergeable.
Thanks for taking the time to review this - lots of new code indeed.
I'll write a bash script to be run as a cron job. Need to look into how to push from CI.
Update script written. Ci-independent for now, I'll add that part later on.
Indeed, which is why I reused that env var in the bash script.
In other news, I have written a draft mail for MM's legal, if you're OK with that I'd like to send it to you first so you can review it.
I'm the maintainer of an open source project that also used to directly download these databases on install (directly from Maxmind, not via this repo). I would love to hear what the response from the legal team is as this is something my project will also have to deal with.
Hey @GitSquared, it was in trash - just found it.
After reading and looking at all the code I still feel a little uncomfortable redistributing databases like that. I really appreciate all the work you've done, however I don't think I'm feeling like accepting this change in the current repository. Whole legal deal makes me feel too uncomfortable.
That said, would you mind cloning/copying this repo and doing exactly what you want? I'm more than happy to add a reference to your repo from this one - no questions at all.
GMail hates personal mailservers. I'd appreciate if you could report it as "not spam". Thanks.
I understand your decision, thanks for reviewing this PR either way.
I know you're not a stranger to them - they reuse your reader lib in their official APIs - so I had wished that with you as a maintainer they would be more likely to try and find a way for the license to work. Guess it's time to test my street cred.
Edit: Revised email sent to MM. (ping @d1str0).
I got an answer from MaxMind.
They provided a helpful link to a page deep inside their support Q&A along with the following statement:
They kept themselves in the clear by adding that they "are not in a position to give legal advice" - as far as I'm involved this is just lawyer speak to avoid any troubles.
So tl;dr they cool with it, especially since it self-updates.
I'll detach & publish my fork in the next few days.