Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download script will no longer work #17

Open
2shortplanks opened this issue Dec 20, 2019 · 21 comments · May be fixed by #19
Open

Download script will no longer work #17

2shortplanks opened this issue Dec 20, 2019 · 21 comments · May be fixed by #19

Comments

@2shortplanks
Copy link

@2shortplanks 2shortplanks commented Dec 20, 2019

Due to upcoming data privacy regulations, MaxMind are making significant changes to how users access free MaxMind GeoLite2 databases starting December 30, 2019. The databases will continue to be available without charge and for redistribution. However, you will be required to create an account and use a license key to download the databases, and agree to a new EULA that addresses applicable data privacy regulations.

Learn more on the MaxMind blog: https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/.

Mark Fowler
MaxMind

@runk

This comment has been minimized.

Copy link
Owner

@runk runk commented Dec 29, 2019

@2shortplanks thanks for raising it!

Would it require each user who downloads the package to have their own license key or I can use shared one?

@2shortplanks

This comment has been minimized.

Copy link
Author

@2shortplanks 2shortplanks commented Dec 30, 2019

@runk Either each user can create a MaxMind account and get their own license key, or you can re-distribute the databases under the terms of our license. In the latter case, you would need to be able to pass along Do Not Sell requests to each of those you'd redistributed the database to along with abiding by the other terms of the agreement. We would recommend having each user obtain their own license key.

@AnandChowdhary

This comment has been minimized.

Copy link

@AnandChowdhary AnandChowdhary commented Dec 30, 2019

Seems like this package is no longer installing. Is this related to this issue?

Error: getaddrinfo ENOTFOUND geolite.maxmind.com
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Emitted 'error' event on ClientRequest instance at:
    at TLSSocket.socketErrorListener (_http_client.js:423:9)
    at TLSSocket.emit (events.js:305:20)
    at emitErrorNT (internal/streams/destroy.js:84:8)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  errno: -3008,
  code: 'ENOTFOUND',
AnandChowdhary added a commit to staart/api that referenced this issue Dec 30, 2019
@AnandChowdhary

This comment has been minimized.

Copy link

@AnandChowdhary AnandChowdhary commented Dec 30, 2019

Turns out the download scripts (e.g., https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz) are no longer operational. You now need to login to your account to download the .mmdb file using a token and they don't have publicly accessible URLs.

The new license the database is under is GeoLite2 End User License Agreement which is summarized below (please note that I am not a lawyer and this is not legal advise, only my understanding of the license):

  • You still need to give Creative Commons-like attribution
  • You cannot use the data for Fair Credit Reporting Act (FCRA) purposes
  • You cannot use the data identify a specific person or address
  • You need to update the database as soon as a new one comes out, and stop using the older version within 30 days of update

This seems like we are still allowed to create a public mirror, as long as we keep updating it. My team has started using Git LFS for a local database copy, which is something that perhaps this package can do too, and then users can download the database from the https://raw.githubusercontent.com... URL?

The only other option is making people login and download the package and use the maxmind package directly, making this one redundant.

@GitSquared

This comment has been minimized.

Copy link
Contributor

@GitSquared GitSquared commented Dec 31, 2019

You need to update the database as soon as a new one comes out, and stop using the older version within 30 days of update

This one is going to be hard to enforce if a Git LFS mirror is setup in this package. How frequent have these updates been so far? Does @runk have a way to get notified when a new one comes out?

you would need to be able to pass along Do Not Sell requests to each of those you'd redistributed the database to

Would changing the license to prevent commercial use be enough? It would be great to warn dependents of this package to ensure they're aware of the new limitations.

@Findus23

This comment has been minimized.

Copy link

@Findus23 Findus23 commented Dec 31, 2019

How frequent have these updates been so far?

The last two versions were uploaded on 17 Dec 2019 and 24 Dec 2019 according to their Last-Modified Header, so I'm assuming they are updated weekly.

@runk

This comment has been minimized.

Copy link
Owner

@runk runk commented Jan 1, 2020

I really want to avoid this package hiding the end user license. Indeed it's going to make it less convenient for npm users, but it is what it is. You either download it manually and manage ad-hoc or setup env var with license and continue using this package.

@runk

This comment has been minimized.

Copy link
Owner

@runk runk commented Jan 1, 2020

#18

@runk

This comment has been minimized.

Copy link
Owner

@runk runk commented Jan 1, 2020

geolite2@3.0.0 is out, give it a go guys

@GitSquared

This comment has been minimized.

Copy link
Contributor

@GitSquared GitSquared commented Jan 2, 2020

While I agree with your solution, requiring each individual user to sign up for and accept an EULA is not doable in my case - nor in any public open-source project of any significant size, I believe.

Either each user can create a MaxMind account and get their own license key, or you can re-distribute the databases under the terms of our license. In the latter case, you would need to be able to pass along Do Not Sell requests to each of those you'd redistributed the database to along with abiding by the other terms of the agreement.

From what I understand from MaxMind's blog post and a quick reading of the different EULAs, "being able to pass along Do Not Sell requests" is mostly a matter of keeping the redistribution up to date. Turns out they do have more info on how to stay current.

I think it should be possible to create an npm package with a suitable redistribution license, which would fetch an always-updated mirror of the databases (through Git LFS or something else), while displaying clear warning messages in install logs about significant details of MM's EULA - like precision & FCRA limitations.

We could even force the package to invalidate the downloaded DBs after 30 days to ensure that newer versions get pushed out to end-users in compliance with Do Not Sell requests.
(Of course this is not a perfect enforcement system and you can work around it, but if you do, you'll be breaking the different licenses & agreements for both the redistribution and the source.)

@2shortplanks Do you think such a redistribution system is something your company would be OK with?

@runk Would you want me to create a new package or perhaps you'd be okay with updating this one instead?

@runk

This comment has been minimized.

Copy link
Owner

@runk runk commented Jan 3, 2020

I'm happy for you to give it a go - writing licences is clearly not my passion.

Re up to date databases - apparently it's fine to omit version (date) of the database you want to download, which results in latest version:

https://github.com/runk/node-geolite2/blob/master/scripts/postinstall.js#L17

@GitSquared

This comment has been minimized.

Copy link
Contributor

@GitSquared GitSquared commented Jan 3, 2020

If we go forward with this, you will need to manage a MaxMind license in your name that will be used for redistribution - which is also why we need to setup a mirror, otherwise you're not redistributing anything and you can't hide your license.

I will look into the technical aspects of this and try to draft a new license for this package - maybe open a PR.

@GitSquared

This comment has been minimized.

Copy link
Contributor

@GitSquared GitSquared commented Jan 3, 2020

Every account using Git Large File Storage receives 1 GB of free storage and 1 GB a month of free bandwidth. If the bandwidth and storage quotas are not enough, you can choose to purchase an additional quota for Git LFS.
(from GitHub Support)

This will be a problem.

@AnandChowdhary

This comment has been minimized.

Copy link

@AnandChowdhary AnandChowdhary commented Jan 3, 2020

This will be a problem.

That’s right, I already finished my quota in the past 2 days, so I switched back to this package earlier today.

@GitSquared

This comment has been minimized.

Copy link
Contributor

@GitSquared GitSquared commented Jan 3, 2020

Either someone assumes the costs of transfer, we setup a redistributing proxy in front of MM's URLs, or we bundle a URL+License Key combo in this package but then we might lose control of that key. In both of those other solutions MaxMind may block us if we reach a certain # of downloads.

This is starting to sound tricky. Any other ideas?

@AnandChowdhary

This comment has been minimized.

Copy link

@AnandChowdhary AnandChowdhary commented Jan 3, 2020

TBH not using Git LFS but directly putting the tar.gz in this repo might work as well. The file is merely 30 MB compressed, and the limit to GitHub repos is 100 GB storage and no hard bandwidth limit, so it shouldn't be a problem. The package's postinstall script will simply extract the file (which now becomes ~60 MB).

NPM packages also don't have a hard size/bandwidth limit, so publishing/installing shouldn't be bad. Alternately, the database file can be in .npmignore, stay in this repo but not be published to NPM, and then use the https://raw.githubusercontent.com/... URL to download and extract it, like the package currently does. That also doesn't have a hard bandwidth limit, and this package would probably use ~90 GB/month (3000 downloads/month), which is basically nothing.

@GitSquared

This comment has been minimized.

Copy link
Contributor

@GitSquared GitSquared commented Jan 3, 2020

Indeed, if we can keep it way below 100mb compressed, this could be a long-lasting solution.

Alternately, the database file can be in .npmignore, stay in this repo but not be published to NPM, and then use the https://raw.githubusercontent.com/... URL to download and extract it, like the package currently does.

Music to my ears. I'll get back to that PR of mine.

@AnandChowdhary

This comment has been minimized.

Copy link

@AnandChowdhary AnandChowdhary commented Jan 3, 2020

Lastly, we can probably use GitHub Actions to schedule a cron job (see Scheduler) that fetches the database files with the "master" API key, updates the repo (see GitHub Push), and then bumps the patch version of the package and publishes to NPM (see Publish) once every week.

@2shortplanks

This comment has been minimized.

Copy link
Author

@2shortplanks 2shortplanks commented Jan 3, 2020

@GitSquared

@2shortplanks Do you think such a redistribution system is something your company would be OK with?

I'm not the best person at MaxMind to answer questions about redistribution directly, but if you email
legal@maxmind.com they'll be able to help you out

@GitSquared

This comment has been minimized.

Copy link
Contributor

@GitSquared GitSquared commented Jan 3, 2020

@2shortplanks All right, thank you. I will send them an email once I have something more concrete to show them.

@GitSquared GitSquared linked a pull request that will close this issue Jan 3, 2020
8 of 9 tasks complete
@GitSquared

This comment has been minimized.

Copy link
Contributor

@GitSquared GitSquared commented Jan 3, 2020

@runk @AnandChowdhary WIP PR opened at #19

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.