-
Notifications
You must be signed in to change notification settings - Fork 8.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add periodic updates of maxmind databases #10983
base: main
Are you sure you want to change the base?
Conversation
✅ Deploy Preview for kubernetes-ingress-nginx canceled.
|
This issue is currently awaiting triage. If Ingress contributors determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Welcome @moleus! |
Hi @moleus. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Moleus The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@@ -81,6 +103,17 @@ func GeoLite2DBExists() bool { | |||
return true | |||
} | |||
|
|||
// DownloadGeoLite2DBPeriodically starts a goroutine to periodically sync the GeoIP databases | |||
func DownloadGeoLite2DBPeriodically(syncPeriod time.Duration) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought Maxmind did this itself, which is why it needed curl; we discovered this when we tried to remove Curl and found it was dependent on Maxmind; it setups a cronjob to download.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apologies for the delayed response, I was on holiday. I've tried to look for cronjobs or any other Maxmind related references in this repository, but couldn't find any curl/cronjob used by Maxmind. As far as I understand the only file which downloads db is internal/nginx/maxmind.go. Could you send me the link where Maxmind setups a cronjob to download a db?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@strongjz Hey! Are there any actions I can do to help with review of this PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rpkatz found it when we were trying to remove curl from the container image.
https://alpine.pkgs.org/3.17/alpine-main-aarch64/libmaxminddb-1.7.1-r0.apk.html
One of the deps is curl.
@@ -43,6 +43,9 @@ They are set in the container spec of the `ingress-nginx-controller` Deployment | |||
| `--maxmind-retries-count` | Number of attempts to download the GeoIP DB. (default 1) | | |||
| `--maxmind-license-key` | Maxmind license key to download GeoLite2 Databases. https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases . | | |||
| `--maxmind-mirror` | Maxmind mirror url (example: http://geoip.local/databases. | | |||
| `--maxmind-enable-sync` | Enable the synchronization of the GeoIP databases. (default false) | | |||
| `--maxmind-sync-period` | Period at which the controller fetches updates for GeoIp Databases (default 24h) | | |||
| `--maxmind-url` | Maxmind url to download GeoLite2 Databases. (default "https://updates.maxmind.com") | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is already --maxmind-mirror
parameter, --maxmind-url
looks like kind of partial duplication. Would it be possible to use only --maxmind-mirror
?
Periodically download new versions of maxmind databases to keep up to date during pod lifetime
What this PR does / why we need it:
Maxmind releases new versions of their databases a few time a week. Currently there is no way to update GeoIp databases after ingress-nginx pod is started. So, after some time GeoIp informationon starts being outdated and even start conflicting with other instances of ingress-nginx, which started later and have newer version of databases.
This PR adds functionality for periodic database updates. This feature is disabled by default.
Any suggestions or improvements are welcome!
Notes
download.maxmind.com
toupdates.maxmind.com
. Like in geoipupdate we use md5 hash to check for new version and skip the download if there are no updates.Types of changes
How Has This Been Tested?
I've mocked maxmind API in maxmind_test.go, test cases are taken from geoipupdate source code. This change only affects how maxmind database is fetched.
Checklist: