excessive polling for geerlinguy.node_exporter can trigger Github ratelimits

[hcoyote]

Looks like geerlingguy.node_exporter may somehow trigger rate limiting on Github's side because the module checks for the latest version each time ansible runs. We need to figure out if there's a way to disable this limiting the frequency of checking.

TASK [geerlingguy.node_exporter : Check current node_exporter version.] *********************************************************************************************************************************************
task path: /Users/hcoyote/.ansible/roles/geerlingguy.node_exporter/tasks/main.yml:2
ok: [XX.YY.138.212] => {"changed": false, "cmd": ["/usr/local/bin/node_exporter", "--version"], "delta": "0:00:00.007202", "end": "2023-01-31 01:48:20.308344", "failed_when_result": false, "msg": "", "rc": 0, "start": "2023-01-31 01:48:20.301142", "stderr": "", "stderr_lines": [], "stdout": "node_exporter, version 1.5.0 (branch: HEAD, revision: 1b48970ffcf5630534fb00bb0687d73c66d1c959)\n  build user:       root@6e7732a7b81b\n  build date:       20221129-18:59:09\n  go version:       go1.19.3\n  platform:         linux/amd64", "stdout_lines": ["node_exporter, version 1.5.0 (branch: HEAD, revision: 1b48970ffcf5630534fb00bb0687d73c66d1c959)", "  build user:       root@6e7732a7b81b", "  build date:       20221129-18:59:09", "  go version:       go1.19.3", "  platform:         linux/amd64"]}
ok: [XX.YY.97.33] => {"changed": false, "cmd": ["/usr/local/bin/node_exporter", "--version"], "delta": "0:00:00.007200", "end": "2023-01-31 01:48:20.685369", "failed_when_result": false, "msg": "", "rc": 0, "start": "2023-01-31 01:48:20.678169", "stderr": "", "stderr_lines": [], "stdout": "node_exporter, version 1.5.0 (branch: HEAD, revision: 1b48970ffcf5630534fb00bb0687d73c66d1c959)\n  build user:       root@6e7732a7b81b\n  build date:       20221129-18:59:09\n  go version:       go1.19.3\n  platform:         linux/amd64", "stdout_lines": ["node_exporter, version 1.5.0 (branch: HEAD, revision: 1b48970ffcf5630534fb00bb0687d73c66d1c959)", "  build user:       root@6e7732a7b81b", "  build date:       20221129-18:59:09", "  go version:       go1.19.3", "  platform:         linux/amd64"]}
ok: [XX.YY.156.16] => {"changed": false, "cmd": ["/usr/local/bin/node_exporter", "--version"], "delta": "0:00:00.007467", "end": "2023-01-31 01:48:20.889350", "failed_when_result": false, "msg": "", "rc": 0, "start": "2023-01-31 01:48:20.881883", "stderr": "", "stderr_lines": [], "stdout": "node_exporter, version 1.5.0 (branch: HEAD, revision: 1b48970ffcf5630534fb00bb0687d73c66d1c959)\n  build user:       root@6e7732a7b81b\n  build date:       20221129-18:59:09\n  go version:       go1.19.3\n  platform:         linux/amd64", "stdout_lines": ["node_exporter, version 1.5.0 (branch: HEAD, revision: 1b48970ffcf5630534fb00bb0687d73c66d1c959)", "  build user:       root@6e7732a7b81b", "  build date:       20221129-18:59:09", "  go version:       go1.19.3", "  platform:         linux/amd64"]}

TASK [geerlingguy.node_exporter : Configure latest version] *********************************************************************************************************************************************************
task path: /Users/hcoyote/.ansible/roles/geerlingguy.node_exporter/tasks/main.yml:8
included: /Users/hcoyote/.ansible/roles/geerlingguy.node_exporter/tasks/config-version.yaml for XX.YY.156.16, XX.YY.97.33, XX.YY.138.212
FAILED - RETRYING: [XX.YY.156.16 -> localhost]: Determine latest GitHub release (local) (5 retries left).
FAILED - RETRYING: [XX.YY.97.33 -> localhost]: Determine latest GitHub release (local) (5 retries left).
FAILED - RETRYING: [XX.YY.138.212 -> localhost]: Determine latest GitHub release (local) (5 retries left).
FAILED - RETRYING: [XX.YY.156.16 -> localhost]: Determine latest GitHub release (local) (4 retries left).
FAILED - RETRYING: [XX.YY.138.212 -> localhost]: Determine latest GitHub release (local) (4 retries left).
FAILED - RETRYING: [XX.YY.97.33 -> localhost]: Determine latest GitHub release (local) (4 retries left).
FAILED - RETRYING: [XX.YY.97.33 -> localhost]: Determine latest GitHub release (local) (3 retries left).
FAILED - RETRYING: [XX.YY.138.212 -> localhost]: Determine latest GitHub release (local) (3 retries left).
FAILED - RETRYING: [XX.YY.156.16 -> localhost]: Determine latest GitHub release (local) (3 retries left).
FAILED - RETRYING: [XX.YY.97.33 -> localhost]: Determine latest GitHub release (local) (2 retries left).
FAILED - RETRYING: [XX.YY.138.212 -> localhost]: Determine latest GitHub release (local) (2 retries left).
FAILED - RETRYING: [XX.YY.156.16 -> localhost]: Determine latest GitHub release (local) (2 retries left).
FAILED - RETRYING: [XX.YY.138.212 -> localhost]: Determine latest GitHub release (local) (1 retries left).
FAILED - RETRYING: [XX.YY.97.33 -> localhost]: Determine latest GitHub release (local) (1 retries left).
FAILED - RETRYING: [XX.YY.156.16 -> localhost]: Determine latest GitHub release (local) (1 retries left).

TASK [geerlingguy.node_exporter : Determine latest GitHub release (local)] ******************************************************************************************************************************************
task path: /Users/hcoyote/.ansible/roles/geerlingguy.node_exporter/tasks/config-version.yaml:2
fatal: [XX.YY.97.33 -> localhost]: FAILED! => {"access_control_allow_origin": "*", "access_control_expose_headers": "ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-RateLimit-Used, X-RateLimit-Resource, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset", "attempts": 5, "changed": false, "connection": "close", "content_length": "279", "content_security_policy": "default-src 'none'; style-src 'unsafe-inline'", "content_type": "application/json; charset=utf-8", "date": "Tue, 31 Jan 2023 01:48:49 GMT", "elapsed": 0, "json": {"documentation_url": "https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting", "message": "API rate limit exceeded for XX.YY.27.195. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)"}, "msg": "Status code was 403 and not [200]: HTTP Error 403: rate limit exceeded", "redirected": false, "referrer_policy": "origin-when-cross-origin, strict-origin-when-cross-origin", "server": "Varnish", "status": 403, "strict_transport_security": "max-age=31536000; includeSubdomains; preload", "url": "https://api.github.com/repos/prometheus/node_exporter/releases/latest", "x_content_type_options": "nosniff", "x_frame_options": "deny", "x_github_media_type": "github.v3; format=json", "x_github_request_id": "REDACTED", "x_ratelimit_limit": "60", "x_ratelimit_remaining": "0", "x_ratelimit_reset": "1675129929", "x_ratelimit_resource": "core", "x_ratelimit_used": "60", "x_xss_protection": "1; mode=block"}
fatal: [XX.YY.138.212 -> localhost]: FAILED! => {"access_control_allow_origin": "*", "access_control_expose_headers": "ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-RateLimit-Used, X-RateLimit-Resource, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset", "attempts": 5, "changed": false, "connection": "close", "content_length": "279", "content_security_policy": "default-src 'none'; style-src 'unsafe-inline'", "content_type": "application/json; charset=utf-8", "date": "Tue, 31 Jan 2023 01:48:49 GMT", "elapsed": 0, "json": {"documentation_url": "https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting", "message": "API rate limit exceeded for XX.YY.27.195. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)"}, "msg": "Status code was 403 and not [200]: HTTP Error 403: rate limit exceeded", "redirected": false, "referrer_policy": "origin-when-cross-origin, strict-origin-when-cross-origin", "server": "Varnish", "status": 403, "strict_transport_security": "max-age=31536000; includeSubdomains; preload", "url": "https://api.github.com/repos/prometheus/node_exporter/releases/latest", "x_content_type_options": "nosniff", "x_frame_options": "deny", "x_github_media_type": "github.v3; format=json", "x_github_request_id": "REDACTED", "x_ratelimit_limit": "60", "x_ratelimit_remaining": "0", "x_ratelimit_reset": "1675129929", "x_ratelimit_resource": "core", "x_ratelimit_used": "60", "x_xss_protection": "1; mode=block"}
fatal: [XX.YY.156.16 -> localhost]: FAILED! => {"access_control_allow_origin": "*", "access_control_expose_headers": "ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-RateLimit-Used, X-RateLimit-Resource, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset", "attempts": 5, "changed": false, "connection": "close", "content_length": "279", "content_security_policy": "default-src 'none'; style-src 'unsafe-inline'", "content_type": "application/json; charset=utf-8", "date": "Tue, 31 Jan 2023 01:48:49 GMT", "elapsed": 0, "json": {"documentation_url": "https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting", "message": "API rate limit exceeded for XX.YY.27.195. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)"}, "msg": "Status code was 403 and not [200]: HTTP Error 403: rate limit exceeded", "redirected": false, "referrer_policy": "origin-when-cross-origin, strict-origin-when-cross-origin", "server": "Varnish", "status": 403, "strict_transport_security": "max-age=31536000; includeSubdomains; preload", "url": "https://api.github.com/repos/prometheus/node_exporter/releases/latest", "x_content_type_options": "nosniff", "x_frame_options": "deny", "x_github_media_type": "github.v3; format=json", "x_github_request_id": "REDACTED", "x_ratelimit_limit": "60", "x_ratelimit_remaining": "0", "x_ratelimit_reset": "1675129929", "x_ratelimit_resource": "core", "x_ratelimit_used": "60", "x_xss_protection": "1; mode=block"}

[hcoyote]

Looks like if node_exporter_version is undefined or latest, it will trigger the attempt to download the latest, so we should probably pin this to a particular version of prometheus/node_exporter. 1.5.0 seems to be the latest.

[tmgstevens]

Added a PR to fix upstream: geerlingguy/ansible-role-node_exporter#21

[gene-redpanda]

I'm not currently seeing this behavior @hcoyote @tmgstevens. If this is still an issue LMK, otherwise I will close this issue out.

[hcoyote]

it'll happen if you repeatedly run ansible in a short period of time. I've hit it pretty frequently.

[gene-redpanda]

I've yet to replicate this issue while running ansible pretty frequently, including in CI testing where multiple clusters are being built simultaneously. What level of frequency are we talking?

[hcoyote]

rate limits are low for non-logged in operations. rate limits may also be different if you're operating from within githubs infrastructure, or in relation to CI things. I don't know. But I know that if I curl from my ip address more than 60 times in an hour then the API rate limit gets triggered. And that's not that hard to trigger with repeated ansible runs from multiple hosts, each running multiple curl operations throughout a run.

x-ratelimit-limit: 60
x-ratelimit-remaining: 54
x-ratelimit-reset: 1686610687
x-ratelimit-resource: core
x-ratelimit-used: 6