Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logging for Rails cache timeouts #21667

Merged
merged 2 commits into from Nov 27, 2022

Conversation

ClearlyClaire
Copy link
Contributor

Currently, Rails cache connection to Redis times out after 20 seconds (for instance because of a firewall misconfiguration), and fails silently. This may cause some Redis-related issues to be difficult to diagnose.

This PR lowers the 20 seconds default to 5 seconds, as 5 seconds is the (default) value we use for other Redis uses, and it changes Rails cache to log errors to the standard Rails logger.

This will likely make the logs very noisy when Redis is timing out:

Nov 26 10:12:03 mastodev bundle[1347]: [ba4ea586-c8f9-41db-84a7-e174ffa473c5] [active_model_serializers] RedisCacheStore: write_entry failed, returned false: Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Redis::TimeoutError)
Nov 26 10:12:08 mastodev bundle[1347]: [ba4ea586-c8f9-41db-84a7-e174ffa473c5] [active_model_serializers] RedisCacheStore: read_entry failed, returned nil: Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Redis::TimeoutError)
Nov 26 10:12:13 mastodev bundle[1347]: [ba4ea586-c8f9-41db-84a7-e174ffa473c5] [active_model_serializers] RedisCacheStore: write_entry failed, returned false: Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Redis::TimeoutError)
Nov 26 10:12:18 mastodev bundle[1347]: [ba4ea586-c8f9-41db-84a7-e174ffa473c5] [active_model_serializers] RedisCacheStore: read_entry failed, returned nil: Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Redis::TimeoutError)
Nov 26 10:12:23 mastodev bundle[1347]: [ba4ea586-c8f9-41db-84a7-e174ffa473c5] [active_model_serializers] RedisCacheStore: write_entry failed, returned false: Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Redis::TimeoutError)
Nov 26 10:12:28 mastodev bundle[1347]: [ba4ea586-c8f9-41db-84a7-e174ffa473c5] [active_model_serializers] RedisCacheStore: read_entry failed, returned nil: Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Redis::TimeoutError)
Nov 26 10:12:33 mastodev bundle[1347]: [ba4ea586-c8f9-41db-84a7-e174ffa473c5] [active_model_serializers] RedisCacheStore: write_entry failed, returned false: Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Redis::TimeoutError)
Nov 26 10:12:38 mastodev bundle[1347]: [ba4ea586-c8f9-41db-84a7-e174ffa473c5] [active_model_serializers] RedisCacheStore: read_entry failed, returned nil: Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Redis::TimeoutError)
Nov 26 10:12:43 mastodev bundle[1347]: [ba4ea586-c8f9-41db-84a7-e174ffa473c5] [active_model_serializers] RedisCacheStore: write_entry failed, returned false: Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Redis::TimeoutError)
Nov 26 10:12:48 mastodev bundle[1347]: [ba4ea586-c8f9-41db-84a7-e174ffa473c5] [active_model_serializers] RedisCacheStore: read_entry failed, returned nil: Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Redis::TimeoutError)
Nov 26 10:12:53 mastodev bundle[1347]: [ba4ea586-c8f9-41db-84a7-e174ffa473c5] [active_model_serializers] RedisCacheStore: write_entry failed, returned false: Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Redis::TimeoutError)
Nov 26 10:12:58 mastodev bundle[1347]: [ba4ea586-c8f9-41db-84a7-e174ffa473c5] RedisCacheStore: read_entry failed, returned nil: Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Redis::TimeoutError)
Nov 26 10:13:03 mastodev bundle[1347]: [ba4ea586-c8f9-41db-84a7-e174ffa473c5] RedisCacheStore: write_entry failed, returned false: Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Redis::TimeoutError)
Nov 26 10:13:08 mastodev bundle[1347]: [ba4ea586-c8f9-41db-84a7-e174ffa473c5] RedisCacheStore: read_entry failed, returned nil: Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Redis::TimeoutError)

But it should not increase logs when Redis is running fine and reachable.

@Gargron Gargron merged commit d587a26 into mastodon:main Nov 27, 2022
@ineffyble ineffyble mentioned this pull request Nov 30, 2022
nametoolong pushed a commit to nametoolong/nuage that referenced this pull request Jan 12, 2023
* Reduce redis cache store connect timeout from default 20 seconds to 5 seconds

* Log cache store errors
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants