Skip to content

Suppress meaningless opentsdb metrics connection warning#450

Merged
cmccully merged 4 commits intomainfrom
fix/remove-meaningless-opensdb-warning
Apr 7, 2026
Merged

Suppress meaningless opentsdb metrics connection warning#450
cmccully merged 4 commits intomainfrom
fix/remove-meaningless-opensdb-warning

Conversation

@timbeccue
Copy link
Copy Markdown
Contributor

@timbeccue timbeccue commented Mar 26, 2026

Set default OPENTSDB_PYTHON_METRICS_TEST_MODE=1 in banzai/__init__.py so users don't see this confusing error on every run:

ERROR: metric_wrappers: ConnectionError(MaxRetryError("HTTPConnectionPool(host='scheduler-dev.lco.gtn', port=80): Max retries exceeded with url: /api/version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x112013230>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))"))

@timbeccue timbeccue marked this pull request as ready for review March 26, 2026 17:25
@timbeccue timbeccue requested a review from cmccully March 26, 2026 17:30
@timbeccue timbeccue force-pushed the fix/remove-meaningless-opensdb-warning branch 2 times, most recently from cedc21e to 47a7498 Compare April 6, 2026 20:47
@timbeccue
Copy link
Copy Markdown
Contributor Author

The current solution only sets OPENTSDB_PYTHON_METRICS_TEST_MODE if we're not in prod because using any value at all evaluated to True. I made an update to opentsdb-python-metrics so in version 0.3.0, setting that variable to '0' or 'false' correctly sets test mode to off.

We could make this solution cleaner by having banzai's production deployment explicitly configure OPENTSDB_PYTHON_METRICS_TEST_MODE to be 'false', then also safely use a default of 'true' for all other cases.

However, the ocs ingester needs to be updated to use opentsdb-python-metrics 0.3.0 first, and then banzai needs to specify that version of the ingester before we can access this change.

Copy link
Copy Markdown
Collaborator

@cmccully cmccully left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a change log message and bump the last digit of the version. Otherwise looks good.

@cmccully cmccully force-pushed the fix/remove-meaningless-opensdb-warning branch from 47a7498 to a076947 Compare April 7, 2026 17:04
@cmccully cmccully merged commit cd3f01b into main Apr 7, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants