New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AIRFLOW-6530] Allow Custom Statsd Client #7227
[AIRFLOW-6530] Allow Custom Statsd Client #7227
Conversation
@kaxil thanks for your suggested changes, i've applied them. I'll figure out how to best test these changes today. Any clues? |
Codecov Report
@@ Coverage Diff @@
## master #7227 +/- ##
==========================================
- Coverage 86.83% 86.16% -0.67%
==========================================
Files 897 915 +18
Lines 42751 44164 +1413
==========================================
+ Hits 37121 38055 +934
- Misses 5630 6109 +479
Continue to review full report at Codecov.
|
@envious - You can add another test at https://github.com/apache/airflow/blob/master/tests/test_stats.py by creating a new Statsd client and passing it using airflow/tests/utils/test_email.py Lines 90 to 98 in d7d2794
and asserting that the client is the same that you were expecting. |
Can you say something more about that? Do you mean any public software or services? |
…sd client - Updated metrics.rst to reflect new custom metric capabilities - Updated the default airfow configuration file to reflect new key which can be added in order to utilise custom statsd client configuration
Co-Authored-By: Kaxil Naik <kaxilnaik@gmail.com>
Co-Authored-By: Kaxil Naik <kaxilnaik@gmail.com>
While tests are being worked on for this, do you have any advice on how/if we can get this "backported" to the next 1.10 release? |
marked it for 1.10.10. We will cherry-pick and backport your changes after merging it to master |
1c1f5a1
to
18db742
Compare
18db742
to
b909492
Compare
airflow/config_templates/config.yml
Outdated
Note: The module path must exist on your PYTHONPATH for Airflow to pick it up | ||
for Airflow to pick it up |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Duplicate wording
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, fixed! 😄
@@ -200,10 +200,35 @@ def __init__(self, *args, **kwargs): | |||
self.__class__.instance = DummyStatsLogger() | |||
except (socket.gaierror, ImportError) as e: | |||
log.warning("Could not configure StatsClient: %s, using DummyStatsLogger instead.", e) | |||
self.__class__.instance = DummyStatsLogger() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why this change?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I found a bug when adding tests for the new functionality. The log message above the line I added states that the class used for the statsd client should fall back to DummyStatsLogger
if the defined statsd client class cannot be imported, or cannot connect to a statsd instance, but this never happens, and self.__class__.instance
remains set to the default value of None
For example, if I try to load an invalid custom statsd class using the functionality that @envious added, triggering an ImportError
, when I then try and call Stats.incr()
(which should still work because the underlying class has fallen back to DummyStatsLogger
), I get errors like:
cls = <class 'airflow.stats.Stats'>, name = 'incr'
def __getattr__(cls, name):
> return getattr(cls.instance, name)
E AttributeError: 'NoneType' object has no attribute 'incr'
airflow/stats.py:188: AttributeError
Co-Authored-By: Kaxil Naik <kaxilnaik@gmail.com>
Awesome work, congrats on your first merged pull request! |
Co-authored-by: Usman Arshad <usman.arshad@skyscanner.net> Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com> Co-authored-by: Craig Rosie <craigrosie7@gmail.com> (cherry picked from commit 8bf6a90)
This change allows Airflow users to utilise their own custom Statsd client as discussed here.
Many companies have their own custom Statsd clients and this change should allow for easier adoption of Airflow by large corporations.
Example usage based off of this change:
As such:
statsd_custom_client_path = company.statsdclient.customclient
Issue link: AIRFLOW-6530
Make sure to mark the boxes below before creating PR: [x]
[AIRFLOW-NNNN]
. AIRFLOW-NNNN = JIRA ID** For document-only changes commit message can start with
[AIRFLOW-XXXX]
.In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.