Change logging for HttpHook to debug #28911
Conversation
airflow/providers/http/hooks/http.py
Outdated
| self.log.info("Sending '%s' to url: %s", self.method, url) | ||
|
|
||
| log_request = request_kwargs.pop('log_request', True) | ||
| if log_request: | ||
| self.log.info("Sending '%s' to url: %s", self.method, url) |
There was a problem hiding this comment.
IMHO, better option change severity to debug.
log_request - neither documented and tested here
There was a problem hiding this comment.
I made it a debug log and then toied to test with the current changes for log_request. I keep encountering this error while loading the testing DAG:
filepath | error │e this message.[0m (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
===========================+===================================================================================│ ____________ _____________
/files/dags/example_api.py | Traceback (most recent call last): │ ____ |__( )_________ __/__ /________ __
| File "/opt/airflow/airflow/models/baseoperator.py", line 397, in apply_defaults │____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
| result = func(self, **kwargs, default_args=default_args) │___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
| File "/opt/airflow/airflow/models/baseoperator.py", line 749, in __init__ │ _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
| raise AirflowException( │[2023-01-13T08:43:32.749+0000] {triggerer_job.py:101} INFO - Starting the triggerer
| airflow.exceptions.AirflowException: Invalid arguments were passed to │
| SimpleHttpOperator (task_id: get_posts). Invalid arguments were: │
| **kwargs: {'log_request': False}
The DAG I am using is attached to this thread (changed to txt to be able to upload here):
example_api.txt
There was a problem hiding this comment.
You need to control logging by setup correct LOGGING_LEVEL (severity) rather then add additional parameters.
If you think that HttpHook info produce to much logs you need to change only self.log.info to self.log.debug
For additional logging control you could specify you own logging.config, see: https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html#advanced-configuration
There was a problem hiding this comment.
Makes sense to have that log in debug logs. Made that change only. It is a simpler change in terms of not introducing an unwanted parameter
Don't try mount .flake8 anymore, it has been removed in apache#28893
amoghrajesh
requested review from
potiuk,
ashb,
mik-laj and
jedcunningham
as code owners
amoghrajesh
requested review from
Taragolis
and removed request for
ashb,
potiuk,
mik-laj and
jedcunningham
potiuk
changed the title
In tasks that perform multiple requests, the log file is getting cluttered by additional info logs in the http hook
This PR adds a kwarg that controls this logs behaviour
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.