Smart Sensors - Inconsistent Logging

[anthonyp97]

Apache Airflow version: 2.0.1

Kubernetes version (if you are using kubernetes) (use kubectl version): N/A

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Ubuntu 18.04.2 LTS
• Kernel (e.g. uname -a): GNU/Linux 4.15.0-1043-aws x86_64
• Others: LocalExecutor with a PARALLELISM of 32, smart sensors enabled using 2 shards.

What happened:

The logs for the 2 smart sensor tasks that we run always show: [2021-03-16 21:07:45,415] {smart_sensor.py:373} INFO - Loaded 0 sensor_works. However, I can confirm that our FTP sensors are getting registered properly in the smart sensors in the FTP sensor logs. Strangely, the logs for a certain FTP sensor will occasionally show {smart_sensor.py:373} INFO - 4 tasks detected. and then the poke information for these 4 sensors appear in this FTP sensor's log, so it looks like the logs are going to the wrong location.

What you expected to happen:

The logs in the smart sensor tasks themselves should show the number of sensor_works loaded in each smart sensor shard (it should not always be 0), and this information should not be in a random FTP sensor's logs. Also the logs for a specific FTP sensor should not include logs from different sensors.

How to reproduce it:

Run a LocalExecutor with several sensors running in parallel, and 2+ shards enabled for the smart sensors. The logs in smart_sensor_group_shard_0, smart_sensor_group_shard_1 etc. should always show Loaded 0 sensor_works when that is not what is expected. You may be able to find the sensor_works loaded information in the logs of one of the sensors in your DAG.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[vikramkoka]

Thank you for reporting this.

@YingboWang @KevinYang21 any thoughts on this?

[YingboWang]

@anthonyp97 Thank you for reporting this.
In our cluster the smart sensor logging works fine. Look to me it may be related to the logging configuration setting.
The smart sensor task has two categories of log:

1. The task log for itself (similar to general task logs).
2. Cached handlers for all registered and sensing sensors. The logger of second group is initiated by SensorWork._get_sensor_logger() which copy the logic of airflow tasks logger and set context based on each specific sensor task. These loggers simulate the behavior of task log to make airflow UI able to retrieve log for sensors without changing the logic.
   Are you using the default airflow logging setup?

[github-actions]

This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

[github-actions]

This issue has been closed because it has not received response from the issue author.