Add recursive flag to glob in filesystem sensor

[ShraddheyaS]

This PR aims to fix #16725 by adding the recursive flag to glob in the filesystem sensor.

closes: #16725

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[ShraddheyaS]

@potiuk should these checks be failing? I didn't think this change would affect integrations with other components.

The exit code is 137, is this due to some memory constraint?

[potiuk]

Yep. We have some instabilities of GitHub runners and memory - constant struggle but seem to intensify a bit recently

[potiuk]

I think, while this change looks initially like it is fully backwards compatible, in fact it is not. If someone had a directory structure in his folder already, this change might detect files which were skipped before.

I think better (and fully backwards-compatible) solution will be to add optional recursive flag with default = False to the sensor constructor. This way you would not even have to add any explanation to UPDATING.md.

[ShraddheyaS]

@potiuk Thanks for the feedback! Let me know if this looks OK now

[kaxil]

An example would be nice too

[ShraddheyaS]

Do you mean an example of invocation like below? Or an example scenario where this would be used like #16725?

FileSensor(
            task_id='test',
            filepath=temp_dir + "/**",
            fs_conn_id='fs_default',
            dag=self.dag,
            timeout=0,
            poke_interval=1,
            recursive=True,
        )

Also, would this example go to the same place? The docstring for the class?

[potiuk]

An example in airflow/example_dags it was missing for filesystem_sensor, but it would be nice to add one since we are changing it.

[kaxil]

Suggested change

	:param recursive: when set True, enables recursive directory matching behavior of
	`**` in glob filepath parameter. Defaults to False.
	:param recursive: when set to ``True``, enables recursive directory matching behavior of
	``**`` in glob filepath parameter. Defaults to ``False``.

[kaxil]

Feel free to do that in a follow-up PR to add an example DAG

Stale Review

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[kaxil]

@potiuk mind taking another look. Since you had requested changes :)

[potiuk]

Fine for me :)

[boring-cyborg]

Awesome work, congrats on your first merged pull request!