Add robots.txt and X-Robots-Tag header

[thejens]

Added a robots.txt file and X-Robots-Tag header to limit some of the damage caused when accidentally exposing airflow to the public internet.

[thejens]

... could use some help with where and how to add tests to this as I didn't find that setup intuitive

[potiuk]

Cool. Thanks for that ! The easiest way to add tests is to add separate package for /robots endpoint + check in the home test view if the header is returned : https://github.com/apache/airflow/blob/main/tests/www/views/test_views_home.py

The setup is rather easy. For those tests you can setup local virtualenv (https://github.com/apache/airflow/blob/main/LOCAL_VIRTUALENV.rst) - classic python virtualenv and run pytest tests there (make sure to init the db before or run the test once with --with-db-init custom flag.

You can also setup BREEZE with (https://github.com/apache/airflow/blob/main/BREEZE.rst) - thi s more compleete environment whicch is docker -compose based and is the exact replica of what happens in hthe CI. This is as simple as running ./breeze and the script should guide you and eventually you should be dropped tinto basjh interpreter inside breeze container with everyything read y to run pytest with the test to run (your sources will be mounted in container so you can edit them locally and run in container immediately) . Also recommend to install pre-commit (pre-commit install) in your repo, so that all static checks will be run at commit time.

[thejens]

@potiuk added tests.

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.