feat: Add robots.txt to site

[imtherealnaska]

Which issue does this PR close?

• Closes Improve search engine / agent discoverability for DataFusion website #23258

What changes are included in this PR?

There are 2 PR's for this issue. This one contains the changes for robots.txt

• added the robots.txt , added license header and allow rules.

Are these changes tested?

Yes. Tested by building the docs and verifying the files getting generated in the build/

Are there any user-facing changes?

No.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.23%. Comparing base (110a90a) to head (9c21847).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24333      +/-   ##
==========================================
- Coverage   81.23%   81.23%   -0.01%     
==========================================
  Files        1111     1111              
  Lines      390208   390208              
  Branches   390208   390208              
==========================================
- Hits       316991   316983       -8     
- Misses      54592    54594       +2     
- Partials    18625    18631       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Jefffrey]

dependent on #24332 since i think we want to include the sitemap as part of the robots.txt

[Jefffrey]

not sure if we're supposed to have this license txt in a robots.txt?

[imtherealnaska]

I added this because , '#' will be ignored by crawlers also I had to add the robots.txt file name to rat_exclude_ files.txt. Im happy to remove it , fine either way .

[Jefffrey]

for reference, airflow and spark dont include the license

• https://airflow.apache.org/robots.txt
• https://spark.apache.org/robots.txt

[imtherealnaska]

Thanks. I have added a commit removing license and following what llms.txt followed.

[Jefffrey]

live now: https://datafusion.apache.org/robots.txt

thanks @imtherealnaska

[imtherealnaska]

wooohooo. first commit . @Jefffrey Thanks for the help.