Add generate_llms_txt.py to make llms.txt

[sfc-gh-tteixeira]

📚 Context

llms.txt is an up-and-coming standard that aims to make it easier for LLMs to consume websites. See https://llmstxt.org/ for more.

This PR introduces an llms.txt for Streamlit's docs, which gets autogenerated when the site is published.

You can view the autogenerated llms.txt in this PR's deploy preview:
https://deploy-preview-1338--streamlit-docs.netlify.app/llms.txt

🧠 Description of Changes

• Adds a script generate_llms_txt.py to generate /public/llms.txt
  • Adds a new property to markdown frontmatter:
    • description: a string description of the menu item, to be shown in llms.txt
  • Adds a few properties to menu.md:
    • description: a string description of the menu item, to be shown in llms.txt if the markdown frontmatter doesn't already have a description.
    • visible_to_llms: if present, then this menu item will only appear in llms.txt if this is True. If not present, then visible dictates whether the item appears.
• Removes generate_content_catalog.py since that's no longer needed for the AI assistant demo (https://demo-ai-assistant.streamlit.app/).

💥 Impact

Size:

• Small
• Not small

🌐 References

Contribution License Agreement

By submitting this pull request you agree that all contributions to this project are made under the Apache 2.0 license.

[sfc-gh-dmatthews]

I would prefer not to introduce duplicate information by putting meta data in menu.md. I have branch I'm reviewing that will update all MD files to have description and keywords in their frontmatter. (I've generated descriptions for everything except the knowledge base and am reading through all of them to tweak.) It's not as good for maintenance to separate the description from the page contents, so I'd like to pull the description from the frontmatter instead of menu.md.

[sfc-gh-tteixeira]

Ok, now the script supports having description: in the markdown frontmatter. If not present, then the description field from menu.md will be used (if present).

[sfc-gh-dmatthews]

I took the liberty of tweaking the format of the header lines so the result looks pretty if you view it as Markdown.