Improve the docs for developers (Create a Developers / Hackers guide for DataFusion)

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
DataFusion is primarily aimed at developers, as explained in https://github.com/apache/arrow-datafusion#use-cases

Thus, it will help if provide documentation that helps developers understand what DataFusion offers and if it is appropriate for their project or not.

Describe the solution you'd like
I would like a DataFusion Architecture guide – aimed at other developers, that contains high level information about how DataFusion is organized (for example, with the content described in #5499)

The trick will be to keep the guide helpful but general enough that it doesn't get out of date too quickly

Topics:

• Basic flow
• Important structures (LogicalPlan, Exprs, SessionContext, PhysicalExpr, ExecutionPlan)
• DataSources (TableProvider trait, etc)

Specific tasks

• Decide where to put the content (ANSWER: main crate page https://docs.rs/datafusion/latest/datafusion/index.html)
• Improve contributor guide and main API landing page #5921
• Improve / Add documentation to TableProvider
• Improve / Add documentation to CatalogProvider
• Improve / Add documentation to SessionContext
• Clarify the difference between SessionContext, SessionState and TaskContext
• Update / Improve the planning flow / description in main API docs page

Describe alternatives you've considered

Additional context
Part of a larger effort to improve documentation #3058

[alamb]

One thing I have been thinking about is where to put this information

So far I have found at least two possiblities:
https://arrow.apache.org/datafusion/contributor-guide/architecture.html
https://docs.rs/datafusion/latest/datafusion/index.html#parse-plan-optimize-execute

In general I think keeping the architecture / code documentation as close to the code as possible is a good idea, so I am leaning towards keeping in in doc comments

[alamb]

I plan to work on this item over the next few weeks

[alamb]

I have begun cleaning up. First PR: #5921