[docs] - Improve Pipes API docs (#21707)

## Summary & Motivation

This PR adds information to the Pipes and `dagster-pipes` API references
to clarify where and how each set of APIs should be used. Previously,
the introductions for these references were vague and confusing - it was
difficult to tell which set of APIs you needed and when.

I've also added broken the APIs up into sections, grouped by type, and
added information about what each group does. I've also added links to
appropriate guides to make cross-referencing content easier.

**After**:

**Preview links**: [Dagster Pipes
reference](https://erin-pipes-api-experiment.dagster.dagster-docs.io/_apidocs/pipes);
[`dagster-pipes`
reference](https://erin-pipes-api-experiment.dagster.dagster-docs.io/_apidocs/libraries/dagster-pipes)

![Screenshot 2024-05-10 at 11 37 05
AM](https://github.com/dagster-io/dagster/assets/16615212/b8ae1d0b-53f3-4a84-8286-f993cc37d117)
![Screenshot 2024-05-10 at 11 37 18
AM](https://github.com/dagster-io/dagster/assets/16615212/b39e6e3b-16fe-4ace-8768-c366d4b76888)


**Before**:

![Screenshot 2024-05-08 at 11 03 29
AM](https://github.com/dagster-io/dagster/assets/16615212/5c9cbac3-56f5-4b8f-8988-2e89a08c654d)

![Screenshot 2024-05-08 at 11 03 44
AM](https://github.com/dagster-io/dagster/assets/16615212/7c29af39-e65d-469b-b66f-54271912356c)

## How I Tested These Changes

👀 , local

docs/sphinx/sections/api/apidocs/libraries/dagster-pipes.rst

@@ -1,51 +1,101 @@
 Pipes (dagster-pipes)
------------------------
-
-This library is intended for inclusion in an external process that integrates
-with Dagster using the Pipes protocol.
+=====================
 
 .. currentmodule:: dagster_pipes
 
+The ``dagster-pipes`` library is intended for inclusion in an external process that integrates with Dagster using the `Pipes </concepts/dagster-pipes>`_ protocol. This could be in an environment like Databricks, Kubernetes, or Docker. Using this library, you can write code in the external process that streams metadata back to Dagster.
+
+For a detailed look at the Pipes process, including how to customize it, refer to the `Dagster Pipes details and customization guide </concepts/dagster-pipes/dagster-pipes-details-and-customization#overview-and-terms>`__.
+
+**Looking to set up a Pipes client in Dagster?** Refer to the `Dagster Pipes API reference </_apidocs/pipes>`_.
+
+**Note**: This library isn't included with ``dagster`` and must be `installed separately <https://pypi.org/project/dagster-pipes/>`_.
+
+----
+
+Context
+-------
+
 .. autofunction:: open_dagster_pipes
 
 .. autoclass:: PipesContext
 
-.. autoclass:: DagsterPipesError
+----
 
-.. autoclass:: DagsterPipesWarning
+Advanced
+--------
 
-.. autofunction:: encode_env_var
+Most Pipes users won't need to use the APIs in the following sections unless they are customizing the Pipes protocol.
 
-.. autofunction:: decode_env_var
+Refer to the `Dagster Pipes details and customization guide </concepts/dagster-pipes/dagster-pipes-details-and-customization#overview-and-terms>`__ for more information.
+
+Context loaders
+^^^^^^^^^^^^^^^
+
+Context loaders load the context payload from the location specified in the bootstrap payload.
 
 .. autoclass:: PipesContextLoader
 
-.. autoclass:: PipesMessageWriter
+.. autoclass:: PipesDefaultContextLoader
 
-.. autoclass:: PipesMessageWriterChannel
+.. autoclass:: PipesDbfsContextLoader
+
+----
+
+Params loaders
+^^^^^^^^^^^^^^
+
+Params loaders load the bootstrap payload from some globally accessible key-value store.
 
 .. autoclass:: PipesParamsLoader
 
-.. autoclass:: PipesBlobStoreMessageWriter
+.. autoclass:: PipesEnvVarParamsLoader
 
-.. autoclass:: PipesBlobStoreMessageWriterChannel
+----
 
-.. autoclass:: PipesBufferedFilesystemMessageWriterChannel
+Message writers
+^^^^^^^^^^^^^^^
 
-.. autoclass:: PipesDefaultContextLoader
+Message writers write messages to the location specified in the bootstrap payload.
+
+.. autoclass:: PipesMessageWriter
 
 .. autoclass:: PipesDefaultMessageWriter
 
+.. autoclass:: PipesBlobStoreMessageWriter
+
+.. autoclass:: PipesS3MessageWriter
+
+.. autoclass:: PipesDbfsMessageWriter
+
+----
+
+Message writer channels
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Message writer channels are objects that write messages back to the Dagster orchestration process.
+
+.. autoclass:: PipesMessageWriterChannel
+
+.. autoclass:: PipesBlobStoreMessageWriterChannel
+
+.. autoclass:: PipesBufferedFilesystemMessageWriterChannel
+
 .. autoclass:: PipesFileMessageWriterChannel
 
 .. autoclass:: PipesStreamMessageWriterChannel
 
-.. autoclass:: PipesEnvVarParamsLoader
+.. autoclass:: PipesS3MessageWriterChannel
 
-.. autoclass:: PipesS3MessageWriter
+----
 
-.. autoclass:: PipesS3MessageWriterChannel
+Utilities
+^^^^^^^^^
 
-.. autoclass:: PipesDbfsContextLoader
+.. autofunction:: encode_env_var
 
-.. autoclass:: PipesDbfsMessageWriter
+.. autofunction:: decode_env_var
+
+.. autoclass:: DagsterPipesError
+
+.. autoclass:: DagsterPipesWarning

docs/sphinx/sections/api/apidocs/pipes.rst

@@ -1,34 +1,71 @@
+Dagster Pipes
+=============
+
 .. currentmodule:: dagster
 
-Pipes
-=====
+`Dagster Pipes </concepts/dagster-pipes>`_  is a toolkit for building integrations between Dagster and external execution environments. This reference outlines the APIs included with the ``dagster`` library, which should be used in the orchestration environment.
+
+For a detailed look at the Pipes process, including how to customize it, refer to the `Dagster Pipes details and customization guide </concepts/dagster-pipes/dagster-pipes-details-and-customization#overview-and-terms>`__.
+
+**Looking to write code in an external process?** Refer to the API reference for the separately-installed `dagster-pipes </_apidocs/libraries/dagster-pipes>`_ library. 
+
+----
+
+Sessions
+--------
+
+.. autoclass:: PipesSession
+
+.. autofunction:: open_pipes_session
+
+.. currentmodule:: dagster._core.pipes.subprocess
+
+----
 
-Abstractions for the orchestration side of the Dagster Pipes protocol.
+Clients
+-------
+
+.. currentmodule:: dagster
 
 .. autoclass:: PipesClient
 
 .. autoclass:: PipesSubprocessClient
 
-.. autoclass:: PipesContextInjector
+----
 
-.. autoclass:: PipesMessageReader
+Advanced
+--------
 
-.. autoclass:: PipesMessageHandler
+Most Pipes users won't need to use the APIs in the following sections unless they are customizing the Pipes protocol.
 
-.. autoclass:: PipesSession
+Refer to the `Dagster Pipes details and customization guide </concepts/dagster-pipes/dagster-pipes-details-and-customization#overview-and-terms>`__ for more information.
 
-.. autoclass:: PipesBlobStoreMessageReader
+Context injectors
+^^^^^^^^^^^^^^^^^
+
+Context injectors write context payloads to an externally accessible location and yield a set of parameters encoding the location for inclusion in the bootstrap payload.
+
+.. autoclass:: PipesContextInjector
 
 .. autoclass:: PipesEnvContextInjector
 
 .. autoclass:: PipesFileContextInjector
 
-.. autoclass:: PipesFileMessageReader
-
 .. autoclass:: PipesTempFileContextInjector
 
-.. autoclass:: PipesTempFileMessageReader
+----
 
-.. autofunction:: open_pipes_session
+Message readers
+^^^^^^^^^^^^^^^
 
-.. currentmodule:: dagster._core.pipes.subprocess
+Message readers read messages (and optionally log files) from an externally accessible location and yield a set of parameters encoding the location in the bootstrap payload. 
+
+.. autoclass:: PipesMessageReader
+
+.. autoclass:: PipesBlobStoreMessageReader
+
+.. autoclass:: PipesFileMessageReader
+
+.. autoclass:: PipesTempFileMessageReader
+
+.. autoclass:: PipesMessageHandler