[docs] - Improve Pipes API docs

docs/sphinx/sections/api/apidocs/libraries/dagster-pipes.rst

@@ -1,51 +1,101 @@
 Pipes (dagster-pipes)
------------------------
-
-This library is intended for inclusion in an external process that integrates
-with Dagster using the Pipes protocol.
+=====================
 
 .. currentmodule:: dagster_pipes
 
+The ``dagster-pipes`` library is intended for inclusion in an external process that integrates with Dagster using the `Pipes </concepts/dagster-pipes>`_ protocol. This could be in an environment like Databricks, Kubernetes, or Docker. Using this library, you can write code in the external process that streams metadata back to Dagster.
+
+For a detailed look at the Pipes process, including how to customize it, refer to the `Dagster Pipes details and customization guide </concepts/dagster-pipes/dagster-pipes-details-and-customization#overview-and-terms>`__.
+
+**Looking to set up a Pipes client in Dagster?** Refer to the `Dagster Pipes API reference </_apidocs/pipes>`_.
+
+**Note**: This library isn't included with ``dagster`` and must be `installed separately <https://pypi.org/project/dagster-pipes/>`_.
+
+----
+
+Context
+-------
+
 .. autofunction:: open_dagster_pipes
 
 .. autoclass:: PipesContext
 
-.. autoclass:: DagsterPipesError
+----
 
-.. autoclass:: DagsterPipesWarning
+Advanced
+--------
 
-.. autofunction:: encode_env_var
+Most Pipes users won't need to use the APIs in the following sections unless they are customizing the Pipes protocol.
 
-.. autofunction:: decode_env_var
+Refer to the `Dagster Pipes details and customization guide </concepts/dagster-pipes/dagster-pipes-details-and-customization#overview-and-terms>`__ for more information.
+
+Context loaders
+^^^^^^^^^^^^^^^
+
+Context loaders load the context payload from the location specified in the bootstrap payload.
 
 .. autoclass:: PipesContextLoader
 
-.. autoclass:: PipesMessageWriter
+.. autoclass:: PipesDefaultContextLoader
 
-.. autoclass:: PipesMessageWriterChannel
+.. autoclass:: PipesDbfsContextLoader
+
+----
+
+Params loaders
+^^^^^^^^^^^^^^
+
+Params loaders load the bootstrap payload from some globally accessible key-value store.
 
 .. autoclass:: PipesParamsLoader
 
-.. autoclass:: PipesBlobStoreMessageWriter
+.. autoclass:: PipesEnvVarParamsLoader
 
-.. autoclass:: PipesBlobStoreMessageWriterChannel
+----
 
-.. autoclass:: PipesBufferedFilesystemMessageWriterChannel
+Message writers
+^^^^^^^^^^^^^^^
 
-.. autoclass:: PipesDefaultContextLoader
+Message writers write messages to the location specified in the bootstrap payload.
+
+.. autoclass:: PipesMessageWriter
 
 .. autoclass:: PipesDefaultMessageWriter
 
+.. autoclass:: PipesBlobStoreMessageWriter
+
+.. autoclass:: PipesS3MessageWriter
+
+.. autoclass:: PipesDbfsMessageWriter
+
+----
+
+Message writer channels
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Message writer channels are objects that write messages back to the Dagster orchestration process.
+
+.. autoclass:: PipesMessageWriterChannel
+
+.. autoclass:: PipesBlobStoreMessageWriterChannel
+
+.. autoclass:: PipesBufferedFilesystemMessageWriterChannel
+
 .. autoclass:: PipesFileMessageWriterChannel
 
 .. autoclass:: PipesStreamMessageWriterChannel
 
-.. autoclass:: PipesEnvVarParamsLoader
+.. autoclass:: PipesS3MessageWriterChannel
 
-.. autoclass:: PipesS3MessageWriter
+----
 
-.. autoclass:: PipesS3MessageWriterChannel
+Utilities
+^^^^^^^^^
 
-.. autoclass:: PipesDbfsContextLoader
+.. autofunction:: encode_env_var
 
-.. autoclass:: PipesDbfsMessageWriter
+.. autofunction:: decode_env_var
+
+.. autoclass:: DagsterPipesError
+
+.. autoclass:: DagsterPipesWarning

docs/sphinx/sections/api/apidocs/pipes.rst

@@ -1,34 +1,71 @@
+Dagster Pipes
+=============
+
 .. currentmodule:: dagster
 
-Pipes
-=====
+`Dagster Pipes </concepts/dagster-pipes>`_  is a toolkit for building integrations between Dagster and external execution environments. This reference outlines the APIs included with the ``dagster`` library, which should be used in the orchestration environment.
+
+For a detailed look at the Pipes process, including how to customize it, refer to the `Dagster Pipes details and customization guide </concepts/dagster-pipes/dagster-pipes-details-and-customization#overview-and-terms>`__.
+
+**Looking to write code in an external process?** Refer to the API reference for the separately-installed `dagster-pipes </_apidocs/libraries/dagster-pipes>`_ library. 
+
+----
+
+Sessions
+--------
+
+.. autoclass:: PipesSession
+
+.. autofunction:: open_pipes_session
+
+.. currentmodule:: dagster._core.pipes.subprocess
+
+----
 
-Abstractions for the orchestration side of the Dagster Pipes protocol.
+Clients
+-------
+
+.. currentmodule:: dagster
 
 .. autoclass:: PipesClient
 
 .. autoclass:: PipesSubprocessClient
 
-.. autoclass:: PipesContextInjector
+----
 
-.. autoclass:: PipesMessageReader
+Advanced
+--------
 
-.. autoclass:: PipesMessageHandler
+Most Pipes users won't need to use the APIs in the following sections unless they are customizing the Pipes protocol.
 
-.. autoclass:: PipesSession
+Refer to the `Dagster Pipes details and customization guide </concepts/dagster-pipes/dagster-pipes-details-and-customization#overview-and-terms>`__ for more information.
 
-.. autoclass:: PipesBlobStoreMessageReader
+Context injectors
+^^^^^^^^^^^^^^^^^
+
+Context injectors write context payloads to an externally accessible location and yield a set of parameters encoding the location for inclusion in the bootstrap payload.
+
+.. autoclass:: PipesContextInjector
 
 .. autoclass:: PipesEnvContextInjector
 
 .. autoclass:: PipesFileContextInjector
 
-.. autoclass:: PipesFileMessageReader
-
 .. autoclass:: PipesTempFileContextInjector
 
-.. autoclass:: PipesTempFileMessageReader
+----
 
-.. autofunction:: open_pipes_session
+Message readers
+^^^^^^^^^^^^^^^
 
-.. currentmodule:: dagster._core.pipes.subprocess
+Message readers read messages (and optionally log files) from an externally accessible location and yield a set of parameters encoding the location in the bootstrap payload. 
+
+.. autoclass:: PipesMessageReader
+
+.. autoclass:: PipesBlobStoreMessageReader
+
+.. autoclass:: PipesFileMessageReader
+
+.. autoclass:: PipesTempFileMessageReader
+
+.. autoclass:: PipesMessageHandler