Docs/core and guides #2417
Docs/core and guides #2417
Conversation
| - `trace_id` is the id of a trace which tracks “the progression of a single request, as it is handled by services that make up an application” - `OpenTelemetry Basic Documentation <https://www.dynatrace.com/support/help/extend-dynatrace/opentelemetry/basics>`_ | ||
| - `span_id is` the id of a span which is contained within a trace. “A span is the building block of a trace and is a named, timed operation that represents a piece of the workflow in the distributed system. Multiple spans are pieced together to create a trace.” - `OpenTelemetry Span Documentation <https://opentelemetry.lightstep.com/spans/>`_ | ||
| - `sampled is` the number of times this trace has been sampled. “Sampling is a mechanism to control the noise and overhead introduced by OpenTelemetry by reducing the number of samples of traces collected and sent to the backend.” - `OpenTelemetry SDK Documentation <https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md>`_ |
There was a problem hiding this comment.
Should we make a table for this section instead?
There was a problem hiding this comment.
I couldn't find a convention for this style in our current doc. Main style I found was enumerating the options in titles like here: https://docs.bentoml.org/en/latest/concepts/building_bentos.html
I feel like bullet points are better. Maybe table if you have more than one thing you want to specify about the parameter?
| Add further reading section | ||
| Further Reading | ||
| --------------- | ||
| - :ref:`API Reference for IO descriptors <api-io-descriptors>` |
There was a problem hiding this comment.
Is this linking back to itself?
There was a problem hiding this comment.
No, it's a little confusing, but this is actually linking to the "API Reference" documentation (https://docs.bentoml.org/en/latest/api/api_io_descriptors.html) for the io descriptors. This current page is called: "api-io-descriptors-page"
| Managing Models and Bentos Remotely with Yatai | ||
| ---------------------------------------------- | ||
|
|
||
| Yatai is BentoML's end to end deployment and monitoring tool. It also functions as a remote model and bento repository. |
There was a problem hiding this comment.
Maybe a platform instead of tool?
|
|
||
| Yatai is BentoML's end to end deployment and monitoring tool. It also functions as a remote model and bento repository. | ||
|
|
||
| To connect the CLI to a remote `Yatai <yatai-service-page>`, use the `bentoml login YATAI_URL` command. |
There was a problem hiding this comment.
Let's have a code tab for bentoml login YATAI_URL
| @@ -3,4 +3,44 @@ | |||
| Adaptive Batching | |||
There was a problem hiding this comment.
Should this be moved under Core Concepts?
There was a problem hiding this comment.
I'm fine with it here for now, but I don't have a strong opinion about this.
| "While serving a TensorFlow model, batching individual model inference requests together can be important for performance. In particular, batching is necessary to unlock the high throughput promised by hardware accelerators such as GPUs." | ||
| -- `TensorFlow documentation <https://github.com/tensorflow/serving/blob/master/tensorflow_serving/batching/README.md>`_ | ||
|
|
||
| As an optimization for a real-time service, batching works off of 2 main concepts. |
There was a problem hiding this comment.
Let's also explain why we do server-side batching. Server side batching is advantageous because 1) it simplifies the client logic 2) and the server can often times batch more efficiently than the client due to traffic volume.
| Running with Adaptive Batching | ||
| ------------------------------ | ||
|
|
||
| There are 2 ways that adaptive batching will run depending on how you've deployed BentoML. |
There was a problem hiding this comment.
Let's create a placeholder page for explaining the standalone mode vs distributed mode.
There was a problem hiding this comment.
do we want to take this explanation out then?
docs/source/guides/logging.rst
Outdated
| .. parsed-literal:: | ||
|
|
||
| "[%(component)s] %(message)s" |
There was a problem hiding this comment.
I think this format is less relevant for users. For access logs, we can breakdown a log entry.
[api_server] 127.0.0.1:55723 (scheme=http,method=POST,path=/classify,type=application/json,length=9) (status=200,type=application/json,length=1) 0.465ms (trace=175923203261911804790364073480797463970,span=2163644694959317809,sampled=0)
In the access log example, the format has the following components.
- Component
- Client IP
- Request
- Response
- Latency
- Traces
docs/source/guides/logging.rst
Outdated
| Logging Configuration | ||
| --------------------- | ||
|
|
||
| Logs can be configured from the bentofile.yaml file for both web requests and model serving requests. |
There was a problem hiding this comment.
Logging is not configured through bentofile.yaml, rather, it is configured through the service configuration documented here, https://docs.bentoml.org/en/latest/guides/configuration.html
Co-authored-by: Sean Sheng <s3sheng@gmail.com>
Co-authored-by: Sean Sheng <s3sheng@gmail.com>
* core concepts edits * added content to a few guides * initial logging documentation * updates to format and added contributer images in readme * more updates to formatting * Update README.md * Update README.md * Update README.md * Update bento_management.rst * Update docs/source/guides/adaptive_batching.rst Co-authored-by: Sean Sheng <s3sheng@gmail.com> * Update docs/source/guides/adaptive_batching.rst Co-authored-by: Sean Sheng <s3sheng@gmail.com> * updates based on sean's feedback Co-authored-by: Tim Liu <timliu@Tims-MBP.attlocal.net> Co-authored-by: Sean Sheng <s3sheng@gmail.com>
Description
Small updates on the Core Concepts
Guides added:
Adaptive Batching
Logging
Multi-Model serving
Custom endpoints
Motivation and Context
How Has This Been Tested?
Checklist:
make formatandmake lintscript have passed(instructions).