Skip to content

DOC-12312: streaming completed requests to local files #273

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Parent: DOC-12829: Publish Couchbase Server 7.6.4 Docs
merged 7 commits into from
Sep 23, 2024
Merged

DOC-12312: streaming completed requests to local files #273

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8092dd0
Add Streaming Completed Requests
simon-dew Sep 17, 2024
7c2d2a5
Try to make tabs sync
simon-dew Sep 17, 2024
76548b4
Add system:completed_requests_history to sysinfo
simon-dew Sep 17, 2024
2534a22
Add links to Admin REST API
simon-dew Sep 18, 2024
8413d07
Attempting to do tab sync
simon-dew Sep 18, 2024
ee74976
Update table formatting for sysinfo
simon-dew Sep 18, 2024
591f514
Add links to gzip and jq
simon-dew Sep 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions modules/n1ql/pages/n1ql-intro/sysinfo.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,31 @@
:description: {sqlpp} has a system namespace that stores metadata about data containers, the Query service, and the system as a whole. \
You can query the system namespace to get this information.

// Pass through HTML table styles for this page

ifdef::basebackend-html[]
++++
<style type="text/css">
/* No maximum width for table cells */
.doc table.spread > tbody > tr > *,
.doc table.stretch > tbody > tr > * {
max-width: none !important;
}

/* Ignore fixed column widths */
table:not(.fixed-width) col{
width: auto !important;
}

/* Do not hyphenate words in the table */
td.tableblock p,
p.tableblock{
hyphens: manual !important;
}
</style>
++++
endif::[]

[abstract]
{description}

Expand All @@ -29,6 +54,7 @@ xref:n1ql:n1ql-manage/monitoring-n1ql-query.adoc#vitals[system:vitals]
xref:n1ql:n1ql-manage/monitoring-n1ql-query.adoc#sys-active-req[system:active_requests]
xref:n1ql:n1ql-manage/monitoring-n1ql-query.adoc#sys-prepared[system:prepareds]
xref:n1ql:n1ql-manage/monitoring-n1ql-query.adoc#sys-completed-req[system:completed_requests]
xref:n1ql:n1ql-manage/monitoring-n1ql-query.adoc#sys-history[system:completed_requests_history]

a| [%hardbreaks]
<<sys_my-user-info,system:my_user_info>>
Expand Down
94 changes: 85 additions & 9 deletions modules/n1ql/pages/n1ql-manage/monitoring-n1ql-query.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ This information can be very useful to assess the current workload and performan
[#sys-vitals-get]
=== Get System Vitals

To view system vitals, use the Admin REST API or a {sqlpp} query.
To view system vitals, use the xref:n1ql:n1ql-rest-api/admin.adoc[Admin REST API] or a {sqlpp} query.

[tabs]
====
Expand Down Expand Up @@ -126,7 +126,7 @@ The `system:active_requests` catalog lists all currently executing active reques
[[sys-active-get]]
=== Get Active Requests

To view active requests, use the Admin REST API or a {sqlpp} query.
To view active requests, use the xref:n1ql:n1ql-rest-api/admin.adoc[Admin REST API] or a {sqlpp} query.

[tabs]
====
Expand Down Expand Up @@ -173,7 +173,7 @@ SELECT *, meta().plan FROM system:active_requests;
[[sys-active-delete]]
=== Terminate an Active Request

The DELETE command can be used to terminate an active request, for instance, a non-responding or a long-running query.
To terminate an active request, for instance, a non-responding or a long-running query, use the xref:n1ql:n1ql-rest-api/admin.adoc[Admin REST API] or a {sqlpp} query.

[tabs]
====
Expand Down Expand Up @@ -272,7 +272,7 @@ When there are multiple prepared statements with the same name in different quer
[[sys-prepared-get]]
=== Get Prepared Statements

To get a list of all known prepared statements, use the Admin REST API or a {sqlpp} query.
To get a list of all known prepared statements, use the xref:n1ql:n1ql-rest-api/admin.adoc[Admin REST API] or a {sqlpp} query.

[tabs]
====
Expand Down Expand Up @@ -346,7 +346,7 @@ SELECT *, meta().plan FROM system:prepareds;
[[sys-prepared-delete]]
=== Delete Prepared Statements

To delete a specific prepared statement, use the Admin REST API or a {sqlpp} query.
To delete a specific prepared statement, use the xref:n1ql:n1ql-rest-api/admin.adoc[Admin REST API] or a {sqlpp} query.

[tabs]
====
Expand Down Expand Up @@ -492,7 +492,7 @@ This information provides a general insight into the health and performance of t
[[sys-completed-get]]
=== Get Completed Requests

To get a list of all logged completed requests, use the Admin REST API or a {sqlpp} query.
To get a list of all logged completed requests, use the xref:n1ql:n1ql-rest-api/admin.adoc[Admin REST API] or a {sqlpp} query.

[tabs]
====
Expand Down Expand Up @@ -559,7 +559,7 @@ SELECT *, meta().plan FROM system:completed_requests;
[[sys-completed-delete]]
=== Purge the Completed Requests

To purge a specific completed request, use the Admin REST API or a {sqlpp} query.
To purge a specific completed request, use the xref:n1ql:n1ql-rest-api/admin.adoc[Admin REST API] or a {sqlpp} query.

[tabs]
====
Expand Down Expand Up @@ -690,9 +690,9 @@ For field names and meanings, see xref:n1ql:n1ql-rest-api/admin.adoc#_requests[R
For query plan field names and meanings, see <<monitor-profile-details>>.

[[sys-completed-config]]
== Configure the Completed Requests
== Configure Completed Requests

You can configure the `system:completed_requests` keyspace by specifying parameters through the Admin API `/admin/settings` endpoint.
You can configure the `system:completed_requests` keyspace by specifying parameters through the xref:n1ql:n1ql-rest-api/admin.adoc[Admin REST API] `/admin/settings` endpoint.

You can specify the conditions for completed request logging using the `completed` field.
This field takes a JSON object containing the names and values of logging qualifiers.
Expand Down Expand Up @@ -819,6 +819,82 @@ curl $BASE_URL/admin/settings -u $USER:$PASSWORD \
-d '{"completed-limit":1000}'
----

[[sys-history]]
== Stream Completed Requests

[.status]#Couchbase Server 7.6.4#

In Couchbase Server 7.6.4 and later, you can stream completed requests to disk.

To enable completed request streaming, use the xref:n1ql:n1ql-rest-api/admin.adoc[Admin REST API] `/admin/settings` endpoint to specify the `completed_stream_size` property.

[source,sh]
----
curl $BASE_URL/admin/settings -u $USER:$PASSWORD \
-H 'Content-Type: application/json' \
-d '{"completed_stream_size":500}'
----

This property is a file size in MiB.
When set to `0` (the default), completed request streaming is disabled.

When set to any size greater than `0`, completed requests are streamed to archive files.
The value of this property determines the size of the data to retain, per node.
The configuration for completed requests determines which requests are saved.

NOTE: The additional processing required to save completed requests to disk may limit overall request throughput on a Query node, but typically only when every completed request is being recorded, and requests are very small or short-lived.
The speed of the file system on which the server logs directory resides naturally affects the potential impact too.

[#sys-history-files]
=== Archived Request Files

When streaming is enabled, completed requests are saved to GZIP archives with the prefix `local_request_log` in the Couchbase Server `logs` directory.
Each saved GZIP archive file contains multiple JSON entries, one for each for each recorded completed request.

Couchbase Server writes multiple archive files in parallel, so whilst the order of requests in a file is sequential, a single given file may not contain a contiguous sequence of requests.

When an archive file reaches or exceeds 100 MiB, it is finalized and saved to disk.
This is not a hard limit -- entries are not truncated to adhere to it.
Files may also be finalized with less content, if nothing has been written to them for an extended period.
Files that are actively being written are not available for reading, and they don't count towards the configured size limit until they're finalized.

Couchbase Server tries to manage and retain archive files such that the total disk space used by the files is within the specified limit for the node.
When the specified limit is reached, older files are removed as necessary to make space for newly finalized files.
When a file is removed, it isn't guaranteed that only the oldest requests are evicted, given that Couchbase Server writes to multiple archive files in parallel.

[#sys-history-view]
=== View Archived Requests

To view archived completed requests, use https://www.gnu.org/software/gzip[gzip] and https://jqlang.github.io/jq[jq] on the command line, or a {sqlpp} query.

[tabs,sync-group-id="REST API|{sqlpp}"]
====
Command Line::
+
--
To view all archived completed requests in `$FILE`:

[source,sh]
----
gzip -qdc $FILE | jq .
----
--

{sqlpp}::
+
--
To get a list of archived completed requests using {sqlpp}:

[source,sqlpp]
----
SELECT * FROM system:completed_requests_history;
----
--
====

The `system:completed_requests_history` keyspace is provided for {sqlpp} access to the archived files, but as they are external GZIP archives performance is restricted, particularly with large histories on clusters with multiple Query service nodes.
Directly reading the files may be more useful in some cases.

[#query-monitoring-settings]
== Query Profiling

Expand Down