Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[docs] Update MSSQL README to use documentation guidelines #8240

Merged
merged 4 commits into from Oct 30, 2023
Merged

[docs] Update MSSQL README to use documentation guidelines #8240

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b8e8ad6
first draft edits to mssql readme
colleenmcginnis Oct 18, 2023
0784a36
Merge branch 'main' into mssql-setup
colleenmcginnis Oct 23, 2023
e8a4bc7
Merge branch 'main' into mssql-setup
colleenmcginnis Oct 30, 2023
fa12eee
address feedback
colleenmcginnis Oct 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
110 changes: 67 additions & 43 deletions packages/microsoft_sqlserver/_dev/build/docs/README.md
View file
Open in desktop
@@ -1,108 +1,132 @@
# Microsoft SQL Server Integration

The Microsoft SQL Server integration package allows you to search, observe and visualize the SQL Server audit logs, as well as performance and transaction log metrics, through Elasticsearch.
The Microsoft SQL Server integration package allows you to search, observe, and visualize the SQL Server audit logs, as well as performance and transaction log metrics, through Elasticsearch.

Auditing an instance of the SQL Server Database Engine or an individual database involves tracking and logging events that occur on the Database Engine.
For more information on SQL Server auditing, refer to [SQL Server Audit](https://docs.microsoft.com/en-us/sql/relational-databases/security/auditing/sql-server-audit-database-engine?view=sql-server-ver15).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The line in the beginning paragraph, somewhat looks mis-placed, since it is referring the audit data-stream primarily. It may wrongly infer referring to all data streams.


SQL Server audit lets you create server audits, which can contain server audit specifications for server-level events, and database audit specifications for database-level events.
## Data streams

See: [SQL Server Audit page](https://docs.microsoft.com/en-us/sql/relational-databases/security/auditing/sql-server-audit-database-engine?view=sql-server-ver15) for more information on SQL Server auditing.
The Microsoft SQL Server integration collects two types of data streams: logs and metrics.

`performance` metrics gathers the list of performance objects available on that server. Each server will have a different list of performance objects depending on the installed software.
**Logs** help you keep a record of events happening in Microsoft SQL Server.
Log data streams collected by the integration include:

`transaction_log` metrics collects all usage stats and the total space usage.
* `audit` provides events from the configured Windows event log channel.
* `logs` parses error logs created by the Microsoft SQL server.

## Named Instance
Other log sources, such as files, are not supported.

Microsoft SQL Server has a feature that allows running multiple databases on the same host (or clustered hosts) with separate settings. Establish a named instance connection by using the instance name along with the hostname (e.g. `host/instance_name` or `host:named_instance_port`) to collect metrics. Details of the host configuration are provided below.
Find more details in [Logs](#logs-reference).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unable to click the #logs-reference, in the generated document. Can we verify the anchors exist?

Screenshot 2023-10-31 at 11 23 41 AM

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix:

s/logs-reference/logs


### Query by Instance Name or Server Name in Kibana
**Metrics** give you insight into the state of Microsoft SQL Server.
Metric data streams collected by the integration include:

The data can be visualized in Kibana by filtering based on the instance name and server name. The instance name can be filtered by `mssql.metrics.instance_name` and the server name by `mssql.metrics.server_name` fields.
* `performance` metrics gathers the list of performance objects available on that server. Each server will have a different list of performance objects depending on the installed software.
* `transaction_log` metrics collects all usage stats and the total space usage.

Find more details in [Metrics](#metrics-reference).
colleenmcginnis marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix:

s/metrics-reference/metrics


## Requirements

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.
You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
colleenmcginnis marked this conversation as resolved.
Show resolved Hide resolved

### Microsoft SQL Server permissions

Before you can start sending data to Elastic, make sure you have the necessary Microsoft SQL Server permissions.

If you browse Microsoft Developer Network (MSDN) for the following tables, you will find a "Permissions" section that defines the permission needed for each table (for example, [the "Permissions" section on the `sys.dm_db_log_space_usage`](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-db-log-space-usage-transact-sql?view=sql-server-ver15#permissions) page).

## Permission/Access required for tables
1. `transaction_log`:
- [sys.databases](https://learn.microsoft.com/en-us/sql/relational-databases/system-compatibility-views/sys-sysdatabases-transact-sql?view=sql-server-ver16)
- [sys.dm_db_log_space_usage](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-db-log-space-usage-transact-sql?view=sql-server-ver16)
- [sys.dm_db_log_stats (DB_ID)](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-db-log-stats-transact-sql?view=sql-server-ver16)
2. `performance`:
- [sys.dm_os_performance_counters](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-os-performance-counters-transact-sql?view=sql-server-ver16)

If you browse MSDN for the following tables, you will find a "Permissions" section that defines the permission needed for each table, e.g. [sys.dm_db_log_space_usage](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-db-log-space-usage-transact-sql?view=sql-server-ver15) Permissions section.

### 1. transaction_log
## Setup

- [sys.databases](https://learn.microsoft.com/en-us/sql/relational-databases/system-compatibility-views/sys-sysdatabases-transact-sql?view=sql-server-ver16)
- [sys.dm_db_log_space_usage](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-db-log-space-usage-transact-sql?view=sql-server-ver16)
- [sys.dm_db_log_stats (DB_ID)](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-db-log-stats-transact-sql?view=sql-server-ver16)
For step-by-step instructions on how to set up any integration, refer to the
{{ url "getting-started-observability" "Getting started" }} guide.

### 2. performance

- [sys.dm_os_performance_counters](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-os-performance-counters-transact-sql?view=sql-server-ver16)
Below you'll find more specific details on setting up the Microsoft SQL Server integration.

## Host Configuration
<!-- Additional set up instructions -->
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this formatting intended: it comes as below:

Screenshot 2023-10-31 at 11 22 49 AM

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#hiding-content-with-comments

Ideally, this should have worked. On GitHub it does. Maybe for the website, we are using a different markdown renderer that does not support this?

### Named Instance

Integration supports collecting metrics from a single host. For multi-host metrics, each host can be run as a new integration.
Microsoft SQL Server has a feature that allows running multiple databases on the same host (or clustered hosts) with separate settings. Establish a named instance connection by using the instance name along with the hostname (e.g. `host/instance_name` or `host:named_instance_port`) to collect metrics. Details of the host configuration are provided below.

#### Query by Instance Name or Server Name in Kibana

The data can be visualized in Kibana by filtering based on the instance name and server name. The instance name can be filtered by `mssql.metrics.instance_name` and the server name by `mssql.metrics.server_name` fields.

As part of the input configuration, need to provide the user name, password and host details. The host configuration supports both named instances or default(no-name) instances, as per the syntax below.
### Host Configuration

### Connecting to Default Instance (host)
As part of the input configuration, you need to provide the user name, password and host details. The host configuration supports both named instances or default (no-name) instances, using the syntax below.

>Note: This integration supports collecting metrics from a single host. For multi-host metrics, each host can be run as a new integration.
colleenmcginnis marked this conversation as resolved.
Show resolved Hide resolved

**Connecting to Default Instance (host)**:

* `host` (e.g. `localhost` (Instance name is not needed when connecting to default instance))
* `host:port` (e.g. `localhost:1433`)

### Connecting to Named Instance (host)
**Connecting to Named Instance (host)**:

* `host/instance_name` (e.g. `localhost/namedinstance_01`)
* `host:named_instance_port` (e.g. `localhost:60873`)

## Compatibility

The package collects `performance` and `transaction_log` metrics, and `audit` events from the event log. Other log sources such as files are not supported.

## Configuration
### Configuration

### audit
#### Audit

There are several levels of auditing for SQL Server, depending on government or standards requirements for your installation. The SQL Server Audit feature enables you to audit server-level and database-level groups of events and individual events.

See: [SQL Server Audit Action Groups and Actions](https://docs.microsoft.com/en-us/sql/relational-databases/security/auditing/sql-server-audit-action-groups-and-actions?view=sql-server-ver15) for more information on the different audit levels.

See: [Instructions on how to enable auditing for SQL Server](https://docs.microsoft.com/en-us/sql/relational-databases/security/auditing/create-a-server-audit-and-server-audit-specification?view=sql-server-ver15).
For more information on the different audit levels, refer to [SQL Server Audit Action Groups and Actions](https://docs.microsoft.com/en-us/sql/relational-databases/security/auditing/sql-server-audit-action-groups-and-actions?view=sql-server-ver15).
Then to enable auditing for SQL Server, refer to these [instructions](https://docs.microsoft.com/en-us/sql/relational-databases/security/auditing/create-a-server-audit-and-server-audit-specification?view=sql-server-ver15).

> Note: For the integration package to be able to read and send audit events the event target must be configured to be Windows event log.
>Note: For the integration package to be able to read and send audit events the event target must be configured to be Windows event log.
colleenmcginnis marked this conversation as resolved.
Show resolved Hide resolved

### Audit events
#### Audit events

Collects SQL Server audit events from the specified windows event log channel.

### Log
#### Log

The SQL Server `log` contains user-defined events and certain system events you can use for troubleshooting.

See: [View the SQL Server error log in SQL Server Management Studio](https://docs.microsoft.com/en-us/sql/relational-databases/performance/view-the-sql-server-error-log-sql-server-management-studio?view=sql-server-ver16)
Read more in [View the SQL Server error log in SQL Server Management Studio](https://docs.microsoft.com/en-us/sql/relational-databases/performance/view-the-sql-server-error-log-sql-server-management-studio?view=sql-server-ver16).

### Performance metrics
#### Performance metrics

Collects the `performance` counter metrics. The dynamic counter feature provides flexibility to collect metrics by providing the counter as an input.
This input can be a regular expression which will filter results based on pattern.
For example, if %grant% is given as input, it will enable metrics collection for all of the counters with names like 'Memory Grants Pending', 'Active memory grants count' etc.
MSSQL supports a limited set of regular expressions, See [here](https://learn.microsoft.com/en-us/previous-versions/sql/sql-server-2008-r2/ms187489(v=sql.105)?redirectedfrom=MSDN) for details.
MSSQL supports a limited set of regular expressions. For more details, refer to [Pattern Matching in Search Conditions](https://learn.microsoft.com/en-us/previous-versions/sql/sql-server-2008-r2/ms187489(v=sql.105)?redirectedfrom=MSDN).

> Note: Dynamic counters will go through some basic ingest pipeline post-processing to make counter names in lowercase and remove special characters and these fields will not have any static field mappings.

The feature `merge_results` has been introduced in 8.4 beats which creates a single event by combining the metrics together in a single event. See [here](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-sql.html#_example_merge_multiple_queries_to_single_event) for details.
The feature `merge_results` has been introduced in 8.4 beats which creates a single event by combining the metrics together in a single event. For more details, refer to [SQL module](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-sql.html#_example_merge_multiple_queries_to_single_event).

colleenmcginnis marked this conversation as resolved.
Show resolved Hide resolved
See: [Instructions about each performance counter metrics](https://docs.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-os-performance-counters-transact-sql?view=sql-server-ver15)
Read more in [Instructions about each performance counter metrics](https://docs.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-os-performance-counters-transact-sql?view=sql-server-ver15).
colleenmcginnis marked this conversation as resolved.
Show resolved Hide resolved

### Transaction log metrics
#### Transaction log metrics

Collects system level `transaction_log` metrics information for SQL Server instance.
Metrics for user-level databases can be collected by providing a list of user databases for which metrics are to be collected.

See: [Instructions and the operations supported by transaction log](https://docs.microsoft.com/en-us/sql/relational-databases/logs/the-transaction-log-sql-server?view=sql-server-ver15)
Read more in [Instructions and the operations supported by transaction log](https://docs.microsoft.com/en-us/sql/relational-databases/logs/the-transaction-log-sql-server?view=sql-server-ver15).
colleenmcginnis marked this conversation as resolved.
Show resolved Hide resolved

### Fetch from all databases
#### Fetch from all databases

To simplify the process of fetching metrics from all databases on the server, you can enable the `Fetch from all databases` toggle when configuring the integration. This field overrides manually entered database names in the `Databases` input and instead fetches the required `transaction_log` metrics from all databases, including system and user-defined databases.

Keep in mind that this feature is disabled by default and needs to be manually enabled to be activated.

### Password URL encoding
#### Password URL encoding

When the password contains special characters, pass these special characters using URL encoding.

Expand Down
5 changes: 5 additions & 0 deletions packages/microsoft_sqlserver/changelog.yml
View file
Open in desktop
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "2.2.3"
changes:
- description: Update README to use documentation guidelines
type: enhancement
link: TBD
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
link: TBD
link: https://github.com/elastic/integrations/pull/8240

- version: "2.2.2"
changes:
- description: Bump minimum kibana version for fetch_from_all_databases feature
Expand Down