diff --git a/samples/manage/sql-assessment-api/docs/Customization/MessageTemplate.md b/samples/manage/sql-assessment-api/docs/Customization/MessageTemplate.md new file mode 100644 index 0000000000..ff23a6198f --- /dev/null +++ b/samples/manage/sql-assessment-api/docs/Customization/MessageTemplate.md @@ -0,0 +1,47 @@ +# Message template + +## Overview + +A check produces a recommendation message specified in the [__mesasge__](./Rule.md#message) property. The message can contain data returned from a probe or calculated by the check. + +Message template resembles [C# string interpolation](https://learn.microsoft.com/dotnet/csharp/language-reference/tokens/interpolated). Data variables can be inserted in curly braces preceded by the 'at' sign '@'. + +In the following example a probe returns `@indexName`, `@Schema`, and `@Object` string variables. The API will substitute the references in curly braces with actual values. + +```json +"message": "Drop hypothetical @{IndexName} index for @{Schema}.@{Object}" +``` + +Message template supports format specifiers as well. The specifier must follow the variable name separated by a colon ':'. For more information on formatting values see [How to format numbers, dates, enums, and other types in .NET](https://learn.microsoft.com/dotnet/standard/base-types/formatting-types) + +In the following example the number `@fragmentation` will be multiplied by 100, rounded to 2 digits after decimal point, and decorated with percent sign '%'. + +```json +"message": "Current fragmentation level is @{fragmentation:P2}" +``` + +Output: + +```plain +Current fragmentation level is 37.20% +``` + +Formatting for string values is extended. The format string will inserted into the resulting message if the value is not null and not an empty string. The pound sign '#' is replaced by the string value itself. This could be useful for adding sentence parts when additional information is present. + +Take the following example: + +```json +"message": "Create index on @{Table} with key columns @{KeyCols}@{IncludedCols: and included columns: #}" +``` + +When `@IncludedCols` is empty the message would be + +```plain +Create index on MyTable with key columns Id, Name +``` + +When `@IncludedCols` is contains names of included columns "_SpaceAvailable, CPULoad_", tha message will be: + +```plain +Create index on MyTable with key columns Id, name and included columns: SpaceAvailable, CPULoad +``` diff --git a/samples/manage/sql-assessment-api/docs/Customization/Probe.md b/samples/manage/sql-assessment-api/docs/Customization/Probe.md index 3b429f53e8..1f2b077af7 100644 --- a/samples/manage/sql-assessment-api/docs/Customization/Probe.md +++ b/samples/manage/sql-assessment-api/docs/Customization/Probe.md @@ -12,7 +12,7 @@ Probes from the default rule set read metadata only, e.g. update logs or server Probe implementation is represented by a JSON object. Its properties define procedures for getting data and selecting appropriate implementation. -![Probe structure](img\ProbeStructure.svg) +![Probe structure](./img/ProbeStructure.svg) ## Probe properties diff --git a/samples/manage/sql-assessment-api/docs/Customization/ProbeReference.md b/samples/manage/sql-assessment-api/docs/Customization/ProbeReference.md index 46ca1f5e78..af37dc1726 100644 --- a/samples/manage/sql-assessment-api/docs/Customization/ProbeReference.md +++ b/samples/manage/sql-assessment-api/docs/Customization/ProbeReference.md @@ -2,7 +2,7 @@ Data for checks comes from probes. Probe reference is a JSON object that describes a probe needed by the check. When a probe is referenced with no additional options or parameters, a probe reference may be abbreviated to a string containing probe ID. -![Probe reference structure](img\ProbeRefStructure.svg) +![Probe reference structure](./img/ProbeRefStructure.svg) ## Properties @@ -94,9 +94,4 @@ In the following example *DatabaseMasterFiles* probe finds volume IDs for all di } } ] -``` - - - - - +``` \ No newline at end of file diff --git a/samples/manage/sql-assessment-api/docs/Customization/Rule.md b/samples/manage/sql-assessment-api/docs/Customization/Rule.md index bafde2c845..6b5367db87 100644 --- a/samples/manage/sql-assessment-api/docs/Customization/Rule.md +++ b/samples/manage/sql-assessment-api/docs/Customization/Rule.md @@ -2,7 +2,7 @@ A rule is represented by a JSON object. A rule applies its properties to a new or existing check. -![Rule structure](img\RuleStructure.svg) +![Rule structure](./img/RuleStructure.svg) ## Rule properties @@ -24,11 +24,11 @@ An array of IDs or tags of checks to be modified by this rule. A rule can modify ### target -Target pattern is a JSON object used to select applicable rules. A rule will be applied if the assessed object matches the `target` pattern. For more information, see [Target pattern](). +Target pattern is a JSON object used to select applicable rules. A rule will be applied if the assessed object matches the `target` pattern. For more information, see [Target pattern](./TargetPattern.md). ### targetFilter -Target pattern with the same syntax as the target property, but used exclusively in overrides. For more information on the usage and object format, see [Overrides]() and [Target pattern](), respectively. +Target pattern with the same syntax as the target property, but used exclusively in overrides. For more information on the usage and object format, see [Overrides](./RulesandProbes.md) and [Target pattern](./TargetPattern.md), respectively. ### displayName @@ -36,7 +36,7 @@ Short display names are shown in checklists. You may think of a display name as ### message -A string which is used as a template for generating messages to the user. Such a message appears when the check detects configurations that are not in compliance with the best practice recommendation. For more information, see [Message template](). +A string which is used as a template for generating messages to the user. Such a message appears when the check detects configurations that are not in compliance with the best practice recommendation. For more information, see [Message template](./MessageTemplate.md). ### description @@ -54,7 +54,7 @@ The severity level of the issue detected by this check. ### probes -The `probes` array contains references to probes to be used to get data required by this check. For more information, see [Probe References](). +The `probes` array contains references to probes to be used to get data required by this check. For more information, see [Probe References](./ProbeReference.md). ### condition diff --git a/samples/manage/sql-assessment-api/docs/Customization/RulesandProbes.md b/samples/manage/sql-assessment-api/docs/Customization/RulesandProbes.md index b1a64e7ca5..025d22e3f6 100644 --- a/samples/manage/sql-assessment-api/docs/Customization/RulesandProbes.md +++ b/samples/manage/sql-assessment-api/docs/Customization/RulesandProbes.md @@ -1,6 +1,6 @@ # Understanding rules and probes -The SQL Assessment API uses sets of best practice recommendations to check if a SQL Server environment or configuration could be improved. Best practices are not universal. Guidelines depend on SQL Server version, edition, and configuration, hosting platform, and even usage pattern. For example, some best practices are applicable only to cloud environments, whereas others work for SQL Server instances installed on a physical machine. Databases may also be of different types. For example, a set of best practices for the `msdb` database might be different from that used for user databases. +The SQL Assessment API uses sets of best practice recommendations to check if a SQL Server environment or configuration could be improved. Best practices are not universal. Guidelines depend on SQL Server version, edition, and configuration, hosting platform, and even usage pattern. For example, some best practices are applicable only to cloud environments, whereas others work for SQL Server instances installed on a physical machine. Databases may also be of different types. For example, a set of best practices for the `msdb` database might be different from that used for user databases. SQL Assessment API makes recommendations more specific by implementing a two-step process: @@ -82,7 +82,7 @@ The following rule disables the `MaxMemory` check and all performance-related ch Each check needs data to analyze. Checks do not retrieve any data from the target server or database but refer to *probes* instead. Most of the probes use T-SQL queries, but the SQL Assessment API also supports probes for WMI, Windows registry, Azure Instance Metadata Service, as well as custom probes implemented as .NET classes. -Each probe returns zero or more data rows containing named items. If a check gets data from multiple probes, the resulting data set is constructed as *all* combinations of rows. This behavior is similar to `CROSS JOIN` in T-SQL. +Each probe returns zero or more data rows containing named items. If a check gets data from multiple probes, the resulting data set is constructed as *all* combinations of rows. This behavior is similar to `CROSS JOIN` in T-SQL. A check has a condition expression which formally describes best practice as an arithmetic expression referring to data from a data row. Condition is calculated and checked if it holds for each row separately. For example, when a database uses 3 disks for storing its files, then a free space best practice may apply to each disk independently. If a check uses data from two probes, and one probe produces 2 rows while another one produces 3, the check condition will be evaluated 2×3=6 times. If condition gives *false* 2 times out of 6, the check will produce 2 messages. diff --git a/samples/manage/sql-assessment-api/docs/Customization/RulesetFileStructure.md b/samples/manage/sql-assessment-api/docs/Customization/RulesetFileStructure.md index 7f2c48159c..72c4735b9c 100644 --- a/samples/manage/sql-assessment-api/docs/Customization/RulesetFileStructure.md +++ b/samples/manage/sql-assessment-api/docs/Customization/RulesetFileStructure.md @@ -3,13 +3,13 @@ Rule set is stored in a text file that contains a JSON object. The rule set object has three simple mandatory properties: `name`, `version`, and `schemaVersion`. The latter specifies the file format version. As of the time of writing this document, the file format version is 1.0. The `name` and `version` properties comprise the identity of the set. -An essential part of a ruleset is the optional `rules` property that stores an array of SQL Assessment API rules. Rules are used to build a checklist. For more information, see [Understanding Rules and Probes](.\RulesandProbes.md). +An essential part of a ruleset is the optional `rules` property that stores an array of SQL Assessment API rules. Rules are used to build a checklist. For more information, see [Understanding Rules and Probes](./RulesandProbes.md). -The optional `probes` property contains definitions for probes. Probes are used to get actual data from target SQL Server instances, hosting machines, and other sources. Each probe definition is a JSON array of one or more implementations. For more information, see [Probes](..\Reference\Probes\README.md). +The optional `probes` property contains definitions for probes. Probes are used to get actual data from target SQL Server instances, hosting machines, and other sources. Each probe definition is a JSON array of one or more implementations. For more information, see [Probes](../Reference/Probes/README.md). The `rules` and `probes` properties are optional because rules from one ruleset can use probes from another one. -![Ruleset Structure](img\RulesetStructure.svg) +![Ruleset Structure](./img/RulesetStructure.svg) ```json { diff --git a/samples/manage/sql-assessment-api/docs/Customization/TargetPattern.md b/samples/manage/sql-assessment-api/docs/Customization/TargetPattern.md new file mode 100644 index 0000000000..e5a5829cd5 --- /dev/null +++ b/samples/manage/sql-assessment-api/docs/Customization/TargetPattern.md @@ -0,0 +1,154 @@ +# Target pattern + +Target pattern specifies criteria for filtering rules and probe implementations. A rule will be applied to those SQL Server instances and databases, which match the rule's [target](./Rule.md#target) pattern. + +this mechanism allows rules specific to certain SQL Server versions or editions, or even specific to a certain database. When used in custom rulesets, SQL Server instances can be distinguished by name to apply different rules. For example, servers servers used for application development might not need frequent backups. + +## Target pattern structure + +Target pattern is represented by a JSON object of the following structure. All properties are optional. Every value is a match when a property is omitted. + +![Target pattern structure](./img/TargetPatternStructure.svg) + +### engineEdition + +__Allowed values__: `PersonalOrDesktopEngine`, `Standard`, `Enterprise`, `Express`, `AzureDatabase`, `DataWarehouse`, `StretchDatabase`, `ManagedInstance`; + +`Azure` = `AzureDatabase`, `DataWarehouse`, `StretchDatabase`, `ManagedInstance`; + +`SqlServer` = `PersonalOrDesktopEngine`, `Standard`, `Enterprise`, `Express`. + +Any combination of comma-separated values above is allowed. Lists SQL Server engine editions matching this target pattern. `Azure` and `SqlServer` are short names for frequently used lists. + +In the following example __engineEdition__ is any on-premises edition or Azure Managed Instance. + +```json +"engineEdtion": "SqlServer, ManagedInstance" +``` + +### name + +A [string pattern](#string-pattern) for the target SQL +Server object name. It's a database name or an instance name. + +In the following example __name__ matches anything except _master_, _tempdb_, and _model_. + +```json +"name": { "not": [ "master", "tempdb", "model"]} +``` + +### machineType + +A [string pattern](#string-pattern) for the type of machine hosting target SQL Server instance. + +Supported machine type values are: `Physical`, `AzureVm`, `Hypervisor`, `Other`. + +In the following example any physical or Azure virtual machine mathes the pattern. + +```json +"machineType": ["AzureVm", "Physical"] +``` + +### platform + +A [string pattern](#string-pattern) for the platform of machine hosting target SQL Server instance. + +Supported machine type values are: `Windows`, `Linux`. + +In the following example Windows and Linux installations match. + +```json +"platform": ["Windows", "Linux"] +``` + +### serverName + +A [string pattern](#string-pattern) for the SQL Server instance name. If target is a SQL SErver instance, then _serverName_ equals _name_, otherwise it's the name of SQL Server instance hosting the target object. + +In the following example all instances match except those with names ending with _Test_. + +```json +"serverName": {"not": "/*.Test$/"} +``` + +### type + +__Allowed values__: `Server`, `Database`. + +Any comma-separated list of allowed values defines which type of SQL Server object matches this pattern. + +Check's target is always either `Sever` or `Database`. For a probe it's often both. + +In the following example only databases match the pattern. + +```json +"type": "Database" +``` + +### version + +A [version pattern](#version-pattern) specifying ranges of matching SQL Server versions. + +In the following example the pattern matches SQL Server 2016 SP2+, 2014 SP3+, and 2019+. + +```json +"version": ["[13.0.5026.0, 14.0)", "[12.0.6024.0, 13.0)", "[15.0)"] +``` + +## String pattern + +String pattern defines matching strings. It can be a string, a JSON object, or a JSON array of string patterns. With arrays, string patterns have a recursive structure. An array matches a string when any of its elements is a match. + +A string pattern can be an arbitrary string. In this case SQL Assessment API looks for an exact match. + +When the string starts and ends with slash '/' ti represents a [regular expression](#regular-expression). + +When string pattern is represented by a JSON object, it may have only one property named __not__. The property value must be any string pattern. This means that the pattern matches anything which is not a match for the __not__property value. + +For example, the following matches anything that does not end with _Test_. + +```json +{ "not": "/*.Test$/c" } +``` + +### Regular expression + +Regular expressions are enclosed in slashes: `"/Win*./"`. Regular expression options can be specified after the closing slash. For example, this is the case sensitive search: `"/win.*/c"`. If the string does not start with a slash, it is treated as an exact string match. The string "_Linux_" is equivalent to the regular expression "_/^Linux$/_". + +SQL Assessment API supports [.NET Regular Expressions.](https://learn.microsoft.com/dotnet/standard/base-types/regular-expressions). + +Regular expression options are listed in the [Regular expression options](https://learn.microsoft.com/dotnet/standard/base-types/regular-expression-options) article. + +__NOTE:__ Regular expressions are case-insensitive by default. For case-sensitive matching use 'c' option. + + +## Version pattern + +The version range list is a single [version range](#version-range) or an array of version ranges. A single version range is equivalent to an array containing the only element. The version range list matches any version matching any of its ranges. + +```json +"version": [ + "[10.0.4326,10.0.4371]", + "[10.0.5794,10.50)", + "[10.50.2806,11.0)", + "[11.0.2316,)" +] +``` + +### Version range + +A range of versions is encoded by a string. The string contains one or two versions delimited by the comma and enclosed into optional parenthesis or brackets. If there are two versions, the former version must be less than or equal to the latter. They represent the range boundaries. The version must be specified at least by two numbers separated by the period: major and minor version numbers. + +Th version range format follows [Nuget version range format](https://learn.microsoft.com/nuget/concepts/package-versioning#version-ranges). + +|Version Range|Matches| +|-|-| +|"10.0"|Version 10.0 exactly.| +|"[10.0, 13.0]"|Anything between 10.0 and 13.0, inclusive: 10.0, 10.50, 11.0.345, 13.0.| +|"(10.0, 13.0)"|Anything between 10.0 and 13.0, exclusive: 10.50, 11.0.345. Does not match 10.0 or 13.0.| +|"[10.0, 13.1)"|Anything between 10.0 and 13.1, excluding the right boundary: 10.0, 10.50, 11.0.345, 13.0, 13.0.234. Does not match 13.1.| +|"(10.0, 13.1]"|Anything between 10.0 and 13.1, excluding 10.0.| +|"[10.0,)"|10.0 and abo| +|"(10.0,)"|Above 10.0, but not itself| +|"(,10.0]"|10.0 or below| +|"(,10.0)"|Below 10.0, but not 10.0.| diff --git a/samples/manage/sql-assessment-api/docs/Customization/img/TargetPatternStructure.svg b/samples/manage/sql-assessment-api/docs/Customization/img/TargetPatternStructure.svg new file mode 100644 index 0000000000..b86ad69ba3 --- /dev/null +++ b/samples/manage/sql-assessment-api/docs/Customization/img/TargetPatternStructure.svg @@ -0,0 +1,21 @@ +type?Server|Databaseversion?Version range listengineEdition?Edition listplatform?String patternname?String patternserverName?String patternmachineType?String pattern \ No newline at end of file diff --git a/samples/manage/sql-assessment-api/docs/Customization/img/src/TargetPatternStructure.plantuml b/samples/manage/sql-assessment-api/docs/Customization/img/src/TargetPatternStructure.plantuml new file mode 100644 index 0000000000..0423247a81 --- /dev/null +++ b/samples/manage/sql-assessment-api/docs/Customization/img/src/TargetPatternStructure.plantuml @@ -0,0 +1,11 @@ +@startjson +{ + "type?": "Server | Database", + "version?": "Version range list", + "engineEdition?": "Edition list", + "platform?": "String pattern", + "name?": "String pattern", + "serverName?": "String pattern", + "machineType?": "String pattern" +} +@end \ No newline at end of file diff --git a/samples/manage/sql-assessment-api/docs/How-To/UsingNonSQLProbes.md b/samples/manage/sql-assessment-api/docs/How-To/UsingNonSQLProbes.md index eff811c385..55857cd448 100644 --- a/samples/manage/sql-assessment-api/docs/How-To/UsingNonSQLProbes.md +++ b/samples/manage/sql-assessment-api/docs/How-To/UsingNonSQLProbes.md @@ -1,6 +1,6 @@ # Retrieving Data from Operating System -In the SQL Assessment API, most of the probes use T-SQL to get data for assessment. However, there are probes that obtain data from the operating system not presented in SQL Server dynamic management views. In order for these probes to get data, the [xp_cmdshell](https://docs.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/xp-cmdshell-transact-sql) stored procedure and PowerShell access should be enabled on the target SQL Server. While these facilities are disabled, some checks may be ckipped. +In the SQL Assessment API, most of the probes use T-SQL to get data for assessment. However, there are probes that obtain data from the operating system not presented in SQL Server dynamic management views. In order for these probes to get data, the [xp_cmdshell](https://docs.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/xp-cmdshell-transact-sql) stored procedure and PowerShell access should be enabled on the target SQL Server. While these facilities are disabled, some checks may be skipped. Keep in mind that the [xp_cmdshell](https://docs.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/xp-cmdshell-transact-sql) stored procedure should be enabled temporarily as it is not recommended by the best practices; once you complete the assessment, make sure to disable it. diff --git a/samples/manage/sql-assessment-api/docs/Reference/DataTransformation/nameValuePairs.md b/samples/manage/sql-assessment-api/docs/Reference/DataTransformation/nameValuePairs.md index 2ba11c0961..dab7fe3db9 100644 --- a/samples/manage/sql-assessment-api/docs/Reference/DataTransformation/nameValuePairs.md +++ b/samples/manage/sql-assessment-api/docs/Reference/DataTransformation/nameValuePairs.md @@ -42,4 +42,4 @@ Source table: Result table: |cpu_utilization|NOPM|disk_space|TPM| |---|---|---|---| -|0.65|120|15|11236| \ No newline at end of file +|0.65|120|15|11236| diff --git a/samples/manage/sql-assessment-api/docs/Reference/DataTransformation/parse.md b/samples/manage/sql-assessment-api/docs/Reference/DataTransformation/parse.md index 074aa140e2..988510ff6f 100644 --- a/samples/manage/sql-assessment-api/docs/Reference/DataTransformation/parse.md +++ b/samples/manage/sql-assessment-api/docs/Reference/DataTransformation/parse.md @@ -134,4 +134,4 @@ Joined: | @a | @b | @c | @notMentioned | | :-- | :-- | :-- | :-: | -| "42, 13, 0" | "Lorem, ipsum, dolor" | "true, false" | - | \ No newline at end of file +| "42, 13, 0" | "Lorem, ipsum, dolor" | "true, false" | - | diff --git a/samples/manage/sql-assessment-api/docs/Reference/Operators.md b/samples/manage/sql-assessment-api/docs/Reference/Operators.md index 6bdb310f9d..d3bf075fd7 100644 --- a/samples/manage/sql-assessment-api/docs/Reference/Operators.md +++ b/samples/manage/sql-assessment-api/docs/Reference/Operators.md @@ -3,6 +3,7 @@ With the SQL Assessment API, you can use different kinds of operators to make your assessment even more precise. ## In This Section + - [Logical](#logical) - [String](#string) - [Math](#math) @@ -67,4 +68,4 @@ With the SQL Assessment API, you can use different kinds of operators to make yo |ine||(*a*, *b*)|Checks if the argument *a* is not equal to the argument *b*. Case-insensitive.| |match||(*a*, *b*)|Regular expression match. The second argument is treated as a regular expression. Case-sensitive.| |imatch||(*a*, *b*)|Regular expression match. The second argument is treated as a regular expression. Case-insensitive.| -|interval||(*a*,*v1*,*t1*,...,*vn*,*tn*,*d*)|Finds the first *ti* that is greater than or equal to *a* and returns the corresponding *vi*. Returns *d* if all *t* are less than *a*.| \ No newline at end of file +|interval||(*a*,*v1*,*t1*,...,*vn*,*tn*,*d*)|Finds the first *ti* that is greater than or equal to *a* and returns the corresponding *vi*. Returns *d* if all *t* are less than *a*.| diff --git a/samples/manage/sql-assessment-api/docs/Reference/Probes/RegistryProbes.md b/samples/manage/sql-assessment-api/docs/Reference/Probes/RegistryProbes.md index fbad766d09..1ddf01d09a 100644 --- a/samples/manage/sql-assessment-api/docs/Reference/Probes/RegistryProbes.md +++ b/samples/manage/sql-assessment-api/docs/Reference/Probes/RegistryProbes.md @@ -43,7 +43,7 @@ where **MSSQL12.SQL2014** is the SQL Server instance name. ### Example 1 -The following registry query returns 2 values for for system hardware. Please, note how multi-string value is handled as multiple strings. Use [join aggregate function](..\DataMorphs\aggregate.md#join) to merge them when needed. +The following registry query returns 2 values for for system hardware. Please, note how multi-string value is handled as multiple strings. Use [join aggregate function](../DataMorphs/aggregate.md#join) to merge them when needed. ```json "probes": { diff --git a/samples/manage/sql-assessment-api/docs/Reference/Probes/TSQLProbes.md b/samples/manage/sql-assessment-api/docs/Reference/Probes/TSQLProbes.md index c6bacbbf0c..6003c13c96 100644 --- a/samples/manage/sql-assessment-api/docs/Reference/Probes/TSQLProbes.md +++ b/samples/manage/sql-assessment-api/docs/Reference/Probes/TSQLProbes.md @@ -34,4 +34,4 @@ Implementation part of the probe definition contains the following parameters. } } } -``` \ No newline at end of file +``` diff --git a/samples/manage/sql-assessment-api/docs/Tutorials/OverridingDefaultThresholds.md b/samples/manage/sql-assessment-api/docs/Tutorials/OverridingDefaultThresholds.md index 72815d44bc..8d78bc58c5 100644 --- a/samples/manage/sql-assessment-api/docs/Tutorials/OverridingDefaultThresholds.md +++ b/samples/manage/sql-assessment-api/docs/Tutorials/OverridingDefaultThresholds.md @@ -54,7 +54,7 @@ Let's say you need to change this rule to validate backups that were created mor Then you can save this file to whatever place you need and invoke the assessment, as the following example demonstrates. ```PowerShell -Invoke-SqlAssessment -Check FullBackup -Configuration .\BackupPolicy.json | Select TagerObject, Message +Invoke-SqlAssessment -Check FullBackup -Configuration .\BackupPolicy.json | Select TargetObject, Message ``` Here, you are basically using the same command, as shown in the example above, but extended with `-Configuration .\BackupPolicy.json`. This command instructs the assessment to use a custom configuration file that keeps your overrides.