Skip to content

Commit

Permalink
Rewrite Validate Command (#341)
Browse files Browse the repository at this point in the history
* Initial validate implementation

* Add function tests to TestFileMismatchCheck

* Add MixedDirectoryCheck and refactor relevant tests

* Add functions and guides to glob pattern

* Update README.md

* Update documentationDirGlob to pass tests

* Add copyright headers

* Resolve linting errors

* Update README.md with a link to provider documentation guidelines

* Resolve linting errors

* Add changelog entries

* Add `NumberOfFilesCheck` to validate

* Make file paths OS-agnostic

* Fix file path in test error assertion

* Update Go version to `1.21`

* Change wording to past tense on changelog entries

* Remove unused file

* Remove `filecheck` interface

* Refactor `schemamd` and `functionmd` packages to `internal/`

* Remove unused method

* Revert "Refactor `schemamd` and `functionmd` packages to `internal/`"

This reverts commit a1b156e.

* Remove go compatability section

* Switch to `yuin/goldmark` for frontmatter parsing

* Remove unused test

* Move global variables to top of file

* Refactor `testcases` to use `map[string]struct`

* Refactor to use `t.tempDir()`

* Fix documentationGlobPattern and add more debug and info logs

* Add negative scenario acceptance tests
  • Loading branch information
SBGoods committed Mar 27, 2024
1 parent 221ea1b commit 836a890
Show file tree
Hide file tree
Showing 108 changed files with 4,271 additions and 214 deletions.
5 changes: 5 additions & 0 deletions .changes/unreleased/FEATURES-20240305-135426.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: FEATURES
body: 'validate: Added support for Provider-defined Function documentation to all checks'
time: 2024-03-05T13:54:26.307742-05:00
custom:
Issue: "341"
6 changes: 6 additions & 0 deletions .changes/unreleased/FEATURES-20240305-135726.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: FEATURES
body: 'validate: Added `InvalidDirectoriesCheck` which checks for valid provider documentation
folder structure'
time: 2024-03-05T13:57:26.273538-05:00
custom:
Issue: "341"
6 changes: 6 additions & 0 deletions .changes/unreleased/FEATURES-20240305-135933.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: FEATURES
body: 'validate: Added `MixedDirectoriesCheck` which throws an error if both legacy
documentation and registry documentation are found'
time: 2024-03-05T13:59:33.741601-05:00
custom:
Issue: "341"
6 changes: 6 additions & 0 deletions .changes/unreleased/FEATURES-20240305-140106.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: FEATURES
body: 'validate: Added `NumberOfFilesCheck` which checks the number of provider
documentation files against the registry limit'
time: 2024-03-05T14:01:06.742843-05:00
custom:
Issue: "341"
6 changes: 6 additions & 0 deletions .changes/unreleased/FEATURES-20240305-140234.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: FEATURES
body: 'validate: Added `FileSizeCheck` which checks the provider documentation file
size against the registry limit'
time: 2024-03-05T14:02:34.112782-05:00
custom:
Issue: "341"
6 changes: 6 additions & 0 deletions .changes/unreleased/FEATURES-20240305-140346.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: FEATURES
body: 'validate: Added `FileExtensionCheck` which checks for valid provider documentation
file extensions'
time: 2024-03-05T14:03:46.816256-05:00
custom:
Issue: "341"
6 changes: 6 additions & 0 deletions .changes/unreleased/FEATURES-20240305-140451.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: FEATURES
body: 'validate: Added `FrontMatterCheck` which checks the YAML frontmatter of provider
documentation for missing required fields or invalid fields'
time: 2024-03-05T14:04:51.781688-05:00
custom:
Issue: "341"
6 changes: 6 additions & 0 deletions .changes/unreleased/FEATURES-20240305-140622.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: FEATURES
body: 'validate: Added `FileMismatchCheck` which checks the names/number of provider
documentation files against the provider schema'
time: 2024-03-05T14:06:22.168518-05:00
custom:
Issue: "341"
23 changes: 22 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Available commands are:
the generate command is run by default
generate generates a plugin website from code, templates, and examples
migrate migrates website files from either the legacy rendered website directory (`website/docs/r`) or the docs rendered website directory (`docs/resources`) to the tfplugindocs supported structure (`templates/`).
validate validates a plugin website for the current directory
validate validates a plugin website
```
Expand Down Expand Up @@ -81,6 +81,11 @@ Usage: tfplugindocs generate [<args>]
$ tfplugindocs validate --help
Usage: tfplugindocs validate [<args>]
--provider-dir <ARG> relative or absolute path to the root provider code directory; this will default to the current working directory if not set
--provider-name <ARG> provider name, as used in Terraform configurations
--providers-schema <ARG> path to the providers schema JSON file, which contains the output of the terraform providers schema -json command. Setting this flag will skip building the provider and calling Terraform CLI
--tf-version <ARG> terraform binary version to download. If not provided, will look for a terraform binary in the local environment. If not found in the environment, will download the latest version of Terraform
```
`migrate` command:
Expand Down Expand Up @@ -145,6 +150,22 @@ Otherwise, the provider developer can set an arbitrary description like this:
// ...
```

#### Validate subcommand

The `validate` subcommand can be used to validate the provider website documentation against the [Terraform Registry's provider documentation guidelines](https://developer.hashicorp.com/terraform/registry/providers/docs) and provider documentation best practices. The current checks in the `validate` command are:
| Check | Description |
|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `InvalidDirectoriesCheck` | Checks for valid subdirectory structure and throws an error if an invalid Terraform Provider documentation subdirectory is found. |
| `MixedDirectoriesCheck` | Throws an error if both legacy documentation (`/website/docs`) and registry documentation (`/docs`) are found. |
| `NumberOfFilesCheck` | Throws an error if the number of files in a directory is larger than the registry limit. |
| `FileSizeCheck` | Throws an error if the documentation file is above the registry storage limit. |
| `FileExtensionCheck` | Throws an error if the extension of the given file is not a valid registry documentation extension. |
| `FrontMatterCheck` | Checks the YAML frontmatter of documentation for missing required fields or invalid fields. |
| `FileMismatchCheck` | Throws an error if the names/number of resources/datasources/functions in the provider schema does not match the names/number of files in the corresponding documentation directory |
All check errors are wrapped and returned as a single error message to stderr.
#### Migrate subcommand
The `migrate` subcommand can be used to migrate website files from either the legacy rendered website directory (`website/docs/r`) or the docs
Expand Down
8 changes: 8 additions & 0 deletions cmd/tfplugindocs/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,3 +54,11 @@ func Test_SchemaJson_MigrateAcceptanceTests(t *testing.T) {
Dir: "testdata/scripts/schema-json/migrate",
})
}

func Test_SchemaJson_ValidateAcceptanceTests(t *testing.T) {
t.Parallel()

testscript.Run(t, testscript.Params{
Dir: "testdata/scripts/schema-json/validate",
})
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: MPL-2.0

# Run of tfplugindocs validate command with a misnamed file
[!unix] skip
! exec tfplugindocs validate --provider-name=terraform-provider-scaffolding --providers-schema=schema.json
stderr 'Error executing command: validation errors found:'
stderr 'matching resource for documentation file \(resource2.md\) not found, file is extraneous or incorrectly named'
stderr 'missing documentation file for resource: scaffolding_example'

-- docs/data-sources/example.md --
---
subcategory: "Example"
page_title: "Example: example_thing"
description: |-
Example description.
---
# Data Fields

Name: {{.Name}}
Type: {{.Type}}
-- docs/resources/resource2.md --
---
subcategory: "Example"
page_title: "Example: example_thing"
description: |-
Example description.
---
# Data Fields

Name: {{.Name}}
Type: {{.Type}}
-- schema.json --
{
"format_version": "1.0",
"provider_schemas": {
"registry.terraform.io/hashicorp/scaffolding": {
"provider": {
"version": 0,
"block": {
"attributes": {
"endpoint": {
"type": "string",
"description": "Example provider attribute",
"description_kind": "markdown",
"optional": true
}
},
"description": "Example provider",
"description_kind": "markdown"
}
},
"resource_schemas": {
"scaffolding_example": {
"version": 0,
"block": {
"attributes": {
"configurable_attribute": {
"type": "string",
"description": "Example configurable attribute",
"description_kind": "markdown",
"optional": true
},
"defaulted": {
"type": "string",
"description": "Example configurable attribute with default value",
"description_kind": "markdown",
"optional": true,
"computed": true
},
"id": {
"type": "string",
"description": "Example identifier",
"description_kind": "markdown",
"computed": true
}
},
"description": "Example resource",
"description_kind": "markdown"
}
}
},
"data_source_schemas": {
"scaffolding_example": {
"version": 0,
"block": {
"attributes": {
"configurable_attribute": {
"type": "string",
"description": "Example configurable attribute",
"description_kind": "markdown",
"optional": true
},
"id": {
"type": "string",
"description": "Example identifier",
"description_kind": "markdown",
"computed": true
}
},
"description": "Example data source",
"description_kind": "markdown"
}
}
},
"functions": {
"example": {
"description": "Given a string value, returns the same value.",
"summary": "Echo a string",
"return_type": "string",
"parameters": [
{
"name": "input",
"description": "Value to echo.",
"type": "string"
}
],
"variadic_parameter": {
"name": "variadicInput",
"description": "Variadic input to echo.",
"type": "string"
}
}
}
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: MPL-2.0

# Run of tfplugindocs validate command with mixed directory structure
[!unix] skip
! exec tfplugindocs validate --provider-name=terraform-provider-scaffolding --providers-schema=schema.json
stderr 'Error executing command: validation errors found:'
stderr 'mixed Terraform Provider documentation directory layouts found, must use only legacy or registry layout'

-- website/docs/d/example.html.md --
---
subcategory: "Example"
layout: "example"
page_title: "Example: example_thing"
description: |-
Example description.
---
# Data Fields

Name: {{.Name}}
Type: {{.Type}}
-- docs/data-sources/example.md --
---
subcategory: "Example"
page_title: "Example: example_thing"
description: |-
Example description.
---
# Data Fields

Name: {{.Name}}
Type: {{.Type}}
-- schema.json --
{
"format_version": "1.0",
"provider_schemas": {
"registry.terraform.io/hashicorp/scaffolding": {
"provider": {
"version": 0,
"block": {
"attributes": {
"endpoint": {
"type": "string",
"description": "Example provider attribute",
"description_kind": "markdown",
"optional": true
}
},
"description": "Example provider",
"description_kind": "markdown"
}
},
"resource_schemas": {
"scaffolding_example": {
"version": 0,
"block": {
"attributes": {
"configurable_attribute": {
"type": "string",
"description": "Example configurable attribute",
"description_kind": "markdown",
"optional": true
},
"defaulted": {
"type": "string",
"description": "Example configurable attribute with default value",
"description_kind": "markdown",
"optional": true,
"computed": true
},
"id": {
"type": "string",
"description": "Example identifier",
"description_kind": "markdown",
"computed": true
}
},
"description": "Example resource",
"description_kind": "markdown"
}
}
},
"data_source_schemas": {
"scaffolding_example": {
"version": 0,
"block": {
"attributes": {
"configurable_attribute": {
"type": "string",
"description": "Example configurable attribute",
"description_kind": "markdown",
"optional": true
},
"id": {
"type": "string",
"description": "Example identifier",
"description_kind": "markdown",
"computed": true
}
},
"description": "Example data source",
"description_kind": "markdown"
}
}
},
"functions": {
"example": {
"description": "Given a string value, returns the same value.",
"summary": "Echo a string",
"return_type": "string",
"parameters": [
{
"name": "input",
"description": "Value to echo.",
"type": "string"
}
],
"variadic_parameter": {
"name": "variadicInput",
"description": "Variadic input to echo.",
"type": "string"
}
}
}
}
}
}
Loading

0 comments on commit 836a890

Please sign in to comment.