Validate all README code snippets. #706
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
One of the ElasticGraph guiding principles is that all documentation should be validated, so that our CI build notifies us when it goes stale and no longer works. It's important that code snippets in READMEs remain working and accurate. To that end, this introduces `script/validate_readme_snippets`. Here's how it works: * A new ElasticGraph project is bootstrapped into `tmp/example_project_for_snippet_validation`. * `bash` snippets are executed from that project directory. * `diff` snippets are applied to that project directory. Then `bundle exec rake` is executed to confirm the build still passes. * `ruby` snippets are dumped into a file (either at a temp path, or at the path identified in a comment) and then executed. * `yaml` snippets are parsed as YAML. * `mermaid` snippets are skipped, because they are generated (and verified) by `script/update_dependency_diagrams`, so we can trust they are valid. * `text` snippets (that is, snippets which don't indicate a type) are not validated. (So a simple way to get the build to pass for a failing snippet is to remove the type, so that it's just text). We have a number of snippets that fail. This fixes the simple ones and converts the ones that can't simply be fixed to `text` so that they are not validated as part of the build, allowing the build to pass. I'll be overhauling the READMEs with updated snippets in a later PR.
|
|
jwondrusch
approved these changes
Jul 20, 2025
|
|
||
| require "yaml" | ||
|
|
||
| class YamlSnippetValidator < SnippetValidator |
Collaborator
There was a problem hiding this comment.
idea (non-blocking): maybe someday the actual yaml could be validated against a schema rather than just confirming it loads.
Collaborator
Author
There was a problem hiding this comment.
Agree. Right now, settings are validated via adhoc logic (example) and I'd like to make it so that config validation is defined using JSON schema. Then we can use that here.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
One of the ElasticGraph guiding principles is that all documentation should be validated, so that our CI build notifies us when it goes stale and no longer works. It's important that code snippets in READMEs remain working and accurate.
To that end, this introduces
script/validate_readme_snippets. Here's how it works:tmp/example_project_for_snippet_validation.bashsnippets are executed from that project directory.diffsnippets are applied to that project directory. Thenbundle exec rakeis executed to confirm the build still passes.rubysnippets are dumped into a file (either at a temp path, or at the path identified in a comment) and then executed.yamlsnippets are parsed as YAML.mermaidsnippets are skipped, because they are generated (and verified) byscript/update_dependency_diagrams, so we can trust they are valid.textsnippets (that is, snippets which don't indicate a type) are not validated. (So a simple way to get the build to pass for a failing snippet is to remove the type, so that it's just text).We have a number of snippets that fail. This fixes the simple ones and converts the ones that can't simply be fixed to
textso that they are not validated as part of the build, allowing the build to pass.I'll be overhauling the READMEs with updated snippets in a later PR.