Validate all README code snippets.

MAINTAINERS_RUNBOOK.md

@@ -272,3 +272,24 @@ script/run_gem_specs elasticgraph-newgem
     pending trusted publisher on rubygems.org (as that's how long a new pending trusted publisher is valid for). Use the release process
     explained at the top of this document, but be sure to use a release candidate version (e.g. `1.2.3.rc1`). We don't want to be trying out
     the publishing of a new gem for the first time when cutting a final release.
+
+## READMEs
+
+The project has a root README and a README within each gem directory. Much of the content of the READMEs is
+generated or validated to ensure accurate documentation.
+
+* The `mermaid` dependency diagrams are automatically generated by `script/update_dependency_diagrams`.
+  CI checks them with `script/update_dependency_diagrams --verify`.
+* All other README code snippets are validated by `script/validate_readme_snippets`. Here's how that validation works:
+  * A new ElasticGraph project is bootstrapped into `tmp/example_project_for_snippet_validation`.
+  * `bash` snippets are executed from that project directory.
+  * `diff` snippets are applied to that project directory. Then `bundle exec rake` is executed to confirm the build still passes.
+  * `ruby` snippets are dumped into a file (either at a temp path, or at the path identified in a comment) and then executed.
+  * `yaml` snippets are parsed as YAML.
+  * `text` snippets (that is, snippets which don't indicate a type) are not validated. (So a simple way to get the build to pass for a failing snippet is to remove the type, so that it's just text).
+
+When updating a README file, you may want to run the validator locally:
+
+```bash
+script/validate_readme_snippets --file elasticgraph-somegem/README.md
+```

ai-memory/README.md

@@ -2,6 +2,16 @@
 
 This document serves as the primary, single source of truth for understanding the ElasticGraph project.
 
+## Development Requirements for AI Agents
+
+**CRITICAL - Ruby Code Standards**: All AI agents working on this project MUST run `bundle exec standardrb --fix` on any Ruby code before completing tasks. This is a mandatory step in the ElasticGraph project workflow. StandardRB linting should be applied to:
+- New Ruby scripts in script/ directory
+- Modified Ruby files
+- Any Ruby code changes
+- Before considering any Ruby coding task complete
+
+The ElasticGraph project uses StandardRB for consistent code formatting and style enforcement. This step is non-negotiable and should be done automatically as part of the development process.
+
 ## Project Context
 
 -   **Why this project exists**: ElasticGraph is a general purpose, near real-time data query and search platform.
@@ -349,6 +359,7 @@ The `script/` directory contains various scripts for development, maintenance, a
 -   `update_licenses`: Scans and updates license information for project dependencies.
 -   `quick_build`: Provides a potentially faster, possibly less comprehensive, build option for local development.
 -   `spellcheck`: Runs spell-checking across the codebase.
+-   `validate_readme_snippets`: Validates all Ruby code snippets and diff snippets in README.md files throughout the repository, following the ElasticGraph principle of "validating all documentation snippets".
 -   `run_gem_specs`: A utility to execute RSpec tests for individual gems or all gems within the monorepo.
 -   `flatware_rspec`: Likely executes RSpec tests in parallel using the Flatware tool.
 -   `update_ci_yaml`: Automates updates to the GitHub Actions CI workflow configuration (`.github/workflows/ci.yaml`), especially for the matrix of datastore versions being tested. It reads version information from `config/tested_datastore_versions.yaml`.

ai_tools/elasticgraph-mcp-server/README.md

@@ -6,7 +6,7 @@ This provides a Model Context Protocol (MCP) server for [ElasticGraph](https://b
 
 1. Install dependencies:
 
-```bash
+```
 # make install
 uv sync
 
@@ -17,7 +17,7 @@ source .venv/bin/activate
 
 The server runs on port 3000, and though there are no logs displayed, it is actively waiting for input.
 
-```bash
+```
 # make server
 uv pip install .
 elasticgraph-mcp-server
@@ -31,7 +31,7 @@ You can test your MCP server with Anthropic's [Inspector](https://modelcontextpr
 
 1. Run the following command, which starts the server as a subprocess and launches the Inspector UI:
 
-```bash
+```
 # make inspector
 mcp dev src/elasticgraph_mcp/server.py
 ```
@@ -46,7 +46,7 @@ Add a development build to Goose:
 2. Set **Type** to **StandardIO**.
 3. Paste the run command for your local development version, it will start with `uv run </path/to/elasticgraph_mcp/.venv/bin/elasticgraph-mcp-server>`
 
-```bash
+```
 # Copy the run command to your clipboard
 echo "uv run $(realpath .venv/bin/elasticgraph-mcp-server)" | pbcopy
 ```
@@ -60,7 +60,7 @@ Ask goose: What tools and resources for ElasticGraph do you have?
 
 This project uses `make` for common development tasks. To see all available commands, run:
 
-```bash
+```
 make help
 ```
 
@@ -72,7 +72,7 @@ You can use [Goose](https://block.github.io/goose/) to improve this MCP server.
 
 1. **Navigate to this MCP Server Directory:**
 
-```bash
+```
 cd ai_tools/elasticgraph-mcp-server
 ```
 
@@ -94,7 +94,7 @@ touch tmp/mcp_for_llm_instructions.md
 
 4. **Start a Goose Session:**
 
-```bash
+```
 goose session
 ```
 

elasticgraph-apollo/README.md

@@ -34,15 +34,15 @@ graph LR;
 
 First, add `elasticgraph-apollo` to your `Gemfile`:
 
-``` ruby
+```
 gem "elasticgraph-apollo"
 ```
 
 Finally, update your ElasticGraph schema artifact rake tasks in your `Rakefile`
 so that `ElasticGraph::GraphQL::Apollo::SchemaDefinition::APIExtension` is
 passed as one of the `extension_modules`:
 
-``` ruby
+```
 require "elastic_graph/schema_definition/rake_tasks"
 require "elastic_graph/apollo/schema_definition/api_extension"
 
@@ -69,7 +69,7 @@ This library supports multiple versions of Apollo federation. As of Jan. 2024, i
 By default, the newest version is targeted. If you need an older version (e.g. because your organization is
 running an older Apollo version), you can configure it in your schema definition with:
 
-```ruby
+```
 schema.target_apollo_federation_version "2.3"
 ```
 

elasticgraph-graphiql/README.md

@@ -4,11 +4,11 @@ Provides a GraphiQL IDE for ElasticGraph projects.
 
 Here's an example `config.ru`:
 
-``` ruby
+```ruby
 require 'elastic_graph/graphql'
 require 'elastic_graph/graphiql'
 
-graphql = ElasticGraph::GraphQL.from_yaml_file("path/to/config.yaml")
+graphql = ElasticGraph::GraphQL.from_yaml_file("config/settings/local.yaml")
 run ElasticGraph::GraphiQL.new(graphql)
 ```
 

elasticgraph-health_check/README.md

@@ -46,7 +46,7 @@ graph LR;
 
 To use, simply register the `EnvoyExtension` when defining your schema:
 
-```ruby
+```
 require(envoy_extension_path = "elastic_graph/health_check/envoy_extension")
 schema.register_graphql_extension ElasticGraph::HealthCheck::EnvoyExtension,
   defined_at: envoy_extension_path,

elasticgraph-indexer_lambda/README.md

@@ -32,7 +32,7 @@ graph LR;
 We use [JSON Lines](http://jsonlines.org/) to encode our indexing events. It is just individual JSON objects
 delimited by a newline control character(not the `\n` string sequence), such as:
 
-```jsonl
+```
 {"op": "upsert", "__typename": "Payment", "id": "123", "version": "1", "record": {...} }
 {"op": "upsert", "__typename": "Payment", "id": "123", "version": "2", record: {...} }
 {"op": "delete", "__typename": "Payment", "id": "123", "version": "3"}

elasticgraph-local/README.md

@@ -46,7 +46,7 @@ graph LR;
 
 Add `elasticgraph-local` to a new project `Gemfile`:
 
-```ruby
+```
 source "https://rubygems.org"
 
 group :development do
@@ -64,7 +64,7 @@ deployment).
 
 Next, install the `elasticgraph-local` rake tasks in your `Rakefile`, with code like:
 
-``` ruby
+```
 require 'elastic_graph/local/rake_tasks'
 
 ElasticGraph::Local::RakeTasks.new(
@@ -85,7 +85,7 @@ end
 Everything you need is provided by rake tasks. Run the following to see what they are:
 
 ```bash
-$ bundle exec rake -T
+bundle exec rake -T
 ```
 
 At a high level, this provides tasks that help you to:

elasticgraph-query_interceptor/README.md

@@ -25,15 +25,15 @@ graph LR;
 
 First, add `elasticgraph-query_interceptor` to your `Gemfile`:
 
-``` ruby
+```
 gem "elasticgraph-query_interceptor"
 ```
 
 Next, configure this library in your ElasticGraph config YAML files.
 An optional "config" dictionary can be provided to pass in values to
 your interceptor when it is initialized.
 
-``` yaml
+```yaml
 extension_modules:
 - require_path: elastic_graph/query_interceptor/graphql_extension
   name: ElasticGraph::QueryInterceptor::GraphQLExtension
@@ -48,7 +48,7 @@ query_interceptor:
 Define your interceptors at the configured paths. Each interceptor must
 implement this interface:
 
-``` ruby
+```ruby
 module YourApp
   class ExampleInterceptor
     def initialize(elasticgraph_graphql:, config:)

elasticgraph-rack/README.md

@@ -35,7 +35,7 @@ example of using it in a Rack `config.ru` file:
 require 'elastic_graph/graphql'
 require 'elastic_graph/rack/graphql_endpoint'
 
-graphql = ElasticGraph::GraphQL.from_yaml_file("path/to/config.yaml")
+graphql = ElasticGraph::GraphQL.from_yaml_file("config/settings/local.yaml")
 run ElasticGraph::Rack::GraphQLEndpoint.new(graphql)
 ```
 

elasticgraph/README.md

@@ -25,6 +25,6 @@ graph LR;
 
 Run this command to bootstrap a new local project:
 
-```bash
+```
 elasticgraph new my_app
 ```

elasticgraph/lib/elastic_graph/project_template/Gemfile.tt

@@ -4,9 +4,11 @@ source "https://rubygems.org"
 elasticgraph_details = <%= ElasticGraph.setup_env.gemfile_elasticgraph_details_code_snippet %>
 
 gem "elasticgraph-local", *elasticgraph_details
-gem "elasticgraph-<%= ElasticGraph.setup_env.datastore %>", *elasticgraph_details
 gem "elasticgraph-query_registry", *elasticgraph_details
 
+# Can be elasticgraph-elasticsearch or elasticgraph-opensearch based on the datastore you want to use.
+gem "elasticgraph-<%= ElasticGraph.setup_env.datastore %>", *elasticgraph_details
+
 gem "httpx", "~> 1.3"
 
 group :development do

script/ci_parts/run_misc_checks

@@ -8,6 +8,7 @@ script/type_check
 script/update_dependency_diagrams --verify
 script/update_ci_yaml --verify
 script/update_licenses --verify
+script/validate_readme_snippets
 
 bundle exec standardrb
 bundle exec rake schema_artifacts:check VERBOSE=true
@@ -29,4 +30,4 @@ halt_datastore_daemon
 # Test against federation v2.0, v2.3, and v2.6.
 elasticgraph-apollo/script/test_compatibility 2.6
 elasticgraph-apollo/script/test_compatibility 2.3
-elasticgraph-apollo/script/test_compatibility 2.0
+elasticgraph-apollo/script/test_compatibility 2.0

script/readme_snippets/bash_snippet_validator.rb

@@ -0,0 +1,31 @@
+# Copyright 2024 - 2025 Block, Inc.
+#
+# Use of this source code is governed by an MIT-style
+# license that can be found in the LICENSE file or at
+# https://opensource.org/licenses/MIT.
+#
+# frozen_string_literal: true
+
+require_relative "snippet_validator"
+require "tempfile"
+
+class BashSnippetValidator < SnippetValidator
+  BASH_TIMEOUT_SECONDS = 5
+
+  def validate(snippet)
+    execute_in_temp_project do
+      Tempfile.create do |output_file|
+        success, _ = execute_process_with_timeout(BASH_TIMEOUT_SECONDS) do
+          spawn("bash", "-c", snippet.content, [:out, :err] => output_file)
+        end
+
+        # Read the output from the file
+        output = File.exist?(output_file) ? File.read(output_file) : ""
+
+        success ? ValidationResult.passed(output) : ValidationResult.failed(output)
+      end
+    end
+  rescue => e
+    ValidationResult.failed("Exception during bash snippet validation: #{e.message}")
+  end
+end

script/readme_snippets/diff_snippet_validator.rb

@@ -0,0 +1,70 @@
+# Copyright 2024 - 2025 Block, Inc.
+#
+# Use of this source code is governed by an MIT-style
+# license that can be found in the LICENSE file or at
+# https://opensource.org/licenses/MIT.
+#
+# frozen_string_literal: true
+
+require_relative "snippet"
+require_relative "snippet_validator"
+
+class DiffSnippetValidator < SnippetValidator
+  def validate(snippet)
+    if snippet.content.match?(/^-gem "elasticgraph-elasticsearch".*\n^\+gem "elasticgraph-opensearch"/m)
+      show_debug_output "Diff snippet assumes the project was bootstrapped with elasticsearch. Apply diffs from `elasticgraph-elasticsearch` to put it into that state."
+
+      # The diff assumes the ElasticGraph project was originally bootstrapped with Elasticsearch.
+      # In reality it bootstraps with OpenSearch and we need to swap it over to Elasticsearch before
+      # we can apply the diff.
+      #
+      # We do so using the diff snippets from `elasticgraph-elasticsearch/README.md`.
+      Snippet
+        .extract_from(::File.expand_path("../../elasticgraph-elasticsearch/README.md", __dir__))
+        .select { |s| s.type == "diff" }
+        .each { |s| apply_diff(s) }
+    end
+
+    apply_diff(snippet) do
+      # If it's a `Gemfile` change, run `bundle install`.
+      # Otherwise, run `rake` to verify the build still passes.
+      if snippet.content.lines.first.strip == "diff --git a/Gemfile b/Gemfile"
+        bundle_output = `bundle install 2>&1`
+
+        if $?.success?
+          ValidationResult.passed
+        else
+          ValidationResult.failed("Bundle install failed:\n#{bundle_output}")
+        end
+      else
+        rake_output = `bundle exec rake 2>&1`
+
+        if $?.success?
+          ValidationResult.passed
+        else
+          ValidationResult.failed("Rake failed:\n#{rake_output}")
+        end
+      end
+    end
+  end
+
+  private
+
+  def apply_diff(snippet)
+    temp_project.in_dir do
+      # Create a temporary diff file
+      Tempfile.create(["snippet", ".diff"]) do |diff_file|
+        diff_file.write(snippet.content)
+        diff_file.flush
+
+        # Apply the diff
+        apply_output = `git apply #{Shellwords.escape(diff_file.path)} 2>&1`
+        unless $?.success?
+          return ValidationResult.failed("Failed to apply diff:\n#{apply_output}")
+        end
+
+        yield if block_given?
+      end
+    end
+  end
+end

script/readme_snippets/fallback_snippet_validator.rb

@@ -0,0 +1,13 @@
+# Copyright 2024 - 2025 Block, Inc.
+#
+# Use of this source code is governed by an MIT-style
+# license that can be found in the LICENSE file or at
+# https://opensource.org/licenses/MIT.
+#
+# frozen_string_literal: true
+
+class FallbackSnippetValidator < SnippetValidator
+  def validate(snippet)
+    ValidationResult.failed("Snippet type `#{snippet.type}` is not yet supported by the validation script. Consider adding a validator for this type.")
+  end
+end