Skip to content

v1.20.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 08 Oct 12:42
· 126 commits to main since this release
v1.20.0
3c9ae40
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

  • http checks requests and responses against an OpenAPI document with openapi, and grpc checks calls against .proto files with proto.
  • probe coverage lists the operations or methods of a document that no step checked.
  • probe check reports mistakes in a workflow without running it.
  • --read-only, --allow-host and --allow-action limit what a run may do.
  • Building Probe, or a module that imports it, needs Go 1.27.

Checking HTTP calls against an OpenAPI document

The http action can now check each request and response against an OpenAPI document, independently of the step's test. Set openapi in a job's defaults to check every request of the job:

- name: Users
  defaults:
    http:
      url: "{{vars.api_url}}"
      openapi:
        spec: ./openapi.yml
  steps:
    - name: Create a user
      uses: http
      with:
        post: /users
        body:
          name: probe
      test: res.body.name == "probe"
  • A request that breaks the document fails the step with the new kind contract_request: a missing or invalid parameter, a body that does not match its schema or Content-Type, or missing credentials that security requires.
  • A response that breaks it fails with contract_response: no operation for the method and path, an undeclared status code, a Content-Type the response does not declare, or a header or body that does not match its schema.
  • The step fails even when its test holds. A step without a test passes when the document allowed the request and the response.
  • request: false checks the response only, for a step that sends an invalid request on purpose. strict: true also fails on undeclared properties, undeclared query parameters, readOnly properties in a request and writeOnly properties in a response. openapi: false turns the check off for one step.
  • Violations are in res.violations and in the terminal output and every report format. Both kinds exit with status 1.

See Checking Against OpenAPI.

Checking gRPC calls against .proto files

The grpc action can check a call against .proto files, such as those in the repository the service is built from, instead of trusting only the definition the server reports through reflection. The files are compiled inside Probe, so protoc is not needed.

defaults:
  grpc:
    addr: localhost:50051
    service: user.v1.UserService
    proto:
      files: [./proto/user/v1/user.proto]
      import_paths: [./proto]
  • The request body is checked against the files' request message (contract_request). The server's definition is compared with the files' field by field, and the response bytes are decoded again with the files' response message (contract_response).
  • buf.validate (protovalidate) rules are checked on the request and the response. google.api.field_behavior is checked as AIP-203 defines it: a REQUIRED field must be in the request, and an INPUT_ONLY field must not be in the response. Both .proto files are built into Probe, so they need not be in an import path.
  • A server without reflection can now be called: when proto is given and reflection is off, the call uses the files' definition.
  • request: false, strict: true and proto: false work as they do for openapi.

See Checking Against .proto Files.

probe coverage

probe coverage reads the JSON report of a run and lists which operations and responses of an OpenAPI document, or which methods of a .proto file, no step checked.

probe --report json workflow.yml
probe coverage openapi.yml probe-report.json
probe coverage proto/users.proto probe-report.json
  • A step counts when its call was checked against the document, whether the step passed or failed.
  • It exits with 0 whatever the coverage, and with 2 when the document or the report cannot be read, or when no step in the report was checked against the document given.
  • The JSON report now records what each step was matched to as contract, and the http and grpc actions return it in res.contract.

See coverage.

probe check

probe check reports mistakes in a workflow without running it. A run ignores keys it does not recognize, so a misspelled test leaves a step unchecked and the run still passes; probe check reports it with the file and line.

workflow.yml:10: error: job 0 "Users", step 0 "Log in": unknown key "tset"; did you mean "test"?
workflow.yml:14: error: job 0 "Users", step 1 "Read": unknown action "htp"; did you mean "http"?
workflow.yml:18: error: job 0 "Users", step 1 "Read": with: outputs.login.tokn is not published: step "login" publishes token
  • Errors: unknown keys in a workflow, job, step, retry, repeat, a built-in action's with (such as bdoy for body) or a job's defaults; unknown actions; bad needs and step ids; expressions and templates that do not parse; outputs that no step publishes or that are read before their step publishes them.
  • Warnings: an output read from a job the reading job does not need, a step that nothing checks, and a test that reads nothing, such as 1 == 1.
  • It exits with 2 when it finds an error, and with 0 otherwise. The skill installed by probe skill now tells coding agents to run it before running a workflow they wrote.

See check.

Running under a guard

Three new flags limit what a run may do. They are set on the command line or in the environment by whoever runs Probe, and a workflow cannot override them.

probe --read-only --allow-host api.staging.example.com,*.internal workflow.yml
probe --read-only --allow-action shell workflow.yml
  • --read-only lets http send only GET, HEAD and OPTIONS, and lets db run only a single SELECT, SHOW, DESCRIBE, DESC, EXPLAIN or WITH statement inside a read-only transaction.
  • --allow-host refuses connections from http and db to hosts not in the list. For db, the host checked is the one the driver itself reads from the DSN.
  • http, db, embedded and hello follow the guard. Every other action, built-in or external, is refused under a guard unless --allow-action names it. A job run by embedded runs under the same guard.
  • A refused step fails before anything is sent, with the new kind refused; it is not retried and exits with status 2.
  • The environment variables are PROBE_READ_ONLY, PROBE_ALLOW_HOSTS and PROBE_ALLOW_ACTIONS. A flag given on the command line takes precedence, so --read-only=false or --allow-action= clears what the environment set.
  • The guard is not a sandbox: it limits what the listed actions send, not what a process can do.

See --read-only.

Other changes

  • The browser action is built on chromedp v0.19.1, which talks to Chrome over a pipe instead of a websocket, so browser steps run faster. A step that waits for an element present on both the current and the next page, such as h1 right after submit, may now read the page before the navigation finishes; wait for an element only the next page has.
  • Installing with go install needs Go 1.27 or later.
  • The skill installed by probe skill now says that templates are evaluated in keys at any depth of with and of a map under vars, but not in the name of a var itself.

For library users

This release breaks the Go API in a minor version. A module that imports probe needs Go 1.27 or later. browser.BrowserRunner.Run and the function fields and recorded calls of browser.MockRunner take chromedp.Action[chromedp.Void] instead of chromedp.Action, following chromedp's change to a generic action type. Workflow files are not affected. See the CHANGELOG for the full list of signatures.

New public API:

  • probe.Check, Finding and CheckOptions run the checks of probe check. actions.Params and actions.AllParams give the keys each built-in action takes, to pass as CheckOptions.Params.
  • probe.Config.Guard sets the guard of a run, and probe.WithGuard the guard of a job run with RunStandalone, whose JobRun.Refused holds the first refusal. actionrpc gains Guard, Refused, Refuse and IsRefused for actions that follow it, and actions.Keeping lists the built-in actions that do.
  • oas.NewCoverage, oas.ReadReport and oas.CountCoverage compute coverage, and actions/grpc.NewCoverage does so for a .proto file.
  • report gains Contract, Violation, FailureContractRequest and FailureContractResponse, and probe gains FailureRefused.

Changelog

  • 115fdf9 Add a section for coding agents to the landing page
  • f8df780 Build reflected definitions with their imports, and read replies apart
  • 5971c8b Bump grpc, mysql, validator and protovalidate
  • 7e01650 Call a server without reflection with the definition of the .proto files
  • 43dd3b5 Call actions actions rather than plugins in the docs
  • 73989ad Carry what each step's response was matched to into the JSON report
  • 241c56d Check an empty body, and take a name starting with two dots as under its path
  • 1c2a7ed Check bodies other than JSON, and empty or null JSON bodies
  • 08bb165 Check gRPC calls against .proto files with proto
  • 57724fd Check http responses against an OpenAPI document with openapi
  • 039dfb5 Check keys from aliases and unclosed openers, and take tags as written
  • a4ad194 Check the MySQL address go-sql-driver dials, not the URL's host
  • f98de36 Check the PostgreSQL servers lib/pq resolves the DSN to
  • 48e1007 Check the constraints .proto files annotate, by buf.validate and field_behavior
  • b1e781b Check the request against the OpenAPI document as well
  • b38e83b Check the server lib/pq connects to, and refuse any inner semicolon
  • 313afb6 Describe calling a server without reflection through proto
  • 0e8be6b Describe checking the request and run it in the end-to-end tests
  • 24a861a Describe coverage of .proto files and count it in the end-to-end tests
  • f1d69af Describe openapi and run its example in the end-to-end tests
  • 66fe453 Describe probe check and check the examples in the end-to-end tests
  • 5cf7b87 Describe probe coverage and run it in the end-to-end tests
  • cb4bd46 Describe proto and run it against the test server in the end-to-end tests
  • 085f643 Describe strict and run it in the end-to-end tests
  • 4b27a68 Describe the constraints checked from annotations of .proto files
  • e8ff5ba Describe the guard and run it against httpbin, PostgreSQL and MySQL
  • 244349e Describe the keys of with and defaults probe check finds
  • 7ac8f12 Fail a body neither definition takes as a request that breaks its contract
  • 5418817 Fail a step whose request breaks its contract as contract_request
  • facfb9e Fail a step whose response breaks its contract as contract_response
  • 5b5191d Fail what the OpenAPI document does not declare with strict
  • 5046eb7 Find keys of with and defaults an action does not take with probe check
  • 8fbb7dd Find what is wrong or weak in a workflow with probe check
  • 0238c7e Keep each publishing step under its id, not the last step given it
  • dbf33c1 Keep the db action to the guard, with the database's own read-only mode
  • 782c26c Keep the http action to the guard, redirects too
  • b924b1c Keep the space before the link in the docs footer
  • c44fa79 Let a guard flag given false or empty win over its environment variable
  • 0540672 List the keys a struct is filled from with mapping.FieldTags
  • 4a62538 Look a service up by its full name, and keep reflection to the reply's files
  • 8673f15 Mark a refusal in its status, apart from a PermissionDenied of an action
  • f17a00f Merge branch 'feat/http-openapi-contract' into feat/http-openapi-request
  • 390f49c Merge branch 'feat/http-openapi-request' into feat/http-openapi-strict
  • f2b69db Merge branch 'feat/http-openapi-strict' into feat/openapi-coverage
  • 367db45 Merge branch 'main' into feat/check-with-keys
  • 5311e72 Merge pull request #326 from linyows/feat/openapi-coverage
  • c8837e0 Merge pull request #328 from linyows/feat/check
  • 141b01c Merge pull request #329 from linyows/docs/skill-template-keys
  • d2c5958 Merge pull request #330 from linyows/feat/guard
  • dcce0d1 Merge pull request #331 from linyows/fix/guard-mysql-addr
  • 1e565b1 Merge pull request #332 from linyows/feat/grpc-proto
  • c296457 Merge pull request #333 from linyows/feat/grpc-proto-constraints
  • d4befff Merge pull request #334 from linyows/feat/grpc-coverage
  • 5b53de9 Merge pull request #335 from linyows/feat/grpc-without-reflection
  • a042cbf Merge pull request #341 from mozership/feat/check-with-keys
  • ec5475b Merge pull request #342 from mozership/ci/probe-action-owner
  • 50c279e Merge pull request #343 from mozership/docs/lp-ai-section
  • f8e5858 Merge pull request #344 from mozership/deps/bump-2026-10
  • 1a37735 Merge pull request #345 from mozership/deps/chromedp-0.19
  • 3c9ae40 Merge pull request #346 from mozership/release/v1.20.0
  • 7e7950a Move the browser action to chromedp v0.19.1 and Go 1.27
  • 3f7e1a2 Name proto beside openapi in what the skill says to do about a contract
  • 482fafd Pin probe-action to v1.2.1
  • b774b65 Pin probe-action to v1.2.2
  • bb8e4f9 Prepare the changelog for v1.20.0
  • 3b6b131 Refuse a DSN that runs a statement on connecting under --read-only
  • f0e4417 Refuse the step that embeds a job whose step the guard refused
  • 797fa41 Retry a step its contract alone checks, and check the contract first
  • 770f574 Run under a guard set by --read-only, --allow-host and --allow-action
  • cc4669b Say how a run treats each mistake check finds, and which actions are resolved
  • 70c3b9e Take a step checked against .proto files as checked in probe check
  • 80dbf69 Take openapi and proto as contracts only for the actions that check them
  • 0bdd523 Tell a required field from an optional one in the server's definition
  • ad692e8 Tell actions the guard of the run, and carry their refusals back
  • 89ffb3e Tell agents that templates in keys are evaluated, but not in var names
  • ed0d7e0 Tell from an expression whether it parses and what it reads
  • 985354a Tell how much of a .proto file a run checked with probe coverage
  • b4902a2 Tell how much of an OpenAPI document a run checked with probe coverage
  • c3da5b9 Tell the keys each built-in action takes in with
  • 47d7853 Tell what a gRPC call was matched to in res.contract
  • a6a97c3 Tell what the response was matched to in res.contract
  • c6145b0 Use probe-action from mozership, where it was transferred
Assets 7
Loading