open-policy-agent / opa

This release contains a number of enhancements and bug fixes.

Tooling

• The eval subcommand now supports a -s/--schema flag that accepts a JSON schema for the input document. The schema is used when type checking the policy so that invalid references to (or operations on) input data are caught at compile time. In the future, the schema support will be expanded to accept multiple schemas and rule-level annotations. See the new Schemas documentation for details. Authored by @aavarghese and @vazirim.
• The eval, test, bench and REPL subcommands now supports a -t/--target flag to set the evaluation engine to use. The default engine is rego referring to the standard Rego interpreter in OPA. Users can now select wasm to enable Wasm compilation and execution of policies (#2878).
• The eval subcommand now supports a raw option for -f/--format that is useful in bash scripts. Authored by @jaspervdj-luminal.
• The test framework now supports "skippable" tests. Prefix the test name with todo_ to have the test runner skip the test, e.g., todo_test_allow { ... }.
• The eval subcommand now correctly supports the --ignore flag. Previously the flag was not being applied.

Server

• The POST /v1/compile API now supports a ?metrics query parameter similar to other APIs. Authored by @jkbschmid.
• The directory used for persisting downloaded bundles can now be configured. See the Configuration page for details.
• The HTTP Decision Logger plugin no longer blocks server shutdown for the grace period when there are no logs to upload.
• The Bundle plugin now unregisters listeners correctly. This issue would cause listeners to be invoked when bundle updates were dispatched even if the listener was unregistered (#3190).
• The server now correctly decodes policy IDs in the HTTP request URL. Authored by @mattmahn (#2116).
• The server now configures the http_request_duration_seconds metric (for all of the server endpoitns) with smaller, more granular buckets that better map to actual response latencies from OPA. Authored by @luong-komorebi (#3196).

Security

• PKCS8 keys are now supported when signing bundles and communicating with control plane services. Previously only PKCS1 keys were supported (#3116).
• The built-in OPA HTTP API authorizer policy can now return a reason to explain why a request to the OPA API is denied (#3056). See the Security documentation for details. Thanks to @ajanthan for helping improve this.

Compiler

• The compiler can be configured to emit debug messages that explain comprehension indexing decisions. Debug messages can be enabled when running opa build with --debug.
• A panic was fixed in one of the rewriting stages when comprehensions were used as object keys (#2915)

Evaluation

• A bug in big integer comparison was fixed. This issue was discovered when comparing serial numbers from X.509 certificates. Authored by @andrehaland (#3147).
• The io.jwt.decode_verify function now uses the environment supplied time-of-day value instead of calling time.Now() (#3105).

Documentation

• The documentation now includes a dedicated section the OPA-Envoy integration. See https://www.openpolicyagent.org/docs/latest/envoy-introduction/ for details.
• The ecosystem page now ranks integrations by number of unique domains instead of the sheer number of references.

WebAssembly

• The data document no longer needs to be initialized to an empty object (#3130).
• The mpd library is now initalized by the module's Start function (#3110).
• The planner now longer re-plans rules blindly when with statements are encountered (#3150).
• The planner and compiler now support dynamic dispatch. Previously the planner would enumerate all functions and invocation was controlled at runtime (#2936).
• The compiler now inserts memoization instructions into function bodies instead of at callsites. This reduces the number of wasm instructions in the resulting binary (#3169).
• The wasmtime runtime is now the default runtime used by OPA to execute compiled policies. The new runtime no longer leaks memory when policies are reloaded.
• The planner and compiler now intern strings and booleans and implement a few micro-optimizations to reduce the size of the resulting binary.
• The capabilities support has been updated to include an ABI major and minor version for tracking backwards compatibility on compiled policies (#3120).

Backwards Compatibility

• The opa test subcommand previously supported a -t flag as shorthand for --timeout. With this release, the -t shorthand has been redefined for --target. After searching GitHub for examples of opa test -t (and finding nothing) we felt comfortable making this backwards incompatible change.
• The Go version used to build the OPA release has been updated from 1.14.9 to 1.15.8. Because of this, TLS certificates that rely on Common Name for verification are no longer supported and will not work. For more information see golang/go#39568.

This release contains a number of enhancements and bug fixes.

Built-in Functions

• This release includes a number of built-in function improvements for Wasm compiled policies. The following built-in functions have been implemented natively and no longer need to be supplied by SDKs: graph.reachable, json.filter, json.remove, object.get, object.remove, and object.union.

• This release fixes several bugs in the Wasm implementation of certain regex built-in functions (#2962), format_int (#2923) and round (#2999).

• This release adds ceil and floor built-in functions. Previously these could be implemented in Rego using round however these are more convenient.

Enhancements

• OPA has been extended support OAuth2 JWT Bearer Grant Type and OAuth2 Client Credential JWT authentication options for communicating with control plane services. This change allows OPA to use services that rely on Ping Identity as well as GCP service accounts for authentication. OPA has also been extended to support custom authentication plugins (thanks @gshively11).

• OPA plugins can now enter a "WARN" state to indicate they are operating in a degraded capacity (thanks @gshively11).

• The opa bench command can now benchmark partial evaluation queries. The options to enable partial evaluation are shared with opa eval. See opa bench --help for details.

• Wasm compiled policies now contain source locations that are included inside of runtime error messages (such as object key conflicts.) In addition, Wasm compiled policies only export the minimal set of APIs described on the WebAssembly#exports page.

Fixes

• ast: Fix parsing of numbers to reject leading zeroes (#2947) authored by @LCartwright.
• bundle: Fix loader to only verify bundle keys if configured to do so (#3028).
• cmd: Fix build to avoid packaging policy.wasm twice (#3007).
• cmd: Fix pretty-printed PE output to hide spurious blank lines
• server: Fix false-positive in bundle root check that would prevent data updates in some cases (#2868).
• server: Fix query cache to respect ?instrument option (#3000).
• server: Fix server to support discovery on inter-query cache configuration
• topdown: Fix PE to avoid generating expressions that do not type check (#3012).
• wasm: Fix planner to avoid generating a conflict error in some cases (#2926).
• wasm: Fix planner to generate correct virtual document iteration instructions (#3065).
• wasm, topdown: Fix with keyword handle to ensure last statement wins (#3010).
• wasm: Fix planner to handle assignment conflicts correctly when else keyword is used (#3031).

Documentation

• Add new section on integrating policies with OAuth2 and OIDC.
• Update Kubernetes admission control tutorial to work as non-root user.
• Fix link to signing documentation (#3027) authored by @princespaghetti.

Backwards Compatibility

• Previously, OPA deduplicated sets and objects in all cases except when iterating over/referring directly to values generated by partial rules. This inconsistency would only be noticed when running ad-hoc queries or within policies when aggregating the results of array comprehensions (e.g., count([1 | p[x]]) could observe duplicates in p.) This release removes the inconsistency by deduplicating sets and objects in all cases (#429). This was the second oldest open issue on the project.

Deprecations

• OPA now logs warnings when it receives legacy bundle config sections instead of the bundles section introduced in v0.13.0.

This release extends the HTTP server authorizer (--authorization=basic) to supply the HTTP message body in the input document. See the Authentication and Authorization section in the security documentation for details.

This release contains a fix for running OPA under Docker with a non-default working directory (#2974):

/opa: error while loading shared libraries: libwasmer.so: cannot open shared object file: No such file or directory

This release contains a number of improvements and fixes. Importantly, this release includes a notable change to built-in function error handling. See the section below for details.

Built-in Function Error Handling

Previously, built-in function errors would cause policy evaluation to halt immediately. Going forward, by default, built-in function errors no longer halt evaluation. Instead, expressions are treated as false/undefined if any of the invoked built-in functions return errors.

This change resolves a common issue people face when passing unsanitized input values to built-in functions. For example, prior to this change the expression io.jwt.decode("GARBAGE") would halt evaluation of the entire policy because the string is not a valid encoding of a JSON Web Token (JWT). If the expression was io.jwt.decode(input.token) and the user passed an invalid string value for input.token the same error would occur. With this change, the same expression is simply undefined, i.e., there is no result. This means policies can use negation to test for invalid values. For example:

decision := {"allowed": allow, "denial_reason": reason}

default allow = false

allow {
  io.jwt.verify_hs256(input.token, "secret")
  [_, payload, _] := io.jwt.decode(input.token)
  payload.role == "admin"
}

reason["invalid JWT supplied as input"] {
  not io.jwt.decode(input.token)
}

If you require the old behaviour, enable "strict" built-in errors on the query:

If you have implemented custom built-in functions and require policy evaluation to halt on error in those built-in functions, modify your built-in functions to return the topdown.Halt error type.

Built-in Functions

This release includes a few new built-in functions:

• base64url.encode_no_pad, hex.encode, and hex.decode for dealing with encoded data (#2849) authored by @johanneslarsson
• json.patch for applying JSON patches to values inside of policies (#2839) authored by @jaspervdj-luminal
• json.is_valid and yaml.is_valid for testing validity of encoded values (authored by @jaspervdj-luminal)

There were also a few fixes to existing built-in functions:

• Fix unicode handling in a few string-related functions (#2799) authored by @anderseknert
• Fix http.send to override no-cache HTTP header when force_cache specified (#2841) authored by @anderseknert
• Fix strings.replace_n to replace overlapping patterns deterministically (#2822)
• Fix panic in units.parse_bytes when passed a zero-length string (#2901)

Miscellaneous

This release adds new credential providers for management services:

• GCP metadata server (#2938) authored by @kelseyhightower
• AWS Web Identity credentials (#2462) authored by @RichiCoder1
• OAuth2 (#1205) authored by @anderseknert

In addition the following server features were added:

• Add shutdown wait period flag to opa run (--shutdown-wait-period) (#2764) authored by @bcarlsson
• Add bundle file size limit configuration option (bundles[_].size_limit_bytes) to override default 1GiB limit (#2781)
• Separate decision log and status message logs from access logs (which useful for running OPA at log level error while continuing to report decision and status log to console) (#2733) authored by @anderseknert

Fixes

• Fix panic caused by race condition in the decision logger (#2835) authored by @kubaj
• Fix decision logger to flush on graceful shutdown (#780) authored by @anderseknert
• Fix --verification-key handling to accept PEM files (#2796)
• Fix --capabilities flag in opa build command (#2848) authored by @srenatus
• Fix loading of signed persisted bundles (#2824)
• Fix API response mutation caused by decision log masking (#2752) authored by @gshively11
• Fix evaluator to prevent with statements from mutating original input document (#2813)
• Fix set iteration runtime to be O(n) instead of O(n^2) (#2966)
• Increased OPA version telemetry report timeout from 1 second to 5 seconds to deal with slow networks

Documentation

• Improve docs to mention built-in function support in WebAssembly compiled policies
• Improve docs around JWT HMAC encoding (#2870) authored by @anderseknert
• Improve HTTP authorization tutorial steps for zsh (#2917 authored by @ClaudenirFreitas)
• Improve docs to describe meaning of Prometheus metrics
• Remove mention of unsafe (and unsupported) "none" signature algorithm from JWT documentation

WebAssembly

This release also includes a number of improvements to the Wasm support in OPA. Importantly, OPA now integrates a Wasm runtime that can be used to execute Wasm compiled policies. The runtime is integrated into the existing "topdown" evaluator so that specific portions of the policy can be compiled to Wasm as a performance optimization. When the evaluator executes a policy using the Wasm runtime it emits a special Wasm trace event. The Wasm runtime support in OPA is currently considered experimental and will be iterated on in coming releases.

This release also extends the Wasm compiler in OPA to natively support the following built-in functions (in alphabetical order):

• base64.encode, base64.decode, base64url.encode, and base64url.decode
• glob.match
• json.marshal and json.unmarshal
• net.cidr_contains, net.cidr_intersects, and net.cidr_overlap
• regex.match, regex.is_valid, and regex.find_all_string_submatch_n
• to_number
• walk

Backwards Compatibility

• The --insecure-addr flag (which was deprecated in v0.10.0) has been removed completely (#763)

This release contains a number of small enhancements and bug fixes.

Bundle Persistence

This release adds support for persisting bundles for recovery purposes. When persistence is enabled, OPA will save activated bundles to disk. On startup, OPA checks for persisted bundles and activates them immediately. This allows OPA to startup if the bundle server is unavailable (#2097). For more information see the Bundle documentation.

Built-in Functions

This release includes a few new built-in functions:

• base64.is_valid for testing if strings are valid base64 encodings (#2690) authored by @carlpett
• net.cidr_merge function for merging sets of IPs and CIDRs (#2692)
• urlquery.decode_object for parsing URL query parameters into objects (#2647) authored by @GBrawl

In addition, http.send has been enhanced to support caching overrides and in-band error handling (#2666 and #2187).

Fixes

• Fix opa build to support custom built-in functions (#2738) authored by @gshively11
• Fix for file watching volume mounted configmaps (#2588) authored by @drewwells
• Fix discovery plugin to set last request and last successful request timestamps in status updates (#2630)
• Fix planner crash on virtual document iteration (#2601)
• Fix decision logger to requeue failed chunks (#2724 authored by @anderseknert)
• Fix object/set implementation in WASM-C library to avoid resizing.
• Fix JSON parser in WASM-C library to copy memory for strings and numbers.
• Improve WASM-C library to recycle object and set element structures while growing.

In addition, this release contains several fixes for panics identified by fuzzing:

• ast: Fix compiler to expand exprs in rule args (#2649)
• ast: Fix output var analysis to accept refs with non-var heads (#2678)
• ast: Fix panic during local var rewriting (#2720)
• ast: Fix panic in local var rewriting caused by object corruption (#2661)
• ast: Fix panic in parser post-processing of expressions (#2714)
• ast: Fix parser to ignore rules with args and key in head (#2662)
• ast: Fix object corruption during safety reordering
• types: Fix panic on reference to object with composite key (#2648)

Backwards Compatibility

• Renamed timer_rego_builtin_http.send_ns to timer_rego_builtin_http_send_ns to avoid issues with periods in metric keys.
• Removed deprecated watch package (#2265)

Miscellaneous

• Add support for H2C on HTTP listener (#2739 thanks @srenatus!).
• Add Go version information to opa version output (thanks @srenatus!)
• The official OPA build has been updated to Go v1.14.9. Previously it was using v1.13.7 which is no longer supported (thanks @srenatus!)