This release contains a fix for crashes experienced when configuring OPA to use S3 signing as service credentials (#3255).
In addition to that, we have a small number of enhancements and fixes:
Tooling
- The
eval
subcommand now allows using--import
without using--package
. Authored by @onelittlenightmusic, #3240.
Compiler
- The
ast
package now exports another method for JSON conversion,ast.JSONWithOpts
, that allows further options to be set (#3244.
Server
- REST plugins using
s3_signing
as credentials method can now include the specified service in the signature (SigV4). Authored by @cogwirrel, #3210.
Documentation
- Remove soon-to-be deprecated
any
andall
from the Policy Reference (#3241) -- see also #2437. - Add missing
discovery.service
field to Discovery configuration table (#3237). - Fix dead links to the Envoy pages (#3248).
WebAssembly
- Executions using the internal Wasm SDK will now be interrupted when the provided context is done (cancelled or deadline reached).
- The generated Wasm modules could become much smaller: unused functions are replaced by
unreachable
stubs, and the heavyweight runtime components related to regular expressions are excluded when none of the regex-related builtins are used:glob.match
,regex.is_valid
,regex.match
,regex.is_valid
, andregex.find_all_string_submatch_n
. - The Wasm runtime now allows passing in the time to be used for evaluation, enabling callers to control the time-of-day observed by Wasm compiled policies.
- Wasmtime runtime has been updated to the latest version (v0.24.0).
Assets
6
This release contains a number of enhancements and bug fixes.
Tooling
- The
eval
subcommand now supports a-s
/--schema
flag that accepts a JSON schema for theinput
document. The schema is used when type checking the policy so that invalid references to (or operations on)input
data are caught at compile time. In the future, the schema support will be expanded to accept multiple schemas and rule-level annotations. See the new Schemas documentation for details. Authored by @aavarghese and @vazirim. - The
eval
,test
,bench
and REPL subcommands now supports a-t
/--target
flag to set the evaluation engine to use. The default engine isrego
referring to the standard Rego interpreter in OPA. Users can now selectwasm
to enable Wasm compilation and execution of policies (#2878). - The
eval
subcommand now supports araw
option for-f
/--format
that is useful in bash scripts. Authored by @jaspervdj-luminal. - The test framework now supports "skippable" tests. Prefix the test name with
todo_
to have the test runner skip the test, e.g.,todo_test_allow { ... }
. - The
eval
subcommand now correctly supports the--ignore
flag. Previously the flag was not being applied.
Server
- The
POST /v1/compile
API now supports a?metrics
query parameter similar to other APIs. Authored by @jkbschmid. - The directory used for persisting downloaded bundles can now be configured. See the Configuration page for details.
- The HTTP Decision Logger plugin no longer blocks server shutdown for the grace period when there are no logs to upload.
- The Bundle plugin now unregisters listeners correctly. This issue would cause listeners to be invoked when bundle updates were dispatched even if the listener was unregistered (#3190).
- The server now correctly decodes policy IDs in the HTTP request URL. Authored by @mattmahn (#2116).
- The server now configures the
http_request_duration_seconds
metric (for all of the server endpoitns) with smaller, more granular buckets that better map to actual response latencies from OPA. Authored by @luong-komorebi (#3196).
Security
- PKCS8 keys are now supported when signing bundles and communicating with control plane services. Previously only PKCS1 keys were supported (#3116).
- The built-in OPA HTTP API authorizer policy can now return a reason to explain why a request to the OPA API is denied (#3056). See the Security documentation for details. Thanks to @ajanthan for helping improve this.
Compiler
- The compiler can be configured to emit debug messages that explain comprehension indexing decisions. Debug messages can be enabled when running
opa build
with--debug
. - A panic was fixed in one of the rewriting stages when comprehensions were used as object keys (#2915)
Evaluation
- A bug in big integer comparison was fixed. This issue was discovered when comparing serial numbers from X.509 certificates. Authored by @andrehaland (#3147).
- The
io.jwt.decode_verify
function now uses the environment supplied time-of-day value instead of callingtime.Now()
(#3105).
Documentation
- The documentation now includes a dedicated section the OPA-Envoy integration. See https://www.openpolicyagent.org/docs/latest/envoy-introduction/ for details.
- The ecosystem page now ranks integrations by number of unique domains instead of the sheer number of references.
WebAssembly
- The
data
document no longer needs to be initialized to an empty object (#3130). - The mpd library is now initalized by the module's
Start
function (#3110). - The planner now longer re-plans rules blindly when
with
statements are encountered (#3150). - The planner and compiler now support dynamic dispatch. Previously the planner would enumerate all functions and invocation was controlled at runtime (#2936).
- The compiler now inserts memoization instructions into function bodies instead of at callsites. This reduces the number of wasm instructions in the resulting binary (#3169).
- The wasmtime runtime is now the default runtime used by OPA to execute compiled policies. The new runtime no longer leaks memory when policies are reloaded.
- The planner and compiler now intern strings and booleans and implement a few micro-optimizations to reduce the size of the resulting binary.
- The capabilities support has been updated to include an ABI major and minor version for tracking backwards compatibility on compiled policies (#3120).
Backwards Compatibility
- The
opa test
subcommand previously supported a-t
flag as shorthand for--timeout
. With this release, the-t
shorthand has been redefined for--target
. After searching GitHub for examples ofopa test -t
(and finding nothing) we felt comfortable making this backwards incompatible change. - The Go version used to build the OPA release has been updated from
1.14.9
to1.15.8
. Because of this, TLS certificates that rely on Common Name for verification are no longer supported and will not work. For more information see golang/go#39568.
Assets
6
This release contains a number of enhancements and bug fixes.
Built-in Functions
-
This release includes a number of built-in function improvements for Wasm compiled policies. The following built-in functions have been implemented natively and no longer need to be supplied by SDKs:
graph.reachable
,json.filter
,json.remove
,object.get
,object.remove
, andobject.union
. -
This release fixes several bugs in the Wasm implementation of certain
regex
built-in functions (#2962),format_int
(#2923) andround
(#2999). -
This release adds
ceil
andfloor
built-in functions. Previously these could be implemented in Rego usinground
however these are more convenient.
Enhancements
-
OPA has been extended support OAuth2 JWT Bearer Grant Type and OAuth2 Client Credential JWT authentication options for communicating with control plane services. This change allows OPA to use services that rely on Ping Identity as well as GCP service accounts for authentication. OPA has also been extended to support custom authentication plugins (thanks @gshively11).
-
OPA plugins can now enter a "WARN" state to indicate they are operating in a degraded capacity (thanks @gshively11).
-
The
opa bench
command can now benchmark partial evaluation queries. The options to enable partial evaluation are shared withopa eval
. Seeopa bench --help
for details. -
Wasm compiled policies now contain source locations that are included inside of runtime error messages (such as object key conflicts.) In addition, Wasm compiled policies only export the minimal set of APIs described on the WebAssembly#exports page.
Fixes
- ast: Fix parsing of numbers to reject leading zeroes (#2947) authored by @LCartwright.
- bundle: Fix loader to only verify bundle keys if configured to do so (#3028).
- cmd: Fix build to avoid packaging policy.wasm twice (#3007).
- cmd: Fix pretty-printed PE output to hide spurious blank lines
- server: Fix false-positive in bundle root check that would prevent data updates in some cases (#2868).
- server: Fix query cache to respect ?instrument option (#3000).
- server: Fix server to support discovery on inter-query cache configuration
- topdown: Fix PE to avoid generating expressions that do not type check (#3012).
- wasm: Fix planner to avoid generating a conflict error in some cases (#2926).
- wasm: Fix planner to generate correct virtual document iteration instructions (#3065).
- wasm, topdown: Fix with keyword handle to ensure last statement wins (#3010).
- wasm: Fix planner to handle assignment conflicts correctly when else keyword is used (#3031).
Documentation
- Add new section on integrating policies with OAuth2 and OIDC.
- Update Kubernetes admission control tutorial to work as non-root user.
- Fix link to signing documentation (#3027) authored by @princespaghetti.
Backwards Compatibility
- Previously, OPA deduplicated sets and objects in all cases except when iterating over/referring directly to values generated by partial rules. This inconsistency would only be noticed when running ad-hoc queries or within policies when aggregating the results of array comprehensions (e.g.,
count([1 | p[x]])
could observe duplicates inp
.) This release removes the inconsistency by deduplicating sets and objects in all cases (#429). This was the second oldest open issue on the project.
Deprecations
- OPA now logs warnings when it receives legacy
bundle
config sections instead of thebundles
section introduced in v0.13.0.
Assets
6
This release extends the HTTP server authorizer (--authorization=basic
) to supply the HTTP message body in the input
document. See the Authentication and Authorization section in the security documentation for details.
Assets
6
This release contains a fix for running OPA under Docker with a non-default working directory (#2974):
/opa: error while loading shared libraries: libwasmer.so: cannot open shared object file: No such file or directory
Assets
6
This release contains a number of improvements and fixes. Importantly, this release includes a notable change to built-in function error handling. See the section below for details.
Built-in Function Error Handling
Previously, built-in function errors would cause policy evaluation to halt immediately. Going forward, by default, built-in function errors no longer halt evaluation. Instead, expressions are treated as false/undefined if any of the invoked built-in functions return errors.
This change resolves a common issue people face when passing unsanitized input values to built-in functions. For example, prior to this change the expression io.jwt.decode("GARBAGE")
would halt evaluation of the entire policy because the string is not a valid encoding of a JSON Web Token (JWT). If the expression was io.jwt.decode(input.token)
and the user passed an invalid string value for input.token
the same error would occur. With this change, the same expression is simply undefined, i.e., there is no result. This means policies can use negation to test for invalid values. For example:
decision := {"allowed": allow, "denial_reason": reason}
default allow = false
allow {
io.jwt.verify_hs256(input.token, "secret")
[_, payload, _] := io.jwt.decode(input.token)
payload.role == "admin"
}
reason["invalid JWT supplied as input"] {
not io.jwt.decode(input.token)
}
If you require the old behaviour, enable "strict" built-in errors on the query:
Caller | Example |
---|---|
HTTP | POST /v1/data/example/allow?strict-builtin-errors |
Go (Library) | rego.New(rego.Query("data.example.allow"), rego.StrictBuiltinErrors(true)) |
CLI | opa eval --strict-builtin-errors 'data.example.allow' |
If you have implemented custom built-in functions and require policy evaluation to halt on error in those built-in functions, modify your built-in functions to return the topdown.Halt error type.
Built-in Functions
This release includes a few new built-in functions:
base64url.encode_no_pad
,hex.encode
, andhex.decode
for dealing with encoded data (#2849) authored by @johanneslarssonjson.patch
for applying JSON patches to values inside of policies (#2839) authored by @jaspervdj-luminaljson.is_valid
andyaml.is_valid
for testing validity of encoded values (authored by @jaspervdj-luminal)
There were also a few fixes to existing built-in functions:
- Fix unicode handling in a few string-related functions (#2799) authored by @anderseknert
- Fix
http.send
to overrideno-cache
HTTP header whenforce_cache
specified (#2841) authored by @anderseknert - Fix
strings.replace_n
to replace overlapping patterns deterministically (#2822) - Fix panic in
units.parse_bytes
when passed a zero-length string (#2901)
Miscellaneous
This release adds new credential providers for management services:
- GCP metadata server (#2938) authored by @kelseyhightower
- AWS Web Identity credentials (#2462) authored by @RichiCoder1
- OAuth2 (#1205) authored by @anderseknert
In addition the following server features were added:
- Add shutdown wait period flag to
opa run
(--shutdown-wait-period
) (#2764) authored by @bcarlsson - Add bundle file size limit configuration option (
bundles[_].size_limit_bytes
) to override default 1GiB limit (#2781) - Separate decision log and status message logs from access logs (which useful for running OPA at log level
error
while continuing to report decision and status log to console) (#2733) authored by @anderseknert
Fixes
- Fix panic caused by race condition in the decision logger (#2835) authored by @kubaj
- Fix decision logger to flush on graceful shutdown (#780) authored by @anderseknert
- Fix
--verification-key
handling to accept PEM files (#2796) - Fix
--capabilities
flag inopa build
command (#2848) authored by @srenatus - Fix loading of signed persisted bundles (#2824)
- Fix API response mutation caused by decision log masking (#2752) authored by @gshively11
- Fix evaluator to prevent
with
statements from mutating originalinput
document (#2813) - Fix set iteration runtime to be O(n) instead of O(n^2) (#2966)
- Increased OPA version telemetry report timeout from 1 second to 5 seconds to deal with slow networks
Documentation
- Improve docs to mention built-in function support in WebAssembly compiled policies
- Improve docs around JWT HMAC encoding (#2870) authored by @anderseknert
- Improve HTTP authorization tutorial steps for zsh (#2917 authored by @ClaudenirFreitas)
- Improve docs to describe meaning of Prometheus metrics
- Remove mention of unsafe (and unsupported) "none" signature algorithm from JWT documentation
WebAssembly
This release also includes a number of improvements to the Wasm support in OPA. Importantly, OPA now integrates a Wasm runtime that can be used to execute Wasm compiled policies. The runtime is integrated into the existing "topdown" evaluator so that specific portions of the policy can be compiled to Wasm as a performance optimization. When the evaluator executes a policy using the Wasm runtime it emits a special Wasm
trace event. The Wasm runtime support in OPA is currently considered experimental and will be iterated on in coming releases.
This release also extends the Wasm compiler in OPA to natively support the following built-in functions (in alphabetical order):
base64.encode
,base64.decode
,base64url.encode
, andbase64url.decode
glob.match
json.marshal
andjson.unmarshal
net.cidr_contains
,net.cidr_intersects
, andnet.cidr_overlap
regex.match
,regex.is_valid
, andregex.find_all_string_submatch_n
to_number
walk
Backwards Compatibility
- The
--insecure-addr
flag (which was deprecated in v0.10.0) has been removed completely (#763)
Assets
6
This release contains a number of small enhancements and bug fixes.
Bundle Persistence
This release adds support for persisting bundles for recovery purposes. When persistence is enabled, OPA will save activated bundles to disk. On startup, OPA checks for persisted bundles and activates them immediately. This allows OPA to startup if the bundle server is unavailable (#2097). For more information see the Bundle documentation.
Built-in Functions
This release includes a few new built-in functions:
base64.is_valid
for testing if strings are valid base64 encodings (#2690) authored by @carlpettnet.cidr_merge function
for merging sets of IPs and CIDRs (#2692)urlquery.decode_object
for parsing URL query parameters into objects (#2647) authored by @GBrawl
In addition, http.send
has been enhanced to support caching overrides and in-band error handling (#2666 and #2187).
Fixes
- Fix
opa build
to support custom built-in functions (#2738) authored by @gshively11 - Fix for file watching volume mounted configmaps (#2588) authored by @drewwells
- Fix discovery plugin to set last request and last successful request timestamps in status updates (#2630)
- Fix planner crash on virtual document iteration (#2601)
- Fix decision logger to requeue failed chunks (#2724 authored by @anderseknert)
- Fix object/set implementation in WASM-C library to avoid resizing.
- Fix JSON parser in WASM-C library to copy memory for strings and numbers.
- Improve WASM-C library to recycle object and set element structures while growing.
In addition, this release contains several fixes for panics identified by fuzzing:
- ast: Fix compiler to expand exprs in rule args (#2649)
- ast: Fix output var analysis to accept refs with non-var heads (#2678)
- ast: Fix panic during local var rewriting (#2720)
- ast: Fix panic in local var rewriting caused by object corruption (#2661)
- ast: Fix panic in parser post-processing of expressions (#2714)
- ast: Fix parser to ignore rules with args and key in head (#2662)
- ast: Fix object corruption during safety reordering
- types: Fix panic on reference to object with composite key (#2648)
Backwards Compatibility
- Renamed
timer_rego_builtin_http.send_ns
totimer_rego_builtin_http_send_ns
to avoid issues with periods in metric keys. - Removed deprecated
watch
package (#2265)