You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Simple regexp length error not caught by Gateway #2543
I did some looking into this. I've seen this in a few places as well as issues opened because 1000 is also not high enough. For example, here are a couple "high cost" expressions:
^/api/v1/object/[^\/]{124}/action$ has program size of 1003
^/static/[a-zA-Z0-9._-]{0,200}/[a-zA-Z0-9._-]{0,200}$ has program size of 2005
Size can increase linearly based on the length of a single match due to how cost is calculated in RE2. For the first example the cost increases by 8 as the length of the match is incremented, or: 11 + 8 * 124 == 1003
Program size of regular expressions can also apparently change between RE2 versions, which happened when the cost of . was reduced in 2019 breaking a test in Envoy (xref9174#issue-530487212)
Kuma raised the error level configured in Envoy to uint32 max value with warn level of 1000 a while back:
Contour did similar and set the warning level to 1000 with an error level of 2 ^ 20 at some point (xref) although it may have changed since then.
As it stands currently this is a serious stability issue, so what I'd like to propose is initially we configure error_level to UINT32_MAX and warn_level to 1000 in the bootstrap template:
With this the error should no longer occur as Envoy will effectively not enforce the deprecated max_program_size option, but will continue to warn for higher complexity matches:
[2024-02-07 17:17:51.358][1][warning][misc] [source/common/common/regex.cc:41] regex '^/static/[a-zA-Z0-9._-]{0,200}/[a-zA-Z0-9._-]{0,200}$' RE2 program size of 2005 > max program size of 1000 set for the warn level threshold. Increase configured max program size if necessary.
[2024-02-07 17:17:51.358][1][warning][misc] [source/common/common/regex.cc:41] regex '^/api/v1/object/[^\/]{124}/action$' RE2 program size of 1003 > max program size of 1000 set for the warn level threshold. Increase configured max program size if necessary.
Envoy will also emit a counter re2.exceeded_warn_level, incremented each time the warn level threshold is exceeded providing visibility into how often this is exceeded.
In the future regex validation and optionally configurable program size limits could be implemented to enforce in the gateway controller to prevent publishing invalid or costly regex matches in the XDS. Envoy proxy comments seem to indicate this is the expected path forward:
// This field is deprecated; regexp validation should be performed on the management server
// instead of being done by each individual client.
If this sounds reasonable, I'll open an MR to add the max_program_size settings to the bootstrap template.
thanks for digging deeper into this @davidalger
agree with the approach of setting error_level to UINT32_MAX and warn_level to 1000 for now
either in the same PR or different, lets also highlight this in the control plane as a status
using https://pkg.go.dev/regexp/syntax (len(prog.Inst) is a close approximation)
Activity
[-]Simple regexp lenght error not caught by Gateway[/-][+]Simple regexp length error not caught by Gateway[/+]zirain commentedon Jan 31, 2024
will custom bootstrap help this? should EG make
runtime_flagsconfigurable?akhenakh commentedon Jan 31, 2024
It seems other gateways are defaulting to 1000.
In the meantime yes I'm trying to change via replacing the bootstrap...
static_layer:
re2.max_program_size.error_level: 1000
arkodg commentedon Jan 31, 2024
thanks for raising this issue, looks like the outcome of this issue is
re2.max_program_size.error_levelshould be for the default caseas mentioned by zirain, workaround for now is to use a custom bootstrap
https://gateway.envoyproxy.io/latest/user/customize-envoyproxy/#customize-envoyproxy-bootstrap-config
akhenakh commentedon Jan 31, 2024
For refs this is a working config:
davidalger commentedon Feb 7, 2024
I did some looking into this. I've seen this in a few places as well as issues opened because 1000 is also not high enough. For example, here are a couple "high cost" expressions:
^/api/v1/object/[^\/]{124}/action$has program size of 1003^/static/[a-zA-Z0-9._-]{0,200}/[a-zA-Z0-9._-]{0,200}$has program size of 2005Size can increase linearly based on the length of a single match due to how cost is calculated in RE2. For the first example the cost increases by 8 as the length of the match is incremented, or:
11 + 8 * 124 == 1003Program size of regular expressions can also apparently change between RE2 versions, which happened when the cost of
.was reduced in 2019 breaking a test in Envoy (xref9174#issue-530487212)Kuma raised the error level configured in Envoy to uint32 max value with warn level of 1000 a while back:
Contour did similar and set the warning level to
1000with an error level of2 ^ 20at some point (xref) although it may have changed since then.As it stands currently this is a serious stability issue, so what I'd like to propose is initially we configure
error_levelto UINT32_MAX andwarn_levelto 1000 in the bootstrap template:With this the error should no longer occur as Envoy will effectively not enforce the deprecated
max_program_sizeoption, but will continue to warn for higher complexity matches:Envoy will also emit a counter
re2.exceeded_warn_level, incremented each time the warn level threshold is exceeded providing visibility into how often this is exceeded.In the future regex validation and optionally configurable program size limits could be implemented to enforce in the gateway controller to prevent publishing invalid or costly regex matches in the XDS. Envoy proxy comments seem to indicate this is the expected path forward:
If this sounds reasonable, I'll open an MR to add the
max_program_sizesettings to the bootstrap template.arkodg commentedon Feb 7, 2024
thanks for digging deeper into this @davidalger
agree with the approach of setting error_level to UINT32_MAX and warn_level to 1000 for now
either in the same PR or different, lets also highlight this in the control plane as a status
using https://pkg.go.dev/regexp/syntax (
len(prog.Inst)is a close approximation)davidalger commentedon Feb 7, 2024
Submitted a PR to amend the bootstrap config. I'll look into validation and exposing in route status as a followup.