randomly failing requests with http2 and server sent events #231
Comments
|
Thanks for the analysis and logs. Indeed, the question seems to be why the stream output beam has been aborted before any response was produced. Do you see any lines with |
|
hello, yes i can see such log lines (270-9 failed, 270-11 succeeded): i also see logs where it says increasing worker, not decreasing: all these lines appear between reading/receiving the request and then, about 0.2 seconds later, when they are beeing "processesd".. however, would this mean that we need to increase H2MaxWorkers or H2MaxSessionStreams? im sure we tried out increasing H2MaxWorkers and it didn't help... |
|
Thanks for digging these up!
No, that does not really change things. I assume you can build the module from source. If you remove lines 1009-1011 in h2_mplx.c, I hope you see the problems go away. These lines in |
|
hi, it works! we have tried this change and i can confirm that the problem is no longer reproducible. will this become a permanent change or is this a fix for our setup only? |
|
I will add this to the server and backport for the next release. Thanks for verifying! |
icing
pushed a commit
that referenced
this issue
May 5, 2022
potential abuse by a client. This proves counter-productive and the abuse detection can be falsly triggered by request using server-side-events. Fixes #231.
icing
pushed a commit
to icing/httpd
that referenced
this issue
May 5, 2022
connection shows potential abuse by a client. This proved
counter-productive and the abuse detection can false flag
requests using server-side-events.
Fixes <icing/mod_h2#231>.
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this issue
Jun 9, 2022
Changes with Apache 2.4.54 *) SECURITY: CVE-2022-31813: mod_proxy X-Forwarded-For dropped by hop-by-hop mechanism (cve.mitre.org) Apache HTTP Server 2.4.53 and earlier may not send the X-Forwarded-* headers to the origin server based on client side Connection header hop-by-hop mechanism. This may be used to bypass IP based authentication on the origin server/application. Credits: The Apache HTTP Server project would like to thank Gaetan Ferry (Synacktiv) for reporting this issue *) SECURITY: CVE-2022-30556: Information Disclosure in mod_lua with websockets (cve.mitre.org) Apache HTTP Server 2.4.53 and earlier may return lengths to applications calling r:wsread() that point past the end of the storage allocated for the buffer. Credits: The Apache HTTP Server project would like to thank Ronald Crane (Zippenhop LLC) for reporting this issue *) SECURITY: CVE-2022-30522: mod_sed denial of service (cve.mitre.org) If Apache HTTP Server 2.4.53 is configured to do transformations with mod_sed in contexts where the input to mod_sed may be very large, mod_sed may make excessively large memory allocations and trigger an abort. Credits: This issue was found by Brian Moussalli from the JFrog Security Research team *) SECURITY: CVE-2022-29404: Denial of service in mod_lua r:parsebody (cve.mitre.org) In Apache HTTP Server 2.4.53 and earlier, a malicious request to a lua script that calls r:parsebody(0) may cause a denial of service due to no default limit on possible input size. Credits: The Apache HTTP Server project would like to thank Ronald Crane (Zippenhop LLC) for reporting this issue *) SECURITY: CVE-2022-28615: Read beyond bounds in ap_strcmp_match() (cve.mitre.org) Apache HTTP Server 2.4.53 and earlier may crash or disclose information due to a read beyond bounds in ap_strcmp_match() when provided with an extremely large input buffer. While no code distributed with the server can be coerced into such a call, third-party modules or lua scripts that use ap_strcmp_match() may hypothetically be affected. Credits: The Apache HTTP Server project would like to thank Ronald Crane (Zippenhop LLC) for reporting this issue *) SECURITY: CVE-2022-28614: read beyond bounds via ap_rwrite() (cve.mitre.org) The ap_rwrite() function in Apache HTTP Server 2.4.53 and earlier may read unintended memory if an attacker can cause the server to reflect very large input using ap_rwrite() or ap_rputs(), such as with mod_luas r:puts() function. Credits: The Apache HTTP Server project would like to thank Ronald Crane (Zippenhop LLC) for reporting this issue *) SECURITY: CVE-2022-28330: read beyond bounds in mod_isapi (cve.mitre.org) Apache HTTP Server 2.4.53 and earlier on Windows may read beyond bounds when configured to process requests with the mod_isapi module. Credits: The Apache HTTP Server project would like to thank Ronald Crane (Zippenhop LLC) for reporting this issue *) SECURITY: CVE-2022-26377: mod_proxy_ajp: Possible request smuggling (cve.mitre.org) Inconsistent Interpretation of HTTP Requests ('HTTP Request Smuggling') vulnerability in mod_proxy_ajp of Apache HTTP Server allows an attacker to smuggle requests to the AJP server it forwards requests to. This issue affects Apache HTTP Server Apache HTTP Server 2.4 version 2.4.53 and prior versions. Credits: Ricter Z @ 360 Noah Lab *) mod_ssl: SSLFIPS compatible with OpenSSL 3.0. *) mod_proxy_http: Avoid 417 responses for non forwardable 100-continue. *) mod_md: a bug was fixed that caused very large MDomains with the combined DNS names exceeding ~7k to fail, as request bodies would contain partially wrong data from uninitialized memory. This would have appeared as failure in signing-up/renewing such configurations. *) mod_proxy_http: Avoid 417 responses for non forwardable 100-continue. *) MPM event: Restart children processes killed before idle maintenance. *) ab: Allow for TLSv1.3 when the SSL library supports it. *) core: Disable TCP_NOPUSH optimization on OSX since it might introduce transmission delays. *) MPM event: Fix accounting of active/total processes on ungraceful restart, *) core: make ap_escape_quotes() work correctly on strings with more than MAX_INT/2 characters, counting quotes double. Credit to <generalbugs@zippenhop.com> for finding this. *) mod_md: the `MDCertificateAuthority` directive can take more than one URL/name of an ACME CA. This gives a failover for renewals when several consecutive attempts to get a certificate failed. A new directive was added: `MDRetryDelay` sets the delay of retries. A new directive was added: `MDRetryFailover` sets the number of errored attempts before an alternate CA is selected for certificate renewals. *) mod_http2: remove unused and insecure code. *) mod_proxy: Add backend port to log messages to ease identification of involved service. *) mod_http2: removing unscheduling of ongoing tasks when connection shows potential abuse by a client. This proved counter-productive and the abuse detection can false flag requests using server-side-events. Fixes <icing/mod_h2#231>. *) mod_md: Implement full auto status ("key: value" type status output). Especially not only status summary counts for certificates and OCSP stapling but also lists. Auto status format is similar to what was used for mod_proxy_balancer. *) mod_md: fixed a bug leading to failed transfers for OCSP stapling information when more than 6 certificates needed updates in the same run. *) mod_proxy: Set a status code of 502 in case the backend just closed the connection in reply to our forwarded request. *) mod_md: a possible NULL pointer deref was fixed in the JSON code for persisting time periods (start+end). Fixes #282 on mod_md's github. Thanks to @marcstern for finding this. *) mod_heartmonitor: Set the documented default value "10" for HeartbeatMaxServers instead of "0". With "0" no shared memory slotmem was initialized. *) mod_md: added support for managing certificates via a local tailscale daemon for users of that secure networking. This gives trusted certificates for tailscale assigned domain names in the *.ts.net space.
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this issue
Jun 11, 2022
www/apache24: security fix Revisions pulled up: - www/apache24/Makefile 1.111 - www/apache24/distinfo 1.53 --- Module Name: pkgsrc Committed By: adam Date: Thu Jun 9 18:15:51 UTC 2022 Modified Files: pkgsrc/www/apache24: Makefile distinfo Log Message: apache24: updated to 2.4.54 Changes with Apache 2.4.54 *) SECURITY: CVE-2022-31813: mod_proxy X-Forwarded-For dropped by hop-by-hop mechanism (cve.mitre.org) Apache HTTP Server 2.4.53 and earlier may not send the X-Forwarded-* headers to the origin server based on client side Connection header hop-by-hop mechanism. This may be used to bypass IP based authentication on the origin server/application. Credits: The Apache HTTP Server project would like to thank Gaetan Ferry (Synacktiv) for reporting this issue *) SECURITY: CVE-2022-30556: Information Disclosure in mod_lua with websockets (cve.mitre.org) Apache HTTP Server 2.4.53 and earlier may return lengths to applications calling r:wsread() that point past the end of the storage allocated for the buffer. Credits: The Apache HTTP Server project would like to thank Ronald Crane (Zippenhop LLC) for reporting this issue *) SECURITY: CVE-2022-30522: mod_sed denial of service (cve.mitre.org) If Apache HTTP Server 2.4.53 is configured to do transformations with mod_sed in contexts where the input to mod_sed may be very large, mod_sed may make excessively large memory allocations and trigger an abort. Credits: This issue was found by Brian Moussalli from the JFrog Security Research team *) SECURITY: CVE-2022-29404: Denial of service in mod_lua r:parsebody (cve.mitre.org) In Apache HTTP Server 2.4.53 and earlier, a malicious request to a lua script that calls r:parsebody(0) may cause a denial of service due to no default limit on possible input size. Credits: The Apache HTTP Server project would like to thank Ronald Crane (Zippenhop LLC) for reporting this issue *) SECURITY: CVE-2022-28615: Read beyond bounds in ap_strcmp_match() (cve.mitre.org) Apache HTTP Server 2.4.53 and earlier may crash or disclose information due to a read beyond bounds in ap_strcmp_match() when provided with an extremely large input buffer. While no code distributed with the server can be coerced into such a call, third-party modules or lua scripts that use ap_strcmp_match() may hypothetically be affected. Credits: The Apache HTTP Server project would like to thank Ronald Crane (Zippenhop LLC) for reporting this issue *) SECURITY: CVE-2022-28614: read beyond bounds via ap_rwrite() (cve.mitre.org) The ap_rwrite() function in Apache HTTP Server 2.4.53 and earlier may read unintended memory if an attacker can cause the server to reflect very large input using ap_rwrite() or ap_rputs(), such as with mod_luas r:puts() function. Credits: The Apache HTTP Server project would like to thank Ronald Crane (Zippenhop LLC) for reporting this issue *) SECURITY: CVE-2022-28330: read beyond bounds in mod_isapi (cve.mitre.org) Apache HTTP Server 2.4.53 and earlier on Windows may read beyond bounds when configured to process requests with the mod_isapi module. Credits: The Apache HTTP Server project would like to thank Ronald Crane (Zippenhop LLC) for reporting this issue *) SECURITY: CVE-2022-26377: mod_proxy_ajp: Possible request smuggling (cve.mitre.org) Inconsistent Interpretation of HTTP Requests ('HTTP Request Smuggling') vulnerability in mod_proxy_ajp of Apache HTTP Server allows an attacker to smuggle requests to the AJP server it forwards requests to. This issue affects Apache HTTP Server Apache HTTP Server 2.4 version 2.4.53 and prior versions. Credits: Ricter Z @ 360 Noah Lab *) mod_ssl: SSLFIPS compatible with OpenSSL 3.0. *) mod_proxy_http: Avoid 417 responses for non forwardable 100-continue. *) mod_md: a bug was fixed that caused very large MDomains with the combined DNS names exceeding ~7k to fail, as request bodies would contain partially wrong data from uninitialized memory. This would have appeared as failure in signing-up/renewing such configurations. *) mod_proxy_http: Avoid 417 responses for non forwardable 100-continue. *) MPM event: Restart children processes killed before idle maintenance. *) ab: Allow for TLSv1.3 when the SSL library supports it. *) core: Disable TCP_NOPUSH optimization on OSX since it might introduce transmission delays. *) MPM event: Fix accounting of active/total processes on ungraceful restart, *) core: make ap_escape_quotes() work correctly on strings with more than MAX_INT/2 characters, counting quotes double. Credit to <generalbugs@zippenhop.com> for finding this. *) mod_md: the `MDCertificateAuthority` directive can take more than one URL/name of an ACME CA. This gives a failover for renewals when several consecutive attempts to get a certificate failed. A new directive was added: `MDRetryDelay` sets the delay of retries. A new directive was added: `MDRetryFailover` sets the number of errored attempts before an alternate CA is selected for certificate renewals. *) mod_http2: remove unused and insecure code. *) mod_proxy: Add backend port to log messages to ease identification of involved service. *) mod_http2: removing unscheduling of ongoing tasks when connection shows potential abuse by a client. This proved counter-productive and the abuse detection can false flag requests using server-side-events. Fixes <icing/mod_h2#231>. *) mod_md: Implement full auto status ("key: value" type status output). Especially not only status summary counts for certificates and OCSP stapling but also lists. Auto status format is similar to what was used for mod_proxy_balancer. *) mod_md: fixed a bug leading to failed transfers for OCSP stapling information when more than 6 certificates needed updates in the same run. *) mod_proxy: Set a status code of 502 in case the backend just closed the connection in reply to our forwarded request. *) mod_md: a possible NULL pointer deref was fixed in the JSON code for persisting time periods (start+end). Fixes #282 on mod_md's github. Thanks to @marcstern for finding this. *) mod_heartmonitor: Set the documented default value "10" for HeartbeatMaxServers instead of "0". With "0" no shared memory slotmem was initialized. *) mod_md: added support for managing certificates via a local tailscale daemon for users of that secure networking. This gives trusted certificates for tailscale assigned domain names in the *.ts.net space.
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this issue
Jan 18, 2023
www/apache24: security fix Revisions pulled up: - www/apache24/Makefile 1.111 - www/apache24/distinfo 1.53 --- Module Name: pkgsrc Committed By: adam Date: Thu Jun 9 18:15:51 UTC 2022 Modified Files: pkgsrc/www/apache24: Makefile distinfo Log Message: apache24: updated to 2.4.54 Changes with Apache 2.4.54 *) SECURITY: CVE-2022-31813: mod_proxy X-Forwarded-For dropped by hop-by-hop mechanism (cve.mitre.org) Apache HTTP Server 2.4.53 and earlier may not send the X-Forwarded-* headers to the origin server based on client side Connection header hop-by-hop mechanism. This may be used to bypass IP based authentication on the origin server/application. Credits: The Apache HTTP Server project would like to thank Gaetan Ferry (Synacktiv) for reporting this issue *) SECURITY: CVE-2022-30556: Information Disclosure in mod_lua with websockets (cve.mitre.org) Apache HTTP Server 2.4.53 and earlier may return lengths to applications calling r:wsread() that point past the end of the storage allocated for the buffer. Credits: The Apache HTTP Server project would like to thank Ronald Crane (Zippenhop LLC) for reporting this issue *) SECURITY: CVE-2022-30522: mod_sed denial of service (cve.mitre.org) If Apache HTTP Server 2.4.53 is configured to do transformations with mod_sed in contexts where the input to mod_sed may be very large, mod_sed may make excessively large memory allocations and trigger an abort. Credits: This issue was found by Brian Moussalli from the JFrog Security Research team *) SECURITY: CVE-2022-29404: Denial of service in mod_lua r:parsebody (cve.mitre.org) In Apache HTTP Server 2.4.53 and earlier, a malicious request to a lua script that calls r:parsebody(0) may cause a denial of service due to no default limit on possible input size. Credits: The Apache HTTP Server project would like to thank Ronald Crane (Zippenhop LLC) for reporting this issue *) SECURITY: CVE-2022-28615: Read beyond bounds in ap_strcmp_match() (cve.mitre.org) Apache HTTP Server 2.4.53 and earlier may crash or disclose information due to a read beyond bounds in ap_strcmp_match() when provided with an extremely large input buffer. While no code distributed with the server can be coerced into such a call, third-party modules or lua scripts that use ap_strcmp_match() may hypothetically be affected. Credits: The Apache HTTP Server project would like to thank Ronald Crane (Zippenhop LLC) for reporting this issue *) SECURITY: CVE-2022-28614: read beyond bounds via ap_rwrite() (cve.mitre.org) The ap_rwrite() function in Apache HTTP Server 2.4.53 and earlier may read unintended memory if an attacker can cause the server to reflect very large input using ap_rwrite() or ap_rputs(), such as with mod_luas r:puts() function. Credits: The Apache HTTP Server project would like to thank Ronald Crane (Zippenhop LLC) for reporting this issue *) SECURITY: CVE-2022-28330: read beyond bounds in mod_isapi (cve.mitre.org) Apache HTTP Server 2.4.53 and earlier on Windows may read beyond bounds when configured to process requests with the mod_isapi module. Credits: The Apache HTTP Server project would like to thank Ronald Crane (Zippenhop LLC) for reporting this issue *) SECURITY: CVE-2022-26377: mod_proxy_ajp: Possible request smuggling (cve.mitre.org) Inconsistent Interpretation of HTTP Requests ('HTTP Request Smuggling') vulnerability in mod_proxy_ajp of Apache HTTP Server allows an attacker to smuggle requests to the AJP server it forwards requests to. This issue affects Apache HTTP Server Apache HTTP Server 2.4 version 2.4.53 and prior versions. Credits: Ricter Z @ 360 Noah Lab *) mod_ssl: SSLFIPS compatible with OpenSSL 3.0. *) mod_proxy_http: Avoid 417 responses for non forwardable 100-continue. *) mod_md: a bug was fixed that caused very large MDomains with the combined DNS names exceeding ~7k to fail, as request bodies would contain partially wrong data from uninitialized memory. This would have appeared as failure in signing-up/renewing such configurations. *) mod_proxy_http: Avoid 417 responses for non forwardable 100-continue. *) MPM event: Restart children processes killed before idle maintenance. *) ab: Allow for TLSv1.3 when the SSL library supports it. *) core: Disable TCP_NOPUSH optimization on OSX since it might introduce transmission delays. *) MPM event: Fix accounting of active/total processes on ungraceful restart, *) core: make ap_escape_quotes() work correctly on strings with more than MAX_INT/2 characters, counting quotes double. Credit to <generalbugs@zippenhop.com> for finding this. *) mod_md: the `MDCertificateAuthority` directive can take more than one URL/name of an ACME CA. This gives a failover for renewals when several consecutive attempts to get a certificate failed. A new directive was added: `MDRetryDelay` sets the delay of retries. A new directive was added: `MDRetryFailover` sets the number of errored attempts before an alternate CA is selected for certificate renewals. *) mod_http2: remove unused and insecure code. *) mod_proxy: Add backend port to log messages to ease identification of involved service. *) mod_http2: removing unscheduling of ongoing tasks when connection shows potential abuse by a client. This proved counter-productive and the abuse detection can false flag requests using server-side-events. Fixes <icing/mod_h2#231>. *) mod_md: Implement full auto status ("key: value" type status output). Especially not only status summary counts for certificates and OCSP stapling but also lists. Auto status format is similar to what was used for mod_proxy_balancer. *) mod_md: fixed a bug leading to failed transfers for OCSP stapling information when more than 6 certificates needed updates in the same run. *) mod_proxy: Set a status code of 502 in case the backend just closed the connection in reply to our forwarded request. *) mod_md: a possible NULL pointer deref was fixed in the JSON code for persisting time periods (start+end). Fixes #282 on mod_md's github. Thanks to @marcstern for finding this. *) mod_heartmonitor: Set the documented default value "10" for HeartbeatMaxServers instead of "0". With "0" no shared memory slotmem was initialized. *) mod_md: added support for managing certificates via a local tailscale daemon for users of that secure networking. This gives trusted certificates for tailscale assigned domain names in the *.ts.net space.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
hello,
we face a strange behavior where some http2 requests fail with the error
unfortunately it seems to be completely random when/which requests fail. we can force it by clicking around in the web application, but we were not able identify a clear way to reproduce. even exactly identical requests from the client sometimes work and sometimes fail. this makes it very difficult to compare "good" cases with "bad" cases...
httpd is version 2.4.53, mod_http2 v1.15.26
(apache is acting as a reverse proxy to a back end application server..)
what we can tell so far is:
we have tested different H2 directives like increasing workers, window size and timeouts, but these all don't seem to have an impact on the issue...
to track the issue down we have enabled tracing in the module:
what i found out of this is:
the log line
originates from the function h2_stream_out_prepare here , so this means that at the time the function h2_beam_receive (the next line in the linked code) is called, the aborted=1 flag is already set. so when i have a look at the code there, i think this results in the function returning the status APR_ECONNABORTED.
as far as i understand this also means that h2_stream_out_prepare returns this status to its caller on_stream_resume in h2_session.c (the caller is visible from the trace log "on_resume").
this condition is then false, because otherwise we would see more output in the trace log. (originated from add_buffered_data function in h2_stream.c)
so i think all that means, that the next two conditions are true :
so for me, the main question is, why is aborted=1 so early ?
what does it mean "we have data to send, but no response"?
maybe the condition
status != APR_EAGAINis wrong, or maybe the case that the state at this point is APR_ECONNABORTED should not even ever happen?for completeness, a trace log of a "good" case, from the comparison i can't really see much differences (except the aborted=1 flag... )
(the stream then continues)
any ideas what could be the issue here, or what we can do to track it more down can be helpful.
also maybe you can read out something more of these trace logs with your detailed knowledge of the module.
the fact that it is not clearly reproducible, and that everything works fine without server sent events somehow lets me think of a race condition or something similar happening when back end sends these events. on the other hand it works with http1, so this is a clear indicator that the mod_http2 is causing the error.
also, its very strange that the server sent events somehow have an impact on other requests made from the client. because the actual request causing the error is not an SSE response its a "normal" http2 request from the client...
the trace log is huge, i tried to only paste the necessary parts here, but let me know if you are missing something.
thanks a lot for your help!
The text was updated successfully, but these errors were encountered: