Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mod_proxy_http2: gRPC bi-directional messages stuck #207

Closed
tschmittni opened this issue Oct 19, 2020 · 26 comments
Closed

mod_proxy_http2: gRPC bi-directional messages stuck #207

tschmittni opened this issue Oct 19, 2020 · 26 comments

Comments

@tschmittni
Copy link

Hi,

maybe you can help me out, I am using gRPC bi-directional messaging with grpc-dotnet. When I directly connect client and server, the messages from the client are streamed to the server without any problems. Once I change the client configuration to connect over the mod_proxy_http2 port, the messages are stuck. It seems like they are buffered because once I start sending messages from the server to the client, sometimes the client messages are delivered, too (depending on the timing) and when I shut down the gRPC server, messages also get delivered sometimes.

Here is my apache configuration:

Protocols h2
SSLProxyEngine On

<Location /tunnel.TunnelMessaging>
    Require all granted
    
    ProxyPass h2://localhost:5001/tunnel.TunnelMessaging retry=0
    ProxyPassReverse https://localhost:5001/tunnel.TunnelMessaging
</Location>

And the debug log output:

Any help would be appreciated!

@icing
Copy link
Owner

icing commented Oct 20, 2020

What versions of apache and mod_proxy_http2 are you running?

@tschmittni
Copy link
Author

Apache 2.4.46

@icing
Copy link
Owner

icing commented Oct 20, 2020

The logs looks as if the client sends a POST with 27 bytes of body, but does not set the end-of-stream flag (flags=0):

recv FRAME[DATA[length=27, flags=0, stream=1, padlen=0]], frames=4/3

and sends nothing ever after. This is buffered, it seems, waiting for more data from the client or an EOS indicator.

@tschmittni
Copy link
Author

Thank you for looking through my logs! Yeah, that is exactly what I am seeing.

I have to look for the gRPC specification but my guess would be that with bi-directional streaming the EOS is not sent because the client could potentially stream additional payloads. Shouldn't the mod_proxy_http2 eventually flush the buffer and forward it downstream? Is there a way to disable buffering so I can validate that this is the root-cause?

@icing
Copy link
Owner

icing commented Oct 20, 2020

No, there is no option to disable this.

I am not very familiar with the gRPC spec, but do they not use a h2 stream for each procedure call? They need to EOS the request stream, but might never have noticed when talking to their own server.

@tschmittni
Copy link
Author

For unary-calls I think this is the case. Each procedure call finishes with EOS. But I don't think this is the case for streaming when the client keeps sending messages over the same h2 stream (as far as I know)

looking here:

gRPC bidirectional streaming can be used to replace unary gRPC calls in high-performance scenarios. Once a bidirectional stream has started, streaming messages back and forth is faster than sending messages with multiple unary gRPC calls. Streamed messages are sent as data on an existing HTTP/2 request and eliminates the overhead of creating a new HTTP/2 request for each unary call.

@icing
Copy link
Owner

icing commented Oct 20, 2020

That is certainly more performant when it works.

/irony off

As you found out, the assumption that gRPC makes here about data handling behaviour is not shared by everyone. I did not find this "long streaming" requests in the original grpc docs. Is this an invention by Microsoft? Looks like it.

@tschmittni
Copy link
Author

tschmittni commented Oct 20, 2020

I just validated that this is not Microsoft-specific. I have just picked a random bi-directional streaming example which is written in golang (also to make sure I didn't do anything wrong in my example):
e.g. https://github.com/pahanini/go-grpc-bidirectional-streaming-example

With exactly the same result. It looks like this is just how gRPC-based streaming works.

Could you give me some pointers at the proxy_http2 code base where I could try out disabling buffering? Or forward the data once I receive a data frame..

@tschmittni
Copy link
Author

btw. I think other HTTP2/gRPC proxies actually have custom handling for gRPC streaming, e.g. HAProxy (see https://www.haproxy.com/blog/haproxy-1-9-2-adds-grpc-support/)

When streaming data from the client to the server, be sure not to set option http-buffer-request. This would pause HAProxy until it receives the full request body, which, when streaming, will be a long time in coming.

@icing
Copy link
Owner

icing commented Oct 20, 2020

If I make a test implementation of such a feature, can you build from here and verify it?

@tschmittni
Copy link
Author

Would be awesome if you were willing to spend the time on this. Sure thing, I am happy to help out. I can build it from here and test against the .net and golang gRPC implementations.

@tschmittni
Copy link
Author

Have you had a chance to look into it? If you don't have time, maybe you can give me some directions so that I can try to put up a PR...

@icing
Copy link
Owner

icing commented Oct 28, 2020

Thanks for the offer. It has been second on my list of things for the last week. Let me get another look, maybe I find a good approach or can give you some pointers.

icing pushed a commit that referenced this issue Oct 28, 2020
…(if there

   is one) can access it immediately. Curious if this alone addresses #207
 * Eliminated some Python deprecation warnings in test code.
@icing
Copy link
Owner

icing commented Oct 28, 2020

You can check the current master if this already solves the problem.

@tschmittni
Copy link
Author

tschmittni commented Oct 28, 2020

Just tried it out, no luck so far. Looks like the on_state_event is not even invoked. I have added some logging around on_state_enter and on_state_event and I only see calls to on_state_enter:

on_state_event_not_invoked.txt

@icing
Copy link
Owner

icing commented Oct 29, 2020

My bad. Will have another shot at it pbly tomorrow.

@icing
Copy link
Owner

icing commented Oct 30, 2020

Could you try the latest master whenever you have the time? Thanks!

@tschmittni
Copy link
Author

tschmittni commented Nov 2, 2020

I have tried it out, but it seems like it doesn't get called there either. See log:
log_output_client_has_sent_message.txt

It looks like it reads blocking from the stream until EOS which does never arrive.

I have also tried to move the flush call in the beginning of the on_event method, there it actually gets called but without any effect:

static int on_event(h2_stream* stream, h2_stream_event_t ev) {
   if (H2_SEV_IN_DATA_PENDING == ev) {
        h2_stream_flush_input(stream);
    }
   if (stream->monitor && stream->monitor->on_event) {
...
}

I have uploaded my C# grpc example which reproduces the problem. The readme explains the apache set up and how to run the example:
https://github.com/tschmittni/grpc-bidirectional-csharp

Maybe this helps you to reproduce and debug the problem.

@icing
Copy link
Owner

icing commented Nov 2, 2020

Thanks for testing. Yes, I agree that I need to write a test case for this. Thanks for the input!

icing pushed a commit that referenced this issue Nov 2, 2020
  - added test case for #207 using new h2test module's echo handler
  - reqeust body data is arriving at handler, but output is still buffered
    and does not reach client. TODO.
icing pushed a commit that referenced this issue Nov 3, 2020
… of stream output.

   The default is on, which is the behaviour of older mod-h2 versions. When off, all
   bytes are made available immediately to the main connection for sending them
   out to the client. This fixes interop issues with certain flavours of gRPC, see
   als #207.
@icing
Copy link
Owner

icing commented Nov 3, 2020

Next iterations: i wrote a test case for the behaviour as I understand it and fixed the buffering. There is a new directive to switch the output buffering off. Please give this a try.

@tschmittni
Copy link
Author

tschmittni commented Nov 4, 2020

Ok, I configured my apache to disable output buffering with H2OutputBuffering Off
But it seems like the same problem. I added logging to the secondary_out method but it doesn't get called at all (until I actually shutdown the connection but not after sending the message from the client).

Does it just wait for more data from the input/client stream? It outputs the following and then just waits for more data:

[Wed Nov 04 15:38:26.594297 2020] [http2:trace2] [pid 11008:tid 2188] h2_stream.c(504): [client ::1:57825] h2_stream(41-1,OPEN): recv DATA, len=32
[Wed Nov 04 15:38:26.594297 2020] [http2:trace2] [pid 11008:tid 2188] h2_stream.c(343): [client ::1:57825] h2_stream(41-1,OPEN): dispatch event 4
[Wed Nov 04 15:38:26.594297 2020] [http2:trace2] [pid 11008:tid 2188] h2_stream.c(355): [client ::1:57825] h2_stream(41-1,OPEN): non-state event 4
[Wed Nov 04 15:38:26.594297 2020] [http2:debug] [pid 11008:tid 2188] h2_session.c(337): [client ::1:57825] AH03066: h2_session(41,BUSY,1): recv FRAME[DATA[length=32, flags=0, stream=1, padlen=0]], frames=4/3 (r/s)
[Wed Nov 04 15:38:26.594297 2020] [http2:debug] [pid 11008:tid 2188] h2_session.c(359): [client ::1:57825] AH02923: h2_stream(41-1,OPEN): DATA, len=32, flags=0
[Wed Nov 04 15:38:26.594297 2020] [http2:trace2] [pid 11008:tid 2188] h2_filter.c(63): [client ::1:57825] h2_session(41,BUSY,1): fed 41 bytes to nghttp2, 41 read
[Wed Nov 04 15:38:26.594297 2020] [http2:trace1] [pid 11008:tid 2188] h2_filter.c(143): [client ::1:57825] h2_session(41): read, NONBLOCK_READ, mode=0, readbytes=65536

once I actually shut down the connection, the stream wakes up and flushes the following output:

[Wed Nov 04 15:39:46.641445 2020] [http2:trace1] [pid 11008:tid 2188] h2_session.c(1694): [client ::1:57825] AH03078: h2_session(41,BUSY,1): transit [WAIT] -- stream change --> [BUSY]
[Wed Nov 04 15:39:46.641445 2020] [http2:trace1] [pid 11008:tid 2188] h2_filter.c(143): [client ::1:57825] h2_session(41): read, NONBLOCK_READ, mode=0, readbytes=65536
[Wed Nov 04 15:39:46.641445 2020] [http2:trace1] [pid 11008:tid 2188] h2_filter.c(190): (11)Resource temporarily unavailable: [client ::1:57825] h2_session(41): read
[Wed Nov 04 15:39:46.641445 2020] [http2:trace2] [pid 11008:tid 2188] h2_mplx.c(1109): [client ::1:57825] h2_mplx(41): dispatch events
[Wed Nov 04 15:39:46.641445 2020] [http2:trace2] [pid 11008:tid 2188] h2_session.c(1511): [client ::1:57825] h2_stream(41-1,OPEN): on_resume
[Wed Nov 04 15:39:46.641445 2020] [http2:trace2] [pid 11008:tid 2188] h2_stream.c(180): [client ::1:57825] h2_stream(41-1,OPEN): out-buffer(add_buffered_data())
[Wed Nov 04 15:39:46.641445 2020] [http2:trace2] [pid 11008:tid 2188] h2_stream.c(180): [client ::1:57825] h2_stream(41-1,OPEN): out-buffer(pre())
[Wed Nov 04 15:39:46.641445 2020] [http2:trace2] [pid 11008:tid 2188] h2_stream.c(180): [client ::1:57825] h2_stream(41-1,OPEN): out-buffer(post(H2HEADERS BEAM[38] FLUSH))
[Wed Nov 04 15:39:46.641445 2020] [http2:trace2] [pid 11008:tid 2188] h2_stream.c(180): [client ::1:57825] h2_stream(41-1,OPEN): out-buffer(add_buffered_data(H2HEADERS BEAM[38] FLUSH))

@icing
Copy link
Owner

icing commented Nov 5, 2020

First of all, thanks for your patience in working this out.

I added a test case for a mod_proxy_http2 setup and this also works for me. To eliminate that the new buffering flag somehow is not in effect for you, I added a TRACE2 statement that gives the flag value on output.

Please let me know what you see on your system. Thanks!

@tschmittni
Copy link
Author

tschmittni commented Nov 5, 2020

No worries, thanks you a lot for your quick feedback!

I've added already the exact same log entry and I even put a log entry in the beginning of the secondary_out function. And it looks like the function is not called at all. (just when I close the connection, that's how I know the logging was working, and once I close the connection the data flushed either way, whether the buffering flag is on or off).

I also debugged a bit and if I understand the code right, the buffering is already happening earlier while reading from the input stream and the stream just goes to sleep waiting for more data.

@icing
Copy link
Owner

icing commented Nov 9, 2020

I have a test case 712_02 added that h2 proxies against itself. I send a h2 request and chunk several input pieces, each 300ms apart and watch when the output arrives. This works fine. I seem to be missing something in our traffic flow that is different.

For you to compare I sent you the log of the test case. Maybe you can spot what is going on here...
test_712_02-error.log

asfgit pushed a commit to apache/httpd that referenced this issue Mar 24, 2021
     - Log requests and sent the configured error response in case of early detected
       errors like too many or too long headers. [Ruediger Pluem]
     - new option 'H2OutputBuffering on/off' which controls the buffering of stream output.
       The default is on, which is the behaviour of older mod-h2 versions. When off, all
       bytes are made available immediately to the main connection for sending them
       out to the client. This fixes interop issues with certain flavours of gRPC, see
       also <icing/mod_h2#207>.



git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x@1888011 13f79535-47bb-0310-9956-ffa450edef68
clrpackages pushed a commit to clearlinux-pkgs/httpd that referenced this issue May 5, 2021
Changes with Apache 2.4.47

  *) mod_dav_fs: Improve logging output when failing to open files for
     writing.  PR 64413.  [Bingyu Shen <ahshenbingyu gmail.com>]

  *) mod_http2: Fixed a race condition that could lead to streams being
     aborted (RST to the client), although a response had been produced.
     [Stefan Eissing]

  *) mod_lua: Add support to Lua 5.4  [Joe Orton, Giovanni Bechis, Ruediger Pluem]

  *) MPM event/worker: Fix possible crash in child process on early signal
     delivery.  PR 64533.  [Ruediger Pluem]

  *) mod_http2: sync with github standalone version 1.15.17
     - Log requests and sent the configured error response in case of early detected
       errors like too many or too long headers. [Ruediger Pluem]
     - new option 'H2OutputBuffering on/off' which controls the buffering of stream output.
       The default is on, which is the behaviour of older mod-h2 versions. When off, all
       bytes are made available immediately to the main connection for sending them
       out to the client. This fixes interop issues with certain flavours of gRPC, see
       also <icing/mod_h2#207>.
       [Stefan Eissing]

  *) mod_unique_id: Fix potential duplicated ID generation under heavy load.
     PR 65159
     [Jonas Müntener <jonas.muentener ergon.ch>, Christophe Jaillet]

  *) "[mod_dav_fs etag handling] should really honor the FileETag setting".
     - It now does.
     - Add "Digest" to FileETag directive, allowing a strong ETag to be
       generated using a file digest.
     - Add ap_make_etag_ex() and ap_set_etag_fd() to allow full control over
       ETag generation.
     - Add concept of "binary notes" to request_rec, allowing packed bit flags
       to be added to a request.
     - First binary note - AP_REQUEST_STRONG_ETAG - allows modules to force
       the ETag to a strong ETag to comply with RFC requirements, such as those
       mandated by various WebDAV extensions.
     [Graham Leggett]

  *) mod_proxy_http: Fix a possibly crash when the origin connection gets
     interrupted before completion.  PR 64234.
     [Barnim Dzwillo <dzwillo strato.de>, Ruediger Pluem]

  *) mod_ssl: Do not keep connections to OCSP responders alive when doing
     OCSP requests.  PR 64135.  [Ruediger Pluem]

  *) mod_ssl: Improve the coalescing filter to buffer into larger TLS
     records, and avoid revealing the HTTP header size via TLS record
     boundaries (for common response generators).
     [Joe Orton, Ruediger Pluem]

  *) mod_proxy_hcheck: Don't pile up health checks if the previous one did
     not finish before hcinterval.  PR 63010.  [Yann Ylavic]

  *) mod_session: Improve session parsing.  [Yann Yalvic]

  *) mod_authnz_ldap: Prevent authentications with empty passwords for the
     initial bind to fail with status 500. [Ruediger Pluem]

  *) mod_auth_digest: Fast validation of the nonce's base64 to fail early if
     the format can't match anyway.  [Yann Ylavic]

  *) mod_proxy_fcgi: Honor "SetEnv proxy-sendcl" to forward a chunked
     Transfer-Encoding from the client, spooling the request body when needed
     to provide a Content-Length to the backend.  PR 57087.  [Yann Ylavic]

  *) mod_proxy: Put mod_proxy_{connect,wstunnel} tunneling code in common in
     proxy_util.  [Yann Ylavic]

  *) mod_proxy: Improve tunneling loop to support half closed connections and
     pending data draining (for protocols like rsync). PR 61616. [Yann Ylavic]

  *) mod_proxy_http: handle Upgrade request, 101 (Switching Protocol) response
     and switched protocol forwarding.  [Yann Ylavic]

  *) mod_proxy_wstunnel: Leave Upgrade requests handling to mod_proxy_http,
     allowing for (non-)Upgrade negotiation with the origin server.
     [Yann Ylavic]

  *) mod_proxy: Allow ProxyErrorOverride to be restricted to specific status
     codes.  PR63628. [Martin Drößler <mail martindroessler.de>]

  *) core: Add ReadBufferSize, FlushMaxThreshold and FlushMaxPipelined
     directives.  [Yann Ylavic]

  *) core: Ensure that aborted connections are logged as such. PR 62823
     [Arnaud Grandville <contact@grandville.net>]

  *) http: Allow unknown response status' lines returned in the form of
     "HTTP/x.x xxx Status xxx".  [Yann Ylavic]

  *) mod_proxy_http: Fix 100-continue deadlock for spooled request bodies,
     leading to Request Timeout (408).  PR 63855.  [Yann Ylavic]

  *) core: Remove headers on 304 Not Modified as specified by RFC7234, as
     opposed to passing an explicit subset of headers. PR 61820.
     [Giovanni Bechis]

  *) mpm_event: Don't reset connections after lingering close, restoring prior
     to 2.4.28 behaviour.  [Yann Ylavic]

  *) mpm_event: Kill connections in keepalive state only when there is no more
     workers available, not when the maximum number of connections is reached,
     restoring prior to 2.4.30 behaviour.  [Yann Ylavic]

  *) mod_unique_id: Use base64url encoding for UNIQUE_ID variable,
     avoiding the use of '@'.  PR 57044.
     [Michael Kaufmann <apache-bugzilla michael-kaufmann.ch>]

  *) mod_rewrite: Extend the [CO] (cookie) flag of RewriteRule to accept a
     SameSite attribute. [Eric Covener]

  *) mod_proxy: Add proxy check_trans hook.  This allows proxy
     modules to decline request handling at early stage.

  *) mod_proxy_wstunnel: Decline requests without an Upgrade
     header so ws/wss can be enabled overlapping with later
     http/https.

  *) mod_http2: Log requests and sent the configured error response in case of
     early detected errors like too many or too long headers.
     [Ruediger Pluem, Stefan Eissing]

  *) mod_md: Lowered the required minimal libcurl version from 7.50 to 7.29
     as proposed by <alexander.gerasimov codeit.pro>. [Stefan Eissing]

  *) mod_ssl: Fix request body buffering with PHA in TLSv1.3.  [Joe Orton]

  *) mod_proxy_uwsgi: Fix a crash when sending environment variables with no
     value. PR 64598 [Ruediger Pluem]

  *) mod_proxy: Recognize parameters from ProxyPassMatch workers with dollar
     substitution, such that they apply to the backend connection.  Note that
     connection reuse is disabled by default to avoid compatibility issues.
     [Takashi Sato, Jan Kaluza, Eric Covener, Yann Ylavic, Jean-Frederic Clere]

Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
nono303 pushed a commit to nono303/win-build-scripts that referenced this issue May 6, 2021
  *) mod_dav_fs: Improve logging output when failing to open files for
     writing.  PR 64413.  [Bingyu Shen < ahshenbingyu gmail.com >]

  *) mod_http2: Fixed a race condition that could lead to streams being
     aborted (RST to the client), although a response had been produced.
     [Stefan Eissing]

  *) mod_lua: Add support to Lua 5.4  [Joe Orton, Giovanni Bechis, Ruediger Pluem]

  *) MPM event/worker: Fix possible crash in child process on early signal
     delivery.  PR 64533.  [Ruediger Pluem]

  *) mod_http2: sync with github standalone version 1.15.17
     - Log requests and sent the configured error response in case of early detected
       errors like too many or too long headers. [Ruediger Pluem]
     - new option 'H2OutputBuffering on/off' which controls the buffering of stream output.
       The default is on, which is the behaviour of older mod-h2 versions. When off, all
       bytes are made available immediately to the main connection for sending them
       out to the client. This fixes interop issues with certain flavours of gRPC, see
       also < icing/mod_h2#207 >.
       [Stefan Eissing]

  *) mod_unique_id: Fix potential duplicated ID generation under heavy load.
     PR 65159
     [Jonas Müntener < jonas.muentener ergon.ch >, Christophe Jaillet]

  *) "[mod_dav_fs etag handling] should really honor the FileETag setting".
     - It now does.
     - Add "Digest" to FileETag directive, allowing a strong ETag to be
       generated using a file digest.
     - Add ap_make_etag_ex() and ap_set_etag_fd() to allow full control over
       ETag generation.
     - Add concept of "binary notes" to request_rec, allowing packed bit flags
       to be added to a request.
     - First binary note - AP_REQUEST_STRONG_ETAG - allows modules to force
       the ETag to a strong ETag to comply with RFC requirements, such as those
       mandated by various WebDAV extensions.
     [Graham Leggett]

  *) mod_proxy_http: Fix a possibly crash when the origin connection gets
     interrupted before completion.  PR 64234.
     [Barnim Dzwillo < dzwillo strato.de >, Ruediger Pluem]

  *) mod_ssl: Do not keep connections to OCSP responders alive when doing
     OCSP requests.  PR 64135.  [Ruediger Pluem]

  *) mod_ssl: Improve the coalescing filter to buffer into larger TLS
     records, and avoid revealing the HTTP header size via TLS record
     boundaries (for common response generators).
     [Joe Orton, Ruediger Pluem]
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Jun 4, 2021
Changes with Apache 2.4.48

  *) mod_proxy_wstunnel: Add ProxyWebsocketFallbackToProxyHttp to opt-out the
     fallback to mod_proxy_http for WebSocket upgrade and tunneling.
     [Yann Ylavic]

  *) mod_proxy: Fix flushing of THRESHOLD_MIN_WRITE data while tunneling.
     BZ 65294.  [Yann Ylavic]

  *) core: Fix a regression that stripped the ETag header from 304 responses.
     PR 61820 [Ruediger Pluem, Roy T. Fielding]

  *) core: Adding SSL related inquiry functions to the server API.
     These function are always available, even when no module providing
     SSL is loaded. They provide their own "shadowing" implementation for
     the optional functions of similar name that mod_ssl and impersonators
     of mod_ssl provide.
     This enables loading of several SSL providing modules when all but
     one of them registers itself into the new hooks. Two old-style SSL
     modules will not work, as they replace the others optional functions
     with their own.
     Modules using the old-style optional functions will continue to work
     as core supplies its own versions of those.
     The following has been added so far:
     - ap_ssl_conn_is_ssl() to query if a connection is using SSL.
     - ap_ssl_var_lookup() to query SSL related variables for a
       server/connection/request.
     - Hooks for 'ssl_conn_is_ssl' and 'ssl_var_lookup' where modules
       providing SSL can install their own value supplying functions.
     - ap_ssl_add_cert_files() to enable other modules like mod_md to provide
       certificate and keys for an SSL module like mod_ssl.
     - ap_ssl_add_fallback_cert_files() to enable other modules like mod_md to
       provide a fallback certificate in case no 'proper' certificate is
       available for an SSL module like mod_ssl.
     - ap_ssl_answer_challenge() to enable other modules like mod_md to
       provide a certificate as used in the RFC 8555 'tls-alpn-01' challenge
       for the ACME protocol for an SSL module like mod_ssl. The function
       and its hook provide PEM encoded data instead of file names.
     - Hooks for 'ssl_add_cert_files', 'ssl_add_fallback_cert_files' and
       'ssl_answer_challenge' where modules like mod_md can provide providers
       to the above mentioned functions.
     - These functions reside in the new 'http_ssl.h' header file.
     [Stefan Eissing]

  *) core/mod_ssl/mod_md: adding OCSP response provisioning as core feature. This
     allows modules to access and provide OCSP response data without being tied
     of each other. The data is exchanged in standard, portable formats (PEM encoded
     certificates and DER encoded responses), so that the actual SSL/crypto
     implementations used by the modules are independant of each other.
     Registration and retrieval happen in the context of a server (server_rec)
     which modules may use to decide if they are configured for this or not.
     The area of changes:
     1. core: defines 2 functions in include/http_ssl.h, so that modules may
        register a certificate, together with its issuer certificate for OCSP
        response provisioning and ask for current response data (DER bytes) later.
        Also, 2 hooks are defined that allow modules to implement this OCSP
        provisioning.
     2. mod_ssl uses the new functions, in addition to what it did already, to
        register its certificates this way. If no one is interested in providing
        OCSP, it falls back to its own (if configured) stapling implementation.
     3. mod_md registers itself at the core hooks for OCSP provisioning. Depending
        on configuration, it will accept registrations of its own certificates only,
        all certificates or none.
     [Stefan Eissing]

 *) mod_md: v2.4.0 with improvements and bugfixes
     - MDPrivateKeys allows the specification of several types. Beside "RSA" plus
     optional key lengths elliptic curves can be configured. This means you can
     have multiple certificates for a Managed Domain with different key types.
     With ```MDPrivateKeys secp384r1 rsa2048``` you get one ECDSA  and one RSA
     certificate and all modern client will use the shorter ECDSA, while older
     client will get the RSA certificate.
     Many thanks to @tlhackque who pushed and helped on this.
     - Support added for MDomains consisting of a wildcard. Configuring
     ```MDomain *.host.net``` will match all virtual hosts matching that pattern
     and obtain one certificate for it (assuming you have 'dns-01' challenge
     support configured). Addresses #239.
     - Removed support for ACMEv1 servers. The only known installation used to
     be Let's Encrypt which has disabled that version more than a year ago for
     new accounts.
     - Andreas Ulm (<https://github.com/root360-AndreasUlm>) implemented the
     ```renewing``` call to ```MDMessageCmd``` that can deny a certificate
     renewal attempt. This is useful in clustered installations, as
     discussed in #233).
     - New event ```challenge-setup:<type>:<domain>```, triggered when the
     challenge data for a domain has been created. This is invoked before the
     ACME server is told to check for it. The type is one of the ACME challenge
     types. This is invoked for every DNS name in a MDomain.
     - The max delay for retries has been raised to daily (this is like all
     retries jittered somewhat to avoid repeats at fixed time of day).
     - Certain error codes reported by the ACME server that indicate a problem
     with the configured data now immediately switch to daily retries. For
     example: if the ACME server rejects a contact email or a domain name,
     frequent retries will most likely not solve the problem. But daily retries
     still make sense as there might be an error at the server and un-supervised
     certificate renewal is the goal. Refs #222.
     - Test case and work around for domain names > 64 octets. Fixes #227.
     When the first DNS name of an MD is longer than 63 octets, the certificate
     request will not contain a CN field, but leave it up to the CA to choose one.
     Currently, Lets Encrypt looks for a shorter name in the SAN list given and
     fails the request if none is found. But it is really up to the CA (and what
     browsers/libs accept here) and may change over the years. That is why
     the decision is best made at the CA.
     - Retry delays now have a random +/-[0-50]% modification applied to let
     retries from several servers spread out more, should they have been
     restarted at the same time of day.
     - Fixed several places where the 'badNonce' return code from an ACME server
     was not handled correctly. The test server 'pebble' simulates this behaviour
     by default and helps nicely in verifying this behaviour. Thanks, pebble!
     - Set the default `MDActivationDelay` to 0. This was confusing to users that
     new certificates were deemed not usably before a day of delay. When clocks are
     correct, using a new certificate right away should not pose a problem.
     - When handling ACME authorization resources, the module no longer requires
     the server to return a "Location" header, as was necessary in ACMEv1.
     Fixes #216.
     - Fixed a theoretical uninitialized read when testing for JSON error responses
     from the ACME CA. Reported at <https://bz.apache.org/bugzilla/show_bug.cgi?id=64297>.
     - ACME problem reports from CAs that include parameters in the Content-Type
     header are handled correctly. (Previously, the problem text would not be
     reported and retries could exceed CA limits.)
     - Account Update transactions to V2 CAs now use the correct POST-AS-GET method.
     Previously, an empty JSON object was sent - which apparently LE accepted,
     but others reject.
     [Stefan Eissing, @tlhackque, Andreas Ulm]

Changes with Apache 2.4.47

  *) mod_dav_fs: Improve logging output when failing to open files for
     writing.  PR 64413.  [Bingyu Shen <ahshenbingyu gmail.com>]

  *) mod_http2: Fixed a race condition that could lead to streams being
     aborted (RST to the client), although a response had been produced.
     [Stefan Eissing]

  *) mod_lua: Add support to Lua 5.4  [Joe Orton, Giovanni Bechis, Ruediger Pluem]

  *) MPM event/worker: Fix possible crash in child process on early signal
     delivery.  PR 64533.  [Ruediger Pluem]

  *) mod_http2: sync with github standalone version 1.15.17
     - Log requests and sent the configured error response in case of early detected
       errors like too many or too long headers. [Ruediger Pluem]
     - new option 'H2OutputBuffering on/off' which controls the buffering of stream output.
       The default is on, which is the behaviour of older mod-h2 versions. When off, all
       bytes are made available immediately to the main connection for sending them
       out to the client. This fixes interop issues with certain flavours of gRPC, see
       also <icing/mod_h2#207>.
       [Stefan Eissing]

  *) mod_unique_id: Fix potential duplicated ID generation under heavy load.
     PR 65159
     [Jonas Müntener <jonas.muentener ergon.ch>, Christophe Jaillet]

  *) "[mod_dav_fs etag handling] should really honor the FileETag setting".
     - It now does.
     - Add "Digest" to FileETag directive, allowing a strong ETag to be
       generated using a file digest.
     - Add ap_make_etag_ex() and ap_set_etag_fd() to allow full control over
       ETag generation.
     - Add concept of "binary notes" to request_rec, allowing packed bit flags
       to be added to a request.
     - First binary note - AP_REQUEST_STRONG_ETAG - allows modules to force
       the ETag to a strong ETag to comply with RFC requirements, such as those
       mandated by various WebDAV extensions.
     [Graham Leggett]

  *) mod_proxy_http: Fix a possibly crash when the origin connection gets
     interrupted before completion.  PR 64234.
     [Barnim Dzwillo <dzwillo strato.de>, Ruediger Pluem]

  *) mod_ssl: Do not keep connections to OCSP responders alive when doing
     OCSP requests.  PR 64135.  [Ruediger Pluem]

  *) mod_ssl: Improve the coalescing filter to buffer into larger TLS
     records, and avoid revealing the HTTP header size via TLS record
     boundaries (for common response generators).
     [Joe Orton, Ruediger Pluem]

  *) mod_proxy_hcheck: Don't pile up health checks if the previous one did
     not finish before hcinterval.  PR 63010.  [Yann Ylavic]

  *) mod_session: Improve session parsing.  [Yann Yalvic]

  *) mod_authnz_ldap: Prevent authentications with empty passwords for the
     initial bind to fail with status 500. [Ruediger Pluem]

  *) mod_auth_digest: Fast validation of the nonce's base64 to fail early if
     the format can't match anyway.  [Yann Ylavic]

  *) mod_proxy_fcgi: Honor "SetEnv proxy-sendcl" to forward a chunked
     Transfer-Encoding from the client, spooling the request body when needed
     to provide a Content-Length to the backend.  PR 57087.  [Yann Ylavic]

  *) mod_proxy: Put mod_proxy_{connect,wstunnel} tunneling code in common in
     proxy_util.  [Yann Ylavic]

  *) mod_proxy: Improve tunneling loop to support half closed connections and
     pending data draining (for protocols like rsync). PR 61616. [Yann Ylavic]

  *) mod_proxy_http: handle Upgrade request, 101 (Switching Protocol) response
     and switched protocol forwarding.  [Yann Ylavic]

  *) mod_proxy_wstunnel: Leave Upgrade requests handling to mod_proxy_http,
     allowing for (non-)Upgrade negotiation with the origin server.
     [Yann Ylavic]

  *) mod_proxy: Allow ProxyErrorOverride to be restricted to specific status
     codes.  PR63628. [Martin Drößler <mail martindroessler.de>]

  *) core: Add ReadBufferSize, FlushMaxThreshold and FlushMaxPipelined
     directives.  [Yann Ylavic]

  *) core: Ensure that aborted connections are logged as such. PR 62823
     [Arnaud Grandville <contact@grandville.net>]

  *) http: Allow unknown response status' lines returned in the form of
     "HTTP/x.x xxx Status xxx".  [Yann Ylavic]

  *) mod_proxy_http: Fix 100-continue deadlock for spooled request bodies,
     leading to Request Timeout (408).  PR 63855.  [Yann Ylavic]

  *) core: Remove headers on 304 Not Modified as specified by RFC7234, as
     opposed to passing an explicit subset of headers. PR 61820.
     [Giovanni Bechis]

  *) mpm_event: Don't reset connections after lingering close, restoring prior
     to 2.4.28 behaviour.  [Yann Ylavic]

  *) mpm_event: Kill connections in keepalive state only when there is no more
     workers available, not when the maximum number of connections is reached,
     restoring prior to 2.4.30 behaviour.  [Yann Ylavic]

  *) mod_unique_id: Use base64url encoding for UNIQUE_ID variable,
     avoiding the use of '@'.  PR 57044.
     [Michael Kaufmann <apache-bugzilla michael-kaufmann.ch>]

  *) mod_rewrite: Extend the [CO] (cookie) flag of RewriteRule to accept a
     SameSite attribute. [Eric Covener]

  *) mod_proxy: Add proxy check_trans hook.  This allows proxy
     modules to decline request handling at early stage.

  *) mod_proxy_wstunnel: Decline requests without an Upgrade
     header so ws/wss can be enabled overlapping with later
     http/https.

  *) mod_http2: Log requests and sent the configured error response in case of
     early detected errors like too many or too long headers.
     [Ruediger Pluem, Stefan Eissing]

  *) mod_md: Lowered the required minimal libcurl version from 7.50 to 7.29
     as proposed by <alexander.gerasimov codeit.pro>. [Stefan Eissing]

  *) mod_ssl: Fix request body buffering with PHA in TLSv1.3.  [Joe Orton]

  *) mod_proxy_uwsgi: Fix a crash when sending environment variables with no
     value. PR 64598 [Ruediger Pluem]

  *) mod_proxy: Recognize parameters from ProxyPassMatch workers with dollar
     substitution, such that they apply to the backend connection.  Note that
     connection reuse is disabled by default to avoid compatibility issues.
     [Takashi Sato, Jan Kaluza, Eric Covener, Yann Ylavic, Jean-Frederic Clere]
@icing
Copy link
Owner

icing commented Jul 10, 2021

Unable to reproduce.

@icing icing closed this as completed Jul 10, 2021
jperkin pushed a commit to TritonDataCenter/pkgsrc that referenced this issue Nov 28, 2021
Changes with Apache 2.4.48

  *) mod_proxy_wstunnel: Add ProxyWebsocketFallbackToProxyHttp to opt-out the
     fallback to mod_proxy_http for WebSocket upgrade and tunneling.
     [Yann Ylavic]

  *) mod_proxy: Fix flushing of THRESHOLD_MIN_WRITE data while tunneling.
     BZ 65294.  [Yann Ylavic]

  *) core: Fix a regression that stripped the ETag header from 304 responses.
     PR 61820 [Ruediger Pluem, Roy T. Fielding]

  *) core: Adding SSL related inquiry functions to the server API.
     These function are always available, even when no module providing
     SSL is loaded. They provide their own "shadowing" implementation for
     the optional functions of similar name that mod_ssl and impersonators
     of mod_ssl provide.
     This enables loading of several SSL providing modules when all but
     one of them registers itself into the new hooks. Two old-style SSL
     modules will not work, as they replace the others optional functions
     with their own.
     Modules using the old-style optional functions will continue to work
     as core supplies its own versions of those.
     The following has been added so far:
     - ap_ssl_conn_is_ssl() to query if a connection is using SSL.
     - ap_ssl_var_lookup() to query SSL related variables for a
       server/connection/request.
     - Hooks for 'ssl_conn_is_ssl' and 'ssl_var_lookup' where modules
       providing SSL can install their own value supplying functions.
     - ap_ssl_add_cert_files() to enable other modules like mod_md to provide
       certificate and keys for an SSL module like mod_ssl.
     - ap_ssl_add_fallback_cert_files() to enable other modules like mod_md to
       provide a fallback certificate in case no 'proper' certificate is
       available for an SSL module like mod_ssl.
     - ap_ssl_answer_challenge() to enable other modules like mod_md to
       provide a certificate as used in the RFC 8555 'tls-alpn-01' challenge
       for the ACME protocol for an SSL module like mod_ssl. The function
       and its hook provide PEM encoded data instead of file names.
     - Hooks for 'ssl_add_cert_files', 'ssl_add_fallback_cert_files' and
       'ssl_answer_challenge' where modules like mod_md can provide providers
       to the above mentioned functions.
     - These functions reside in the new 'http_ssl.h' header file.
     [Stefan Eissing]

  *) core/mod_ssl/mod_md: adding OCSP response provisioning as core feature. This
     allows modules to access and provide OCSP response data without being tied
     of each other. The data is exchanged in standard, portable formats (PEM encoded
     certificates and DER encoded responses), so that the actual SSL/crypto
     implementations used by the modules are independant of each other.
     Registration and retrieval happen in the context of a server (server_rec)
     which modules may use to decide if they are configured for this or not.
     The area of changes:
     1. core: defines 2 functions in include/http_ssl.h, so that modules may
        register a certificate, together with its issuer certificate for OCSP
        response provisioning and ask for current response data (DER bytes) later.
        Also, 2 hooks are defined that allow modules to implement this OCSP
        provisioning.
     2. mod_ssl uses the new functions, in addition to what it did already, to
        register its certificates this way. If no one is interested in providing
        OCSP, it falls back to its own (if configured) stapling implementation.
     3. mod_md registers itself at the core hooks for OCSP provisioning. Depending
        on configuration, it will accept registrations of its own certificates only,
        all certificates or none.
     [Stefan Eissing]

 *) mod_md: v2.4.0 with improvements and bugfixes
     - MDPrivateKeys allows the specification of several types. Beside "RSA" plus
     optional key lengths elliptic curves can be configured. This means you can
     have multiple certificates for a Managed Domain with different key types.
     With ```MDPrivateKeys secp384r1 rsa2048``` you get one ECDSA  and one RSA
     certificate and all modern client will use the shorter ECDSA, while older
     client will get the RSA certificate.
     Many thanks to @tlhackque who pushed and helped on this.
     - Support added for MDomains consisting of a wildcard. Configuring
     ```MDomain *.host.net``` will match all virtual hosts matching that pattern
     and obtain one certificate for it (assuming you have 'dns-01' challenge
     support configured). Addresses #239.
     - Removed support for ACMEv1 servers. The only known installation used to
     be Let's Encrypt which has disabled that version more than a year ago for
     new accounts.
     - Andreas Ulm (<https://github.com/root360-AndreasUlm>) implemented the
     ```renewing``` call to ```MDMessageCmd``` that can deny a certificate
     renewal attempt. This is useful in clustered installations, as
     discussed in #233).
     - New event ```challenge-setup:<type>:<domain>```, triggered when the
     challenge data for a domain has been created. This is invoked before the
     ACME server is told to check for it. The type is one of the ACME challenge
     types. This is invoked for every DNS name in a MDomain.
     - The max delay for retries has been raised to daily (this is like all
     retries jittered somewhat to avoid repeats at fixed time of day).
     - Certain error codes reported by the ACME server that indicate a problem
     with the configured data now immediately switch to daily retries. For
     example: if the ACME server rejects a contact email or a domain name,
     frequent retries will most likely not solve the problem. But daily retries
     still make sense as there might be an error at the server and un-supervised
     certificate renewal is the goal. Refs #222.
     - Test case and work around for domain names > 64 octets. Fixes #227.
     When the first DNS name of an MD is longer than 63 octets, the certificate
     request will not contain a CN field, but leave it up to the CA to choose one.
     Currently, Lets Encrypt looks for a shorter name in the SAN list given and
     fails the request if none is found. But it is really up to the CA (and what
     browsers/libs accept here) and may change over the years. That is why
     the decision is best made at the CA.
     - Retry delays now have a random +/-[0-50]% modification applied to let
     retries from several servers spread out more, should they have been
     restarted at the same time of day.
     - Fixed several places where the 'badNonce' return code from an ACME server
     was not handled correctly. The test server 'pebble' simulates this behaviour
     by default and helps nicely in verifying this behaviour. Thanks, pebble!
     - Set the default `MDActivationDelay` to 0. This was confusing to users that
     new certificates were deemed not usably before a day of delay. When clocks are
     correct, using a new certificate right away should not pose a problem.
     - When handling ACME authorization resources, the module no longer requires
     the server to return a "Location" header, as was necessary in ACMEv1.
     Fixes #216.
     - Fixed a theoretical uninitialized read when testing for JSON error responses
     from the ACME CA. Reported at <https://bz.apache.org/bugzilla/show_bug.cgi?id=64297>.
     - ACME problem reports from CAs that include parameters in the Content-Type
     header are handled correctly. (Previously, the problem text would not be
     reported and retries could exceed CA limits.)
     - Account Update transactions to V2 CAs now use the correct POST-AS-GET method.
     Previously, an empty JSON object was sent - which apparently LE accepted,
     but others reject.
     [Stefan Eissing, @tlhackque, Andreas Ulm]

Changes with Apache 2.4.47

  *) mod_dav_fs: Improve logging output when failing to open files for
     writing.  PR 64413.  [Bingyu Shen <ahshenbingyu gmail.com>]

  *) mod_http2: Fixed a race condition that could lead to streams being
     aborted (RST to the client), although a response had been produced.
     [Stefan Eissing]

  *) mod_lua: Add support to Lua 5.4  [Joe Orton, Giovanni Bechis, Ruediger Pluem]

  *) MPM event/worker: Fix possible crash in child process on early signal
     delivery.  PR 64533.  [Ruediger Pluem]

  *) mod_http2: sync with github standalone version 1.15.17
     - Log requests and sent the configured error response in case of early detected
       errors like too many or too long headers. [Ruediger Pluem]
     - new option 'H2OutputBuffering on/off' which controls the buffering of stream output.
       The default is on, which is the behaviour of older mod-h2 versions. When off, all
       bytes are made available immediately to the main connection for sending them
       out to the client. This fixes interop issues with certain flavours of gRPC, see
       also <icing/mod_h2#207>.
       [Stefan Eissing]

  *) mod_unique_id: Fix potential duplicated ID generation under heavy load.
     PR 65159
     [Jonas Müntener <jonas.muentener ergon.ch>, Christophe Jaillet]

  *) "[mod_dav_fs etag handling] should really honor the FileETag setting".
     - It now does.
     - Add "Digest" to FileETag directive, allowing a strong ETag to be
       generated using a file digest.
     - Add ap_make_etag_ex() and ap_set_etag_fd() to allow full control over
       ETag generation.
     - Add concept of "binary notes" to request_rec, allowing packed bit flags
       to be added to a request.
     - First binary note - AP_REQUEST_STRONG_ETAG - allows modules to force
       the ETag to a strong ETag to comply with RFC requirements, such as those
       mandated by various WebDAV extensions.
     [Graham Leggett]

  *) mod_proxy_http: Fix a possibly crash when the origin connection gets
     interrupted before completion.  PR 64234.
     [Barnim Dzwillo <dzwillo strato.de>, Ruediger Pluem]

  *) mod_ssl: Do not keep connections to OCSP responders alive when doing
     OCSP requests.  PR 64135.  [Ruediger Pluem]

  *) mod_ssl: Improve the coalescing filter to buffer into larger TLS
     records, and avoid revealing the HTTP header size via TLS record
     boundaries (for common response generators).
     [Joe Orton, Ruediger Pluem]

  *) mod_proxy_hcheck: Don't pile up health checks if the previous one did
     not finish before hcinterval.  PR 63010.  [Yann Ylavic]

  *) mod_session: Improve session parsing.  [Yann Yalvic]

  *) mod_authnz_ldap: Prevent authentications with empty passwords for the
     initial bind to fail with status 500. [Ruediger Pluem]

  *) mod_auth_digest: Fast validation of the nonce's base64 to fail early if
     the format can't match anyway.  [Yann Ylavic]

  *) mod_proxy_fcgi: Honor "SetEnv proxy-sendcl" to forward a chunked
     Transfer-Encoding from the client, spooling the request body when needed
     to provide a Content-Length to the backend.  PR 57087.  [Yann Ylavic]

  *) mod_proxy: Put mod_proxy_{connect,wstunnel} tunneling code in common in
     proxy_util.  [Yann Ylavic]

  *) mod_proxy: Improve tunneling loop to support half closed connections and
     pending data draining (for protocols like rsync). PR 61616. [Yann Ylavic]

  *) mod_proxy_http: handle Upgrade request, 101 (Switching Protocol) response
     and switched protocol forwarding.  [Yann Ylavic]

  *) mod_proxy_wstunnel: Leave Upgrade requests handling to mod_proxy_http,
     allowing for (non-)Upgrade negotiation with the origin server.
     [Yann Ylavic]

  *) mod_proxy: Allow ProxyErrorOverride to be restricted to specific status
     codes.  PR63628. [Martin Drößler <mail martindroessler.de>]

  *) core: Add ReadBufferSize, FlushMaxThreshold and FlushMaxPipelined
     directives.  [Yann Ylavic]

  *) core: Ensure that aborted connections are logged as such. PR 62823
     [Arnaud Grandville <contact@grandville.net>]

  *) http: Allow unknown response status' lines returned in the form of
     "HTTP/x.x xxx Status xxx".  [Yann Ylavic]

  *) mod_proxy_http: Fix 100-continue deadlock for spooled request bodies,
     leading to Request Timeout (408).  PR 63855.  [Yann Ylavic]

  *) core: Remove headers on 304 Not Modified as specified by RFC7234, as
     opposed to passing an explicit subset of headers. PR 61820.
     [Giovanni Bechis]

  *) mpm_event: Don't reset connections after lingering close, restoring prior
     to 2.4.28 behaviour.  [Yann Ylavic]

  *) mpm_event: Kill connections in keepalive state only when there is no more
     workers available, not when the maximum number of connections is reached,
     restoring prior to 2.4.30 behaviour.  [Yann Ylavic]

  *) mod_unique_id: Use base64url encoding for UNIQUE_ID variable,
     avoiding the use of '@'.  PR 57044.
     [Michael Kaufmann <apache-bugzilla michael-kaufmann.ch>]

  *) mod_rewrite: Extend the [CO] (cookie) flag of RewriteRule to accept a
     SameSite attribute. [Eric Covener]

  *) mod_proxy: Add proxy check_trans hook.  This allows proxy
     modules to decline request handling at early stage.

  *) mod_proxy_wstunnel: Decline requests without an Upgrade
     header so ws/wss can be enabled overlapping with later
     http/https.

  *) mod_http2: Log requests and sent the configured error response in case of
     early detected errors like too many or too long headers.
     [Ruediger Pluem, Stefan Eissing]

  *) mod_md: Lowered the required minimal libcurl version from 7.50 to 7.29
     as proposed by <alexander.gerasimov codeit.pro>. [Stefan Eissing]

  *) mod_ssl: Fix request body buffering with PHA in TLSv1.3.  [Joe Orton]

  *) mod_proxy_uwsgi: Fix a crash when sending environment variables with no
     value. PR 64598 [Ruediger Pluem]

  *) mod_proxy: Recognize parameters from ProxyPassMatch workers with dollar
     substitution, such that they apply to the backend connection.  Note that
     connection reuse is disabled by default to avoid compatibility issues.
     [Takashi Sato, Jan Kaluza, Eric Covener, Yann Ylavic, Jean-Frederic Clere]
@a4chet
Copy link

a4chet commented Jan 14, 2022

Tried this in 2.4.52 on Windows and it does work with unary & server side streaming, partial config below...

# Main host
<VirtualHost _default_:443>
    ServerAdmin webmaster@dummy-host.example.com
    DocumentRoot "C:/Apache24/htdocs"
    ServerName localhost
    ServerAlias apache-httpd
    	
    Protocols h2 http/1.1

    SSLEngine on    
    SSLCertificateFile "C:/Apache24/certs/..."
    SSLCertificateKeyFile "C:/Apache24/certs/..."
    LogLevel proxy:info
    
    ############### GRPC Proxying #####################
    SSLProxyEngine on
    SSLProxyMachineCertificateFile "C:/Apache24/certs/..."
    SSLProxyVerify none
    SSLProxyCheckPeerCN off
    SSLProxyCheckPeerName off
    SSLProxyCheckPeerExpire off
    H2OutputBuffering off
    Timeout 600

    <Location "/svc">
      LogLevel http2:info
      LogLevel proxy_http2:info
      
      ProxyPass h2://<backend>
      ProxyPassReverse https://<backend>
    </Location>
    ############### END GRPC Proxying #####################
    ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants