Port to librustls 0.12.0

[kpcyrd]

Resolves #12737, relates to rustls/rustls-ffi#384.

I've tried to port this as literal as possible, then slightly restructured it since both the ca_info_blob branch and the ssl_cafile branch started having a lot of duplicate code. I suspect the else if(ca_info_blob || ssl_cafile) { can possibly be replaced by else {.

I'm also somewhat certain I've overlook some things for correctly *_free-ing everything.

This is part of the effort of integrating librustls into Arch Linux and Gentoo: rustls/rustls-ffi#274.

[bagder]

Thanks!

[gvanem]

From clang-cl, there is still these 2 warnings:

vtls/rustls.c(295,53): warning: format specifies type 'long' but the argument has type 'size_t' (aka 'unsigned long long') [-Wformat]
  295 |   CURL_TRC_CF(data, cf, "cf_send: %ld plain bytes", plainlen);
      |                                   ~~~               ^~~~~~~~
      |                                   %zu
./curl_trc.h(79,38): note: expanded from macro 'CURL_TRC_CF'
   79 |          Curl_trc_cf_infof(data, cf, __VA_ARGS__); } while(0)
      |                                      ^~~~~~~~~~~
vtls/rustls.c(411,23): warning: incompatible function pointer types passing 'enum rustls_result (void *, const rustls_verify_server_cert_params *)' (aka
      'enum rustls_result (void *, const struct rustls_verify_server_cert_params *)') to parameter of type 'rustls_verify_server_cert_callback' (aka
      'unsigned int (*)(void *, const struct rustls_verify_server_cert_params *)') [-Wincompatible-function-pointer-types-strict]
  411 |       config_builder, cr_verify_none);
      |                       ^~~~~~~~~~~~~~
f:/MinGW32/src/inet/Crypto/RusTls/src\rustls.h(1272,114): note: passing argument to parameter 'callback' here
 1272 |                                                                               rustls_verify_server_cert_callback callback);
      |                                                                                                                  ^
2 warnings generated.

which I fixed by:

--- a/vtls/rustls.c 2024-02-26 11:45:13
+++ b/vtls/rustls.c 2024-02-26 11:58:58
@@ -292,7 +292,7 @@
   DEBUGASSERT(backend);
   rconn = backend->conn;

-  CURL_TRC_CF(data, cf, "cf_send: %ld plain bytes", plainlen);
+  CURL_TRC_CF(data, cf, "cf_send: %zu plain bytes", plainlen);

   io_ctx.cf = cf;
   io_ctx.data = data;
@@ -343,7 +343,7 @@

 /* A server certificate verify callback for rustls that always returns
    RUSTLS_RESULT_OK, or in other words disable certificate verification. */
-static enum rustls_result
+static uint32_t
 cr_verify_none(void *userdata UNUSED_PARAM,
                const rustls_verify_server_cert_params *params UNUSED_PARAM)
 {

[kpcyrd]

@gvanem thanks for pointing this out, do you want to submit the patch yourself or should I take care of this?

[gvanem]

@kpcyrd Please go ahead.