add sni support (IDFGH-8363) #9833

axos88 · 2022-09-22T13:30:15Z

This PR adds full server side SNI support.

Server side SNI is only supported by mbedTLS.
A new KConfig value has been introduced, it depends on mbedTLS being selected
the sni callback and the supplied data can be used to select a certificate based on the incoming SNI header containing the host the client is looking for.
pkey, cert, client verification ca bundle, authmode can also be selected: https://mbed-tls.readthedocs.io/en/latest/kb/how-to/use-sni/#server-side
a TLS server can now be initialized WITHOUT a pkey and a certificate as long as SNI is enabled, and an SNI callback is set up. A warning will be issued that handshakes will fail unless the SNI callback sets up the pkey and cert to use for each handshake.
Using this mechanism the server certificate and private key can be changed without restarting the server, allowing the certificate to be updated without reboots (certificate renewal for example)

CLAassistant · 2022-09-22T13:30:21Z

All committers have signed the CLA.

KaeLL · 2022-09-22T13:56:11Z

I believe PRs are supposed to be made against the master branch.

axos88 · 2022-09-24T19:31:57Z

For some odd reason I cannot get current master to compile, so I can't port this to the current master, and there are merge conflicts, so I can't confidently resolve them.

AdityaHPatwardhan · 2022-09-26T10:20:09Z

@axos88 Thanks for adding the support of the feature in ESP-IDF.
But as mentioned by @KaeLL, We only accept pull requests against the master branch. Please raise a PR against the master branch.

Just to note here, We are using mbedtls-3.x, Please make sure you have followed, migration guide for using mbedtls-3.x on the master branch.

axos88 · 2022-09-26T12:26:03Z

@AdityaHPatwardhan, I am using esp-rs/esp-idf-sys to build esp-idf, but I get the following error on master (works on latest stable though):


running: "cmake" "/home/akos/projects/rust/esp-rs/esp-doorlock/target/riscv32imc-esp-espidf/release/build/esp-idf-sys-740a94ad1cf0d2c8/out" "-G" "Ninja" "-DCMAKE_TOOLCHAIN_FILE=/home/akos/projects/rust/esp-rs/esp-idf/tools/cmake/toolchain-esp32c3.cmake" "-DCMAKE_BUILD_TYPE=" "-DPYTHON=/home/akos/.espressif/python_env/idf5.1_py3.10_env/bin/python" "-DCMAKE_INSTALL_PREFIX=/home/akos/projects/rust/esp-rs/esp-doorlock/target/riscv32imc-esp-espidf/release/build/esp-idf-sys-740a94ad1cf0d2c8/out" "-DCMAKE_C_FLAGS= -march=rv32imc -ffunction-sections -fdata-sections -march=rv32imc -mabi=ilp32 -mcmodel=medany" "-DCMAKE_CXX_FLAGS= -march=rv32imc -ffunction-sections -fdata-sections -march=rv32imc -mabi=ilp32 -mcmodel=medany" "-DCMAKE_ASM_FLAGS=  -ffunction-sections -fdata-sections -march=rv32imc -mabi=ilp32 -mcmodel=medany"
  -- Checking Python dependencies...
  pkg_resources cannot be imported. The most common cause is a missing pip or setuptools package. If you've installed a custom Python then these packages are provided separately and have to be installed as well. Please refer to the Get Started section of the ESP-IDF Programming Guide for setting up the required packages.
  Skipping the download of /home/akos/projects/rust/esp-rs/esp-doorlock/.embuild/espressif/espidf.constraints.v5.1.txt because it was downloaded recently.
  Constraint file: /home/akos/projects/rust/esp-rs/esp-doorlock/.embuild/espressif/espidf.constraints.v5.1.txt
  Requirement files:
   - /home/akos/projects/rust/esp-rs/esp-idf/tools/requirements/requirements.core.txt
  Python being checked: /home/akos/projects/rust/esp-rs/esp-doorlock/.embuild/espressif/python_env/idf5.1_py3.10_env/bin/python
  -- Configuring incomplete, errors occurred!
  See also "/home/akos/projects/rust/esp-rs/esp-doorlock/target/riscv32imc-esp-espidf/release/build/esp-idf-sys-740a94ad1cf0d2c8/out/build/CMakeFiles/CMakeOutput.log".


 --- stderr
  Build configuration: BuildConfig {
      esp_idf_tools_install_dir: None,
      esp_idf_sdkconfig: None,
      esp_idf_sdkconfig_defaults: None,
      mcu: None,
      native: NativeConfig {
          esp_idf_version: Some(
              Tag(
                  "v5.1",
              ),
          ),
          esp_idf_repository: None,
          esp_idf_cmake_generator: None,
          idf_path: Some(
              "/home/akos/projects/rust/esp-rs/esp-idf",
          ),
          extra_components: [],
          esp_idf_components: None,
      },
      esp_idf_sys_root_crate: None,
  }
  Using activated esp-idf v5.1.0 environment at '/home/akos/projects/rust/esp-rs/esp-idf'
  CMake Error at /home/akos/projects/rust/esp-rs/esp-idf/tools/cmake/build.cmake:345 (message):
    Failed to run Python dependency check.  Python:
    /home/akos/.espressif/python_env/idf5.1_py3.10_env/bin/python, Error: 255
  Call Stack (most recent call first):
    /home/akos/projects/rust/esp-rs/esp-idf/tools/cmake/build.cmake:480 (__build_check_python)
    CMakeLists.txt:14 (idf_build_process)

axos88 · 2022-09-26T14:15:33Z

fixed build issue, rebased onto master.

AdityaHPatwardhan

@axos88 Thanks for updating the changes.

I have reviewed them in detail. I have left some comments and suggestions.
I also wanted to request you, if possible could you keep a separate commit for the fix that you have added? I think we may have to backport it.

components/esp-tls/esp_tls.c

components/esp-tls/esp_tls_mbedtls.c

AdityaHPatwardhan · 2022-09-29T08:17:25Z

components/esp-tls/esp_tls_mbedtls.c

+#if defined(CONFIG_ESP_TLS_SERVER_SNI_HOOK)
+    if (cfg->sni_callback != NULL) {
+        ESP_LOGI(TAG, "Initializing server side SNI callback");
+        mbedtls_ssl_conf_sni(&tls->conf, cfg->sni_callback, cfg->sni_callback_p_info);
+    }
+#endif


This part should come under set_server_config API since this part also does server configuration.
I think we can keep this conf_sni part in the else condition added above.

I'm not sure this is the case. The set_server_config is called when a new server side socket is created, while server_session_create is called each time a client connection is accepted. It is at that point that SNI has to be initialized, per incoming socket.

Adding it to set_server_config did not actually activate SNI as far as I remember - that's what I tried first.

@axos88 Sorry for the delayed reply, I wanted to test this first at my end before commenting.
I will do that now and give my comments in some time.
Do you have a MWE which I could use to test out this feature ?
Thanks!

Not currently, and I won't be able to provide you one until next week (I'm currently away from home).
My project is written in rust though with the esp-rs/esp-idf-sys wrapper, so I'm not sure it's of much help to you if you want to do the testing directly.

Hi @axos88 I was able to test the SNI feature locally.
I can confirm that with current approach if servercert and server key are not supplied and we configure the sni_callback using mbedtls_ssl_conf_sni in the ser_server_config function. Then the sni callback is set at once and is used for every connection formed by that specific context. There is no need to set the sni callback each time in the esp_mbedtls_server_session_create. I will update my comments based on that.

components/esp_https_server/src/https_server.c

components/esp-tls/esp_tls.h

components/esp_https_server/src/https_server.c

axos88 · 2022-10-06T09:33:38Z

@AdityaHPatwardhan let me know if you have outstanding concerns.

AdityaHPatwardhan · 2022-10-07T08:05:12Z

components/esp-tls/esp_tls_mbedtls.c

@@ -566,7 +566,16 @@ esp_err_t set_server_config(esp_tls_cfg_server_t *cfg, esp_tls_t *tls)
            return esp_ret;
        }
    } else {
+#if defined(CONFIG_ESP_TLS_SERVER_SNI_HOOK)
+        if (cfg->sni_callback == NULL) {


I think the SNI callback needs to be configured in this function ( but outside this if-else condition.)
I think only the check if sni_callback is present needs to be in the else part.
We should allow users to set the sni_callback along with providing server cert and keys. I think the sni callback would only be triggered when the ClientHello message would contain the ServerName extension. Otherwise, the normal certs would be used.

AdityaHPatwardhan · 2022-10-07T08:08:39Z

components/esp_https_server/src/https_server.c

+                free((void *) cfg->servercert_buf);
+                free((void *) cfg->cacert_buf);
+                free(cfg);
+                free(ssl_ctx);
+                return NULL;


I think this part is duplicated 3 times, can we have a goto: statement for this so that the code would jump to this in case of error.

AdityaHPatwardhan · 2022-10-07T08:09:34Z

@axos88 Sorry for the delayed review, Thanks again for your contribution for adding the sni feature in esp-tls.
I have left some small suggestions overall the MR looks good to me.
Thanks,
Aditya

axos88 · 2022-10-07T12:33:55Z

Thanks, I'll update the pr next week, when I'm back

…since it has access to more information, so the application can make a more informed decision on which certificate to serve (such as alpn value, server certificate type, etc.)

axos88 · 2022-10-10T15:31:31Z

Updated per your comments. Also discovered the certificate selection hook in mbedtls Mbed-TLS/mbedtls#5430, so moved to use that one instead of the SNI hook, since it has guaranteed access to all TLS extensions received in the ClientHello.

This means that the application can make a better decision on which certificate to send, based on the ALPN, or the server certificate type, or any other TLS extension received.

This callback does not take a void* unfortunately, so it is a bit more circumstancial to use IMHO, so I have added support for setting the user_data of the ssl_conf as well, so one can supply their inputs there.

axos88 · 2022-10-11T08:16:21Z

@AdityaHPatwardhan ^

AdityaHPatwardhan · 2022-10-12T07:31:01Z

@axos88 Thanks for updating the MR. Adding the option for cert_callback would indeed be helpful.
I think with this change we should make the option mutually exclusive to the option of providing private_key and ca_cert. AFAIK if we enable the cert callback then the verification using the cert and private key would be skipped entirely, so is there any point allocating memory for them?

Also I was also curious as to what the reply for Mbed-TLS/mbedtls#5454 (comment) would be from mbedTLS side, I think that would help me better understand the feature and its application here.

axos88 · 2022-10-12T17:42:15Z

No, if the cert_callback does not set the private key and the ca_cert, the ones provided will be used by default, so it does make sense to allocate memory if the user wants a default.

The cert_selection callback can be used for setitng other parameters, such as verification mode, client ca bundle, etc.

axos88 · 2022-10-12T17:43:43Z

Yeah I'm curious too, but as far as I have seen this is a strict superset. Just made that comment to have an official confirmation on this.

As far as https://mbed-tls.readthedocs.io/en/latest/kb/how-to/use-sni/ is concerned, the certificate_selection callback can call all of these functions described there, but has access to more information.

AdityaHPatwardhan · 2022-10-14T07:06:30Z

@axos88 Thank you very much for the PR. I will see to it that these changes are merged in our internal code-base, after that they should be available on GitHub in a few days.

axos88 · 2022-10-14T21:18:10Z

So do I understand correctly that GitHub is a mirror of some interior repository you use? Saw some strange comments in some PRS with shasums, that would explain them

axos88 · 2022-10-22T18:24:38Z

@AdityaHPatwardhan, do you have an estimate of how long until this can be merged?

AdityaHPatwardhan · 2022-10-27T19:24:44Z

Hi @axos88 The MR should be merged and should be available on GitHub by next week.

… access to required information so that the application can make a more informed decision on which certificate to serve (such as alpn value, server certificate type, etc.) Closes #9833 Signed-off-by: Aditya Patwardhan <aditya.patwardhan@espressif.com>

espressif-bot added the Status: Opened Issue is new label Sep 22, 2022

github-actions bot changed the title ~~add sni support~~ add sni support (IDFGH-8363) Sep 22, 2022

espressif-bot assigned mahavirj Sep 23, 2022

espressif-bot assigned AdityaHPatwardhan and unassigned mahavirj Sep 26, 2022

This was referenced Sep 26, 2022

adds https support esp-rs/esp-idf-svc#136

Closed

Add sni support for https esp-rs/esp-idf-svc#137

Open

axos88 changed the base branch from release/v4.4 to master September 26, 2022 14:15

axos88 force-pushed the add-server-side-sni-support branch from 95af1bd to b1c5f53 Compare September 26, 2022 14:15

axos88 mentioned this pull request Sep 26, 2022

Fix wrong error reporting when https server tries to malloc(0) (IDFGH-8338) #9813

Closed

espressif-bot added Status: In Progress Work is in progress and removed Status: Opened Issue is new labels Sep 27, 2022

AdityaHPatwardhan reviewed Sep 29, 2022

View reviewed changes

axos88 added 3 commits October 3, 2022 08:21

add sni support

b70794b

fix pr comments

5aeb623

fix semicolons

690617b

axos88 requested a review from AdityaHPatwardhan October 6, 2022 09:33

AdityaHPatwardhan reviewed Oct 7, 2022

View reviewed changes

moved to use the CERTIFICATE SELECTION HOOK instead of the SNI HOOK, …

39a6ec0

…since it has access to more information, so the application can make a more informed decision on which certificate to serve (such as alpn value, server certificate type, etc.)

axos88 force-pushed the add-server-side-sni-support branch from 355e718 to 39a6ec0 Compare October 10, 2022 15:27

AdityaHPatwardhan approved these changes Oct 14, 2022

View reviewed changes

espressif-bot added Resolution: Done Issue is done internally Status: Done Issue is done internally and removed Status: In Progress Work is in progress labels Nov 4, 2022

espressif-bot closed this in e9e3dc7 Nov 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add sni support (IDFGH-8363) #9833

add sni support (IDFGH-8363) #9833

axos88 commented Sep 22, 2022 •

edited

CLAassistant commented Sep 22, 2022 •

edited

KaeLL commented Sep 22, 2022

axos88 commented Sep 24, 2022

AdityaHPatwardhan commented Sep 26, 2022

axos88 commented Sep 26, 2022 •

edited

axos88 commented Sep 26, 2022

AdityaHPatwardhan left a comment

AdityaHPatwardhan Sep 29, 2022

axos88 Oct 1, 2022

AdityaHPatwardhan Oct 6, 2022

axos88 Oct 6, 2022 •

edited

AdityaHPatwardhan Oct 7, 2022

axos88 commented Oct 6, 2022

AdityaHPatwardhan Oct 7, 2022 •

edited

AdityaHPatwardhan Oct 7, 2022

AdityaHPatwardhan commented Oct 7, 2022

axos88 commented Oct 7, 2022

axos88 commented Oct 10, 2022

axos88 commented Oct 11, 2022

AdityaHPatwardhan commented Oct 12, 2022

axos88 commented Oct 12, 2022

axos88 commented Oct 12, 2022

AdityaHPatwardhan commented Oct 14, 2022

axos88 commented Oct 14, 2022

axos88 commented Oct 22, 2022

AdityaHPatwardhan commented Oct 27, 2022

add sni support (IDFGH-8363) #9833

add sni support (IDFGH-8363) #9833

Conversation

axos88 commented Sep 22, 2022 • edited

CLAassistant commented Sep 22, 2022 • edited

KaeLL commented Sep 22, 2022

axos88 commented Sep 24, 2022

AdityaHPatwardhan commented Sep 26, 2022

axos88 commented Sep 26, 2022 • edited

axos88 commented Sep 26, 2022

AdityaHPatwardhan left a comment

Choose a reason for hiding this comment

AdityaHPatwardhan Sep 29, 2022

Choose a reason for hiding this comment

axos88 Oct 1, 2022

Choose a reason for hiding this comment

AdityaHPatwardhan Oct 6, 2022

Choose a reason for hiding this comment

axos88 Oct 6, 2022 • edited

Choose a reason for hiding this comment

AdityaHPatwardhan Oct 7, 2022

Choose a reason for hiding this comment

axos88 commented Oct 6, 2022

AdityaHPatwardhan Oct 7, 2022 • edited

Choose a reason for hiding this comment

AdityaHPatwardhan Oct 7, 2022

Choose a reason for hiding this comment

AdityaHPatwardhan commented Oct 7, 2022

axos88 commented Oct 7, 2022

axos88 commented Oct 10, 2022

axos88 commented Oct 11, 2022

AdityaHPatwardhan commented Oct 12, 2022

axos88 commented Oct 12, 2022

axos88 commented Oct 12, 2022

AdityaHPatwardhan commented Oct 14, 2022

axos88 commented Oct 14, 2022

axos88 commented Oct 22, 2022

AdityaHPatwardhan commented Oct 27, 2022

axos88 commented Sep 22, 2022 •

edited

CLAassistant commented Sep 22, 2022 •

edited

axos88 commented Sep 26, 2022 •

edited

axos88 Oct 6, 2022 •

edited

AdityaHPatwardhan Oct 7, 2022 •

edited