Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support CA bundle in memory with CURLOPT_CAINFO_BLOB and CURLOPT_PROXY_CAINFO_BLOB #6662

Closed
wants to merge 1 commit into from

Conversation

@gvollant
Copy link
Contributor

@gvollant gvollant commented Feb 25, 2021

Adds CURLOPT_CAINFO_BLOB and CURLOPT_PROXY_CAINFO_BLOB that let applications provide the CA bundle to libcurl as a PEM formatted in-memory buffer.

This is an alternative of pr #6109 , which uses blob (introduced with #5357 on curl 7.71) instead char* string to hold PEM certificates.

On #6109 , @bagder and @jay discuss about using blob possible advantage

@gvollant gvollant force-pushed the gvollant:gv_blob_cacert branch from 006fdd7 to 56ce2e9 Feb 25, 2021
@gvollant
Copy link
Contributor Author

@gvollant gvollant commented Feb 25, 2021

@JCMais and @nadrolinux your test are also welcomes

@gvollant
Copy link
Contributor Author

@gvollant gvollant commented Feb 26, 2021

I don't understand the two check problem, probably not related to my code.
I don't plan other modification.
If you suggest documentation modification, can you give me exact text (as you see, my english is not perfect).

regards

@gvollant gvollant force-pushed the gvollant:gv_blob_cacert branch 2 times, most recently from 6134a0a to e52a690 Feb 26, 2021
@jay
Copy link
Member

@jay jay commented Mar 3, 2021

I have made some suggested changes. One of them is that the blob override regular cainfo. I don't understand why some ssl primary options are accessed by SSL_CONN_CONFIG and the others are accessed by SSL_SET_OPTION(primary.foo, for example SSL_CONN_CONFIG(CAfile) but for the blob it's SSL_SET_OPTION(primary.ca_info_blob). Why is it not SSL_CONN_CONFIG(ca_info_blob);, if a blob is copied to the connection shouldn't we use that copy or do I not understand this?

@gvollant gvollant force-pushed the gvollant:gv_blob_cacert branch from 7be118e to 8fc81a2 Mar 6, 2021
@gvollant
Copy link
Contributor Author

@gvollant gvollant commented Mar 6, 2021

SSL_CONN_CONFIG

@jay Thank you for your modification.
I switched from SSL_SET_OPTION to SSL_CONN_CONFIG for ca_info_blob, like CAfile, and merged your modification

@gvollant gvollant force-pushed the gvollant:gv_blob_cacert branch 2 times, most recently from 1f5506b to 575b344 Mar 9, 2021
@nadrolinux
Copy link

@nadrolinux nadrolinux commented Mar 12, 2021

Glad to see progress on this stuff. Later this week I'll try to test this version on both Linux and Android.

@gvollant
Copy link
Contributor Author

@gvollant gvollant commented Mar 22, 2021

I added support on Secure Transport and Schannel backends, like others blobs
Next step is probably all blob on toher popular vtls ...

@jay
Copy link
Member

@jay jay commented Mar 22, 2021

I added support on Secure Transport and Schannel backends, like others blobs

That's great. I still don't know whether it's proper to access the blobs with SSL_SET_OPTION(primary.foo or SSL_CONN_CONFIG(foo). My understanding of the latter is it carries with the connection, so I guess is copied from the transfer. I wonder why though CAfile or ca blob needs to go with the connection. In the case of blob it just seems like an extra unnecessary copy.

@gvollant
Copy link
Contributor Author

@gvollant gvollant commented Mar 23, 2021

That's great. I still don't know whether it's proper to access the blobs with SSL_SET_OPTION(primary.foo or SSL_CONN_CONFIG(foo). My understanding of the latter is it carries with the connection, so I guess is copied from the transfer. I wonder why though CAfile or ca blob needs to go with the connection. In the case of blob it just seems like an extra unnecessary copy.

I've no opinion, but I suppose the method for CAINFO (filename) and CAINFO_BLOB must be the same. @bagder probably have an idea...

@gvollant gvollant force-pushed the gvollant:gv_blob_cacert branch 2 times, most recently from f81292c to c504789 Mar 23, 2021
@gvollant gvollant force-pushed the gvollant:gv_blob_cacert branch 2 times, most recently from 4deeb2e to d47fc0a Mar 31, 2021
@gvollant
Copy link
Contributor Author

@gvollant gvollant commented Mar 31, 2021

I wrote on #6820 a comment about blob test.

@jay
Copy link
Member

@jay jay commented Mar 31, 2021

I wrote on #6820 a comment about blob test.

I've copied the relevant part of your comment below.

--cert "loadmem=F\:\\demosync\\cert_4\\iosad.p12:vfdffrefre" 

will load the file and use CURLOPT_SSLCERT_BLOB

That seems unnecessary for a blob test. Rather than use the curl tool to test you can use just libcurl, refer to libtest directory

@gvollant
Copy link
Contributor Author

@gvollant gvollant commented Apr 6, 2021

Do you think there are more work needed on this PR ?

@gvollant gvollant force-pushed the gvollant:gv_blob_cacert branch from d47fc0a to ecc18ad Apr 6, 2021
@jay
Copy link
Member

@jay jay commented Apr 7, 2021

Do you think there are more work needed on this PR ?

I will review it again when the feature window opens

@gvollant
Copy link
Contributor Author

@gvollant gvollant commented Apr 15, 2021

Do you think there are more work needed on this PR ?

I will review it again when the feature window opens

Thank you. So feature window is only 2 week I believe.

@jay
Copy link
Member

@jay jay commented Apr 15, 2021

Thank you. So feature window is only 2 week I believe.

yes if there are no serious problems in 7.76.1 then it runs from next week until may 5. release calendar

@gvollant
Copy link
Contributor Author

@gvollant gvollant commented Apr 20, 2021

I rebase to master each time openssl.c is modifed.

@bagder : what is you opinion about the @jay question about SSL_SET_OPTION(primary.foo vs SSL_CONN_CONFIG(foo). ?
I just think blob and filename must use smae method for same information

@bagder
Copy link
Member

@bagder bagder commented May 2, 2021

I want the test(s) to get run by the curl test suite so that A) we don't inadvertently break this functionality going forward and B) we can run torture, valgrind and sanitizer tests on it.

If you can write up a small stand-alone libcurl-using application that uses this functionality, I can help you adjust it to work in the test suite. Since we know what cert the test server is running with, I figure we could create working blobs that can work to verify that cert. And to fail the verification to have that path tested too!

@gvollant
Copy link
Contributor Author

@gvollant gvollant commented May 2, 2021

@bagder I wrote a simple code and uploaded it to http://gvollant.free.fr/demo_curl_portion.c

You can probably use it with a test based on test 310 , for schannel, openssl and sectransp

@gvollant
Copy link
Contributor Author

@gvollant gvollant commented May 2, 2021

static int test_loadfile(const char *filename, void **filedata, size_t *filesize)
{
  size_t datasize = 0;
  void *data = NULL;
  if(filename) {
    FILE *fInCert = fopen(filename, "rb");

    if(fInCert) {
      long cert_tell = 0;
      bool continue_reading = fseek(fInCert, 0, SEEK_END) == 0;
      if(continue_reading)
        cert_tell = ftell(fInCert);
      if(cert_tell < 0)
        continue_reading = FALSE;
      else
        datasize = (size_t)cert_tell;
      if(continue_reading)
         continue_reading = fseek(fInCert, 0, SEEK_SET) == 0;
      if(continue_reading)
          data = malloc(datasize + 1);
      if((!data) ||
          ((int)fread(data, datasize, 1, fInCert) != 1))
          continue_reading = FALSE;
      fclose(fInCert);
      if(!continue_reading) {
        free(data);
        datasize = 0;
        data = NULL;
      }
   }
  }
  *filesize = datasize;
  *filedata = data;
  return data ? 1 : 0;
}

CURLcode test_cert_file(const char *url, const char *cafile)
{
  CURLcode code;
  CURL *curl;

  curl = curl_easy_init();

  curl_easy_setopt(curl, CURLOPT_VERBOSE,    1L);
  curl_easy_setopt(curl, CURLOPT_URL,        url);
  curl_easy_setopt(curl, CURLOPT_CAINFO,     cafile);
  if(!curl) {
    fprintf(stderr, "curl_easy_init() failed\n");
    return CURLE_FAILED_INIT;
  }

  printf("PERFORM CURLOPT_CAINFO\n");
  code = curl_easy_perform(curl);

test_cleanup:
  printf("CLEANUP\n");
  curl_easy_cleanup(curl);
  curl_global_cleanup();

  return code;
}


CURLcode test_cert_blob(const char *url, const char *cafile)
{
  CURLcode code;
  CURL *curl;
  struct curl_blob blob;
  size_t certsize;
  void *certdata;

  curl = curl_easy_init();
  if(!curl) {
    fprintf(stderr, "curl_easy_init() failed\n");
    return CURLE_FAILED_INIT;
  }

  if(!test_loadfile(cafile, &certdata, &certsize)) {
    return CURLE_READ_ERROR;
  }

  curl_easy_setopt(curl, CURLOPT_VERBOSE,     1L);
  curl_easy_setopt(curl, CURLOPT_URL,         url);

  blob.data = certdata;
  blob.len = certsize;
  blob.flags = CURL_BLOB_COPY;
  curl_easy_setopt(curl, CURLOPT_CAINFO_BLOB, &blob);
  free(certdata);


  printf("PERFORM CURLOPT_CAINFO_BLOB\n");
  code = curl_easy_perform(curl);

test_cleanup:
  printf("CLEANUP\n");
  curl_easy_cleanup(curl);
  curl_global_cleanup();

  return code;
}


CURLcode test_cert_both(const char *url, const char *cafile)
{
    CURLcode res = test_cert_file(url, cafile);
    if(!res)
      res = test_cert_blob(url, cafile);
    return res;
}
@gvollant gvollant force-pushed the gvollant:gv_blob_cacert branch from ab90637 to 0fed758 May 3, 2021
lib/vtls/sectransp.c Show resolved Hide resolved
@bagder
Copy link
Member

@bagder bagder commented May 4, 2021

I wrote this commit that adds a check for memmem so that the native function can be used on platforms that have it.

@bagder
Copy link
Member

@bagder bagder commented May 4, 2021

Here's a patch that adds test 678 that verifies CURLOPT_CAINFO_BLOB. It is based on #6662 (comment) and it worked when I try it here in my local test of this branch. It might need some adjustments to also work with torture tests (./runtests.pl -t 678) but I didn't check.

0001-test678-verify-CURLOPT_CAINFO_BLOB.patch.txt

@gvollant gvollant force-pushed the gvollant:gv_blob_cacert branch 2 times, most recently from e5b3a5f to c928c4f May 4, 2021
@gvollant
Copy link
Contributor Author

@gvollant gvollant commented May 4, 2021

Here's a patch that adds test 678 that verifies CURLOPT_CAINFO_BLOB. It is based on #6662 (comment) and it worked when I try it here in my local test of this branch. It might need some adjustments to also work with torture tests (./runtests.pl -t 678) but I didn't check.

0001-test678-verify-CURLOPT_CAINFO_BLOB.patch.txt

I integrated your test. What about adding a section to test with openssl, schannel and sectransport only?

@gvollant
Copy link
Contributor Author

@gvollant gvollant commented May 4, 2021

On you file, this portion of file test678 have CRLF ending. Is it needed?

GET /%TESTNUMBER HTTP/1.1
Host: localhost:%HTTPSPORT
User-Agent: CURLOPT_CAINFO_BLOB
Accept: */*
@bagder
Copy link
Member

@bagder bagder commented May 4, 2021

this portion of file test678 have CRLF ending. Is it needed?

It is needed, and you probably see that the test fails without them. They are present in my local version, but I forgot that they don't survive a git format-patch.

@bagder
Copy link
Member

@bagder bagder commented May 4, 2021

What about adding a section to test with openssl, schannel and sectransport only?

Ah right. That needs to be handled too somehow. I'll work on it.

@gvollant
Copy link
Contributor Author

@gvollant gvollant commented May 4, 2021

There is fail on https://dev.azure.com/daniel0244/5571c33c-81e1-43d7-93ee-d3d4152f27b2/_apis/build/builds/5612/logs/100 and I don't have knowledge to fix them.

test 678 seem ok at https://cirrus-ci.com/task/5675663530131456

Note: You can commit on my github fork.I invited you, and I beleive as maintainer of original repo, you have modification right on my repo

@bagder
Copy link
Member

@bagder bagder commented May 4, 2021

Okay, I added a commit that makes test 678 only run if it can use the setopt. So I made the setopt fail properly if libcurl doesn't support the option in that build setup.

@gvollant
Copy link
Contributor Author

@gvollant gvollant commented May 4, 2021

Do you want I rebase and merge all commit ?

@gvollant
Copy link
Contributor Author

@gvollant gvollant commented May 4, 2021

They are some schannel error

https://dev.azure.com/daniel0244/5571c33c-81e1-43d7-93ee-d3d4152f27b2/_apis/build/builds/5616/logs/137

2021-05-04T14:44:18.6522953Z  * schannel: added 1 certificate(s) from CA file '(memory blob)'
2021-05-04T14:44:18.6523738Z  * schannel: CertGetCertificateChain trust error CERT_TRUST_REVOCATION_STATUS_UNKNOWN

Pehaps solution is using CURLSSLOPT_NO_REVOKE or CURLSSLOPT_REVOKE_BEST_EFFORT on test 678

…Y_CAINFO_PEM

Let applications provide the CA bundle to libcurl as a PEM formatted in-memory buffer.

Rebased branch bundled_cacert from moparisthebest

Validation : 397e11b [397e11b]
Parents : 4506607
Auteur : moparisthebest <admin@moparisthebest.com>
Date : vendredi 3 avril 2020 00:32:42
Auteur : moparisthebest
Date de validation : samedi 4 avril 2020 00:31:23
Add CURLOPT_CAINFO_PEM
@gvollant gvollant force-pushed the gvollant:gv_blob_cacert branch from c091660 to d87860a May 5, 2021
@gvollant
Copy link
Contributor Author

@gvollant gvollant commented May 5, 2021

It seem test failure are now not related with these PR modification. I do a rebase

@jay jay closed this in 77fc385 May 5, 2021
@jay
Copy link
Member

@jay jay commented May 5, 2021

Thanks guys

@gvollant gvollant deleted the gvollant:gv_blob_cacert branch May 5, 2021
@mback2k
Copy link
Member

@mback2k mback2k commented May 5, 2021

Actually lib678_SOURCES is missing $(MULTIBYTE) in tests\libtest\Makefile.inc since you are using fopen which is defined to curlx_win32_fopen in Windows Unicode builds. Should I do a follow up or will one of you do it? @jay @gvollant @bagder

Error can be seen here: https://cirrus-ci.com/task/4924099123216384?logs=compile#L1095

jay added a commit that referenced this pull request May 5, 2021
Follow-up to 77fc385 from yesterday.

Bug: #6662 (comment)
Reported-by: Marc Hörsken
@jay
Copy link
Member

@jay jay commented May 5, 2021

Should I do a follow up or will one of you do it?

My fault. Thanks, landed in 5a1ec19.

@JCMais
Copy link
Contributor

@JCMais JCMais commented May 5, 2021

Awesome job you all on getting this PR merged. This feature will be very useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants