Fix issue on truncated certdata.txt file that happens sometimes

[CaViCcHi]

it is possible that sometimes the certdata.txt may not be downloaded correctly, unfortunately there is no checksum available on the file as it is considered a "will always work 100%" but it's happened that the certdata.txt file was downloaded only partially and mk-ca-bundle.pl accepted the file (since it makes no check as of now) and the resulting .crt did not contain all certificates expected

"Added parameter -e to verify that the download of the certdata.txt actually completed succesfully as it has happened that the file was truncated on download and did not contain all certificates expected"

[mention-bot]

@CaViCcHi, thanks for your PR! By analyzing the history of the files in this pull request, we identified @watson81, @gknauf and @bagder to be potential reviewers.

[coveralls]

Coverage increased (+0.03%) to 73.198% when pulling a7a0814 on CaViCcHi:feature/verify_certdatatxt into 615326f on curl:master.

[bagder]

Cool. I approve of these changes, even if the directory check is highly dependent on the serving software providing that directory listing in that format, which makes this feature risking to break in a future when that change. Still I think it can serve a use until then.

But please also update docs/mk-ca-bundle.1 to describe the new option!

[bagder]

I think a better fix, possibly in addition to this, is to check that curl doesn't exit with a non-zero return code. This site provides Content-Length for the data so curl will return an error if it doesn't actually download exactly that amount, which should be sufficient for you...

Your size check is also probably better done as a -I on the actual file URL and not on the directory, as that will make it more robust and likely to be somewhat future-proof.

[jay]

I disagree. This to me is a bug. The server provides a content length. If there's a truncation attack CURLE_RECV_ERROR or CURLE_PARTIAL_FILE would happen. It is our mistake for not catching it. I appreciate the contribution but I'd rather we just fix the bug, I can't see any reason to allow a partial certdata.

diff --git a/lib/mk-ca-bundle.pl b/lib/mk-ca-bundle.pl
index 9574f1d..cc36c76 100755
--- a/lib/mk-ca-bundle.pl
+++ b/lib/mk-ca-bundle.pl
@@ -310,7 +310,7 @@ if(!$opt_n) {
         my $proto = !$opt_k ? "--proto =https" : "";
         my $quiet = $opt_q ? "-s" : "";
         my @out = `curl -w %{response_code} $proto $quiet -o "$txt" "$url"`;
-        if(@out && $out[0] == 200) {
+        if(!$? && @out && $out[0] == 200) {
           $fetched = 1;
           report "Downloaded $txt";
         }

[bagder]

I'm changing my view on this. We should primarily check the return code .

[jay]

I'm changing my view on this. We should primarily check the return code .

curl return code check landed in ec92afc. Thanks for the report @CaViCcHi.

[bagder]

thanks @jay!

@CaViCcHi, with this fix landed, is there actually any issue remaining for you?

[CaViCcHi]

No guys, thanks; I think you know better how the innerworks of curl operate :) thanks @jay && @bagder