New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alternative CA database generator #22254

Merged
merged 2 commits into from Nov 26, 2018

Conversation

Projects
None yet
6 participants
@Darkspirit
Contributor

Darkspirit commented Nov 23, 2018

resources/certs is Servo's CA database and was last updated on 3 Apr 2017. Symantec has been revoked in the meantime.
etc/cert_generator.js is an XPCShell script for updating the certs file, but I found it too complex to run, so I made a simpler script.

What it does:

  1. Download Mozilla's official CA database CSV file with curl and process it with awk: https://ccadb-public.secure.force.com/mozilla/IncludedCACertificateReportPEMCSV
  2. rows end with "\n
  3. split each row by ^" and "," into columns. (I had to remove "$ because of servo-tidy.)
  4. remove single and double quotes from column 30
  5. if column 13 (12 in the csv file) contains Websites (some are Email-only), print column 30, the raw certificate
  6. put everything into a certs file

servo-tidy wasn't happy so I had to make it a bit uglier. lol

./etc/cert_generator.sh:8: Line is longer than 80 characters
./etc/cert_generator.sh:8: script is missing options "set -o errexit", "set -o nounset", "set -o pipefail"
./etc/cert_generator.sh:8: variable substitutions should use the full "${VAR}" form
./etc/cert_generator.sh:8: variable substitutions should use the full "${VAR}" form
./etc/cert_generator.sh:8: variable substitutions should use the full "${VAR}" form
./etc/cert_generator.sh:8: variable substitutions should use the full "${VAR}" form


  • ./mach test-tidy does not report any errors

This change is Reviewable

@paulrouget

This comment has been minimized.

Contributor

paulrouget commented Nov 23, 2018

Can you add the above comments to the script?
Otherwise, looks good to me. I had to replace awk with gawk on Mac.

@Darkspirit Darkspirit force-pushed the Darkspirit:ca_db_generator branch from 37bd256 to 84f720b Nov 23, 2018

@Darkspirit

This comment has been minimized.

Contributor

Darkspirit commented Nov 23, 2018

Ok. Here on my Debian Testing, awk is a shortcut for gawk, so we might just use gawk directly. Or would you prefer changing it back to awk?

@Darkspirit Darkspirit force-pushed the Darkspirit:ca_db_generator branch from 84f720b to 8f4a026 Nov 24, 2018

Darkspirit added some commits Nov 24, 2018

@Darkspirit Darkspirit force-pushed the Darkspirit:ca_db_generator branch from 8f4a026 to 949c224 Nov 24, 2018

@paulrouget

This comment has been minimized.

Contributor

paulrouget commented Nov 25, 2018

gawk works. Thank you.

@bors-servo r+

@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Nov 25, 2018

📌 Commit 949c224 has been approved by paulrouget

@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Nov 25, 2018

⌛️ Testing commit 949c224 with merge 3ca1d55...

bors-servo added a commit that referenced this pull request Nov 25, 2018

Auto merge of #22254 - Darkspirit:ca_db_generator, r=paulrouget
Alternative CA database generator

[resources/certs](https://github.com/servo/servo/blob/master/resources/certs) is Servo's CA database and was last updated on 3 Apr 2017. Symantec has been revoked in the meantime.
[etc/cert_generator.js](https://github.com/servo/servo/blob/b9847e29537f2c7dc3515f28342f9b519f2c9790/etc/cert_generator.js) is an XPCShell script for updating the certs file, but I found it too complex to run, so I made a simpler script.

What it does:
1. Download Mozilla's [official](https://wiki.mozilla.org/CA/Included_Certificates) CA database CSV file with curl and process it with awk: https://ccadb-public.secure.force.com/mozilla/IncludedCACertificateReportPEMCSV
2. rows end with "\n
3. split each row by `^"` and `","` into columns. (I had to remove `"$` because of servo-tidy.)
4. remove single and double quotes from column 30
5. if column 13 (12 in the csv file) contains `Websites` (some are Email-only), print column 30, the raw certificate
6. put everything into a `certs` file

servo-tidy wasn't happy so I had to make it a bit uglier. lol
> ./etc/cert_generator.sh:8: Line is longer than 80 characters
> ./etc/cert_generator.sh:8: script is missing options "set -o errexit", "set -o nounset", "set -o pipefail"
> ./etc/cert_generator.sh:8: variable substitutions should use the full "${VAR}" form
> ./etc/cert_generator.sh:8: variable substitutions should use the full "${VAR}" form
> ./etc/cert_generator.sh:8: variable substitutions should use the full "${VAR}" form
> ./etc/cert_generator.sh:8: variable substitutions should use the full "${VAR}" form

---
- [x] `./mach test-tidy` does not report any errors

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/22254)
<!-- Reviewable:end -->
@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Nov 25, 2018

💔 Test failed - linux-rel-wpt

@Darkspirit

This comment has been minimized.

Contributor

Darkspirit commented Nov 25, 2018

@paulrouget

This comment has been minimized.

Contributor

paulrouget commented Nov 26, 2018

@bors-servo

This comment has been minimized.

@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Nov 26, 2018

💔 Test failed - linux-rel-css

@CYBAI

This comment has been minimized.

Collaborator

CYBAI commented Nov 26, 2018

Failed with servo/intermittent-failure-tracker#5 again. Maybe need someone to fix it in builders.

@jdm

This comment has been minimized.

Member

jdm commented Nov 26, 2018

@bors-servo retry

@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Nov 26, 2018

⌛️ Testing commit 949c224 with merge 524bd72...

bors-servo added a commit that referenced this pull request Nov 26, 2018

Auto merge of #22254 - Darkspirit:ca_db_generator, r=paulrouget
Alternative CA database generator

[resources/certs](https://github.com/servo/servo/blob/master/resources/certs) is Servo's CA database and was last updated on 3 Apr 2017. Symantec has been revoked in the meantime.
[etc/cert_generator.js](https://github.com/servo/servo/blob/b9847e29537f2c7dc3515f28342f9b519f2c9790/etc/cert_generator.js) is an XPCShell script for updating the certs file, but I found it too complex to run, so I made a simpler script.

What it does:
1. Download Mozilla's [official](https://wiki.mozilla.org/CA/Included_Certificates) CA database CSV file with curl and process it with awk: https://ccadb-public.secure.force.com/mozilla/IncludedCACertificateReportPEMCSV
2. rows end with "\n
3. split each row by `^"` and `","` into columns. (I had to remove `"$` because of servo-tidy.)
4. remove single and double quotes from column 30
5. if column 13 (12 in the csv file) contains `Websites` (some are Email-only), print column 30, the raw certificate
6. put everything into a `certs` file

servo-tidy wasn't happy so I had to make it a bit uglier. lol
> ./etc/cert_generator.sh:8: Line is longer than 80 characters
> ./etc/cert_generator.sh:8: script is missing options "set -o errexit", "set -o nounset", "set -o pipefail"
> ./etc/cert_generator.sh:8: variable substitutions should use the full "${VAR}" form
> ./etc/cert_generator.sh:8: variable substitutions should use the full "${VAR}" form
> ./etc/cert_generator.sh:8: variable substitutions should use the full "${VAR}" form
> ./etc/cert_generator.sh:8: variable substitutions should use the full "${VAR}" form

---
- [x] `./mach test-tidy` does not report any errors

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/22254)
<!-- Reviewable:end -->
@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Nov 26, 2018

💔 Test failed - linux-rel-wpt

@jdm

This comment has been minimized.

Member

jdm commented Nov 26, 2018

@bors-servo

This comment has been minimized.

@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Nov 26, 2018

💔 Test failed - mac-rel-wpt1

@jdm

This comment has been minimized.

Member

jdm commented Nov 26, 2018

@bors-servo

This comment has been minimized.

@bors-servo

This comment has been minimized.

@bors-servo bors-servo merged commit 949c224 into servo:master Nov 26, 2018

3 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
homu Test successful
Details

@Darkspirit Darkspirit deleted the Darkspirit:ca_db_generator branch Nov 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment