Avoid inserting duplicate certificates or precertificates

[pgporada]

@jcjones mentioned in #5467

It's always going to be possible now to accidentally have rows in the certificates and precertificates table now referring to the same serial number, so we need to be a bit more robust in handling it.

[boulder]> select registrationID, serial, digest, issued, expires from certificates where serial = 'fac11d1135582055be574949f41cdf603e83';
+----------------+--------------------------------------+---------------------------------------------+---------------------+---------------------+
| registrationID | serial                               | digest                                      | issued              | expires             |
+----------------+--------------------------------------+---------------------------------------------+---------------------+---------------------+
|       xxxxxxxx | fac11d1135582055be574949f41cdf603e83 | C753-HBhX1bU6wCgzbS4IZCQ50DreAdWqjnlQgj7zTU | 2021-03-17 21:48:28 | 2021-06-15 20:48:23 |
|       xxxxxxxx | fac11d1135582055be574949f41cdf603e83 | C753-HBhX1bU6wCgzbS4IZCQ50DreAdWqjnlQgj7zTU | 2021-03-17 21:48:23 | 2021-06-15 20:48:23 |
+----------------+--------------------------------------+---------------------------------------------+---------------------+---------------------+
2 rows in set (0.000 sec)

2021-03-17T21:49:21.054916+00:00 ca02 staging 6 boulder-ca[1138777]: ur73igM [AUDIT] Incorporated orphaned certificate: serial=[fac11d1135582055be574949f41cdf603e83] cert=[308 ...

Coincidently this occurred when we deployed #5311 to staging and restarted the boulder stack. Metrics showed ~1400 orphaned certificates detected as a result of the restart with ~1390 of them being adopted.

We've scanned the production DB and have not detected any duplicate certs there.

[jcjones]

Note: This could be as simple as starting a transaction and doing a SelectCertificate(map, serial) and ensuring we get Err.NoRows before continuing with orphan management and then committing the transaction.

While this could be done to all saves to Certificates, it makes sense to me to start with the orphans and, if we end up re-opening, then we could add a robust "insert certificate" procedure for all cases. Given the overhead on a hot path, I'd argue we should consider that as a real stored procedure rather than making the SA do multiple roundtrips on each issuance -- so maybe let's start simple.

[aarongable]

JC says this is high priority. Grabbing for this milestone immediately. Short term: add transaction get-then-insert to Orphan Finder. Mid term: see if there are any other codepaths that might result in duplicate entries. Long term: if yes, consider replacing transaction with stored procedure to cover all possible inserters.