1620226: Perform CRL generation without exceeding memory and maxing CPU

[mstead]

Using a HashSet instead of an ArrayList to store the serials improves
the performance of .contains when checking if an entry in the CRL file
should be removed due to expiry.

Batch lookup/process fetched serials to keep the memory footprint down
and to allow garbage collection of the serials earlier. The number of
records per batch can be configured via the candlepin config file.

NOTE: Due to the way the CRL generation framework works, when batching
the serial processing, we end up re-scanning the crl num_serials/batch_size times.
This isn't ideal, but we will make due until we make the move to generating
a plain text serial file.

Testing

1. Deploy candlepin from the master branch.
2. Stop tomcat, drop the candlepin DB and replace it with one loaded with millions of cert serials:

$ wget http://file.rdu.redhat.com/mstead/pr/1620226/db-with-lots-of-serials-for-crl-testing.tar.gz
$ sudo systemctl stop tomcat
$ tar xvzf db-with-lots-of-serials-for-crl-testing.tar.gz
$ dropdb -U candlepin candlepin && createdb -U candlepin candlepin && psql -U candlepin candlepin < lots-of-serials-for-crl-gen-testing.sql

3. Restart tomcat and manually run the CRL job. It won't take long before OOM exception is thrown.

$ sudo systemctl start tomcat
$ curl -k -u admin:admin  -X POST https://localhost:8443/candlepin/jobs/schedule/CertificateRevocationListTask

4. Set up logging so that you can track the progress of the job.

log4j.logger.org.candlepin.util.CrlFileUtil=DEBUG

5. Deploy candlepin from this PR's branch against the same database.

$ bin/deploy

6. In a separate terminal, tail the candlepin log so that you can see what's going on.

# Generally I tail the log just to get the job id, then use grep to narrow to just that job.
tail -f /var/log/candlepin/candlepin.log
# once the job is running, re-tail and use grep.
tail -f /var/log/candlepin/candlepin.log | grep <YOUR_JOB_ID>

7. In a separate terminal, use 'top' to monitor tomcat's cpu and memory usage.

top | grep tomcat

8. Run the CRL job again. It will take about 1h - 1h20m to fully complete. Memory usage shouldn't rise more than a couple Gig and CPU should only go to 100% in bursts.

curl -k -u admin:admin  -X POST https://localhost:8443/candlepin/jobs/schedule/CertificateRevocationListTask

[mstead]

retest this please

[Ceiu]

👍

[Ceiu]

Given how we use the batchSize, is it something we want to do once here rather than implicitly forcing the configured value to be looked up by the two callers?

[mstead]

I'm Ok with either. I was just trying to avoid injecting the candlepin config into the curator. No big deal if you'd like to see this changed.

[Ceiu]

Up to you, given discussions around this class.

For what it's worth, I see the config lookup being done here internally to the CrlFileUtil rather than the curator. But if the curator itself ended up needing it, it's already injected into every curator through the AbstractHibernatCurator; so we're good there.

[mstead]

Fair points, I'll move it into the CrlFileUtil.

[mstead]

DONE. Once you give the changes a once over I'll squash the update commit before it is merged.

[Ceiu]

Changes look good

[Ceiu]

This is good to be merged, but I want to give a chance to respond to the comments above regarding the location of the config reading.

[Ceiu]

Changes look good. Squash away when you have time.

[mstead]

Changes look good. Squash away when you have time.

DONE