Letsencrypt autorenewal fails for OKD 3.11

[bdlink]

The install successfully gets a certificate with dns validation.
cron command is run weekly but fails to renew the certificate, even though it determines it needs renewal.
cron command: certbot renew --pre-hook="oc scale --replicas=0 dc router" --post-hook="oc scale --replicas=1 dc router"
Error message: An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.

Error log:
2019-05-17 20:35:30,543:DEBUG:certbot.main:certbot version: 0.30.2
2019-05-17 20:35:30,543:DEBUG:certbot.main:Arguments: ['--pre-hook=oc scale --replicas=0 dc router', '--post-hook=oc scale --replicas=1 dc router']
2019-05-17 20:35:30,543:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-05-17 20:35:30,567:DEBUG:certbot.log:Root logging level set at 20
2019-05-17 20:35:30,567:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2019-05-17 20:35:30,580:DEBUG:certbot.plugins.selection:Requested authenticator <certbot.cli._Default object at 0x7fe028108450> and installer <certbot.cli._Default object at 0x7fe028108450>
2019-05-17 20:35:30,588:DEBUG:certbot.storage:Should renew, less than 30 days before certificate expiry 2019-05-24 18:21:43 UTC.
2019-05-17 20:35:30,589:INFO:certbot.renewal:Cert is due for renewal, auto-renewing...
2019-05-17 20:35:30,589:DEBUG:certbot.plugins.selection:Requested authenticator manual and installer None
2019-05-17 20:35:30,590:DEBUG:certbot.plugins.disco:Other error:(PluginEntryPoint#manual): An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/certbot/plugins/disco.py", line 132, in prepare
self._initialized.prepare()
File "/usr/lib/python2.7/site-packages/certbot/plugins/manual.py", line 133, in prepare
self.option_name('auth-hook')))
PluginError: An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.
2019-05-17 20:35:30,590:DEBUG:certbot.plugins.selection:No candidate plugin
2019-05-17 20:35:30,590:DEBUG:certbot.plugins.selection:Selected authenticator None and installer None
2019-05-17 20:35:30,590:INFO:certbot.main:Could not choose appropriate plugin: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.',)
2019-05-17 20:35:30,590:WARNING:certbot.renewal:Attempting to renew cert (shift.infoteach.ca) from /etc/letsencrypt/renewal/shift.infoteach.ca.conf produced an unexpected error: The manual plugin is not working; th
ere may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.',). Skipping.
2019-05-17 20:35:30,591:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 452, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1186, in renew_cert
installer, auth = plug_sel.choose_configurator_plugins(config, plugins, "certonly")
File "/usr/lib/python2.7/site-packages/certbot/plugins/selection.py", line 237, in choose_configurator_plugins
diagnose_configurator_problem("authenticator", req_auth, plugins)
File "/usr/lib/python2.7/site-packages/certbot/plugins/selection.py", line 341, in diagnose_configurator_problem
raise errors.PluginSelectionError(msg)
PluginSelectionError: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.',)

2019-05-17 20:35:30,591:ERROR:certbot.renewal:All renewal attempts failed. The following certs could not be renewed:
2019-05-17 20:35:30,592:ERROR:certbot.renewal: /etc/letsencrypt/live/shift.infoteach.ca/fullchain.pem (failure)
2019-05-17 20:35:30,592:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File "/bin/certbot", line 9, in
load_entry_point('certbot==0.30.2', 'console_scripts', 'certbot')()
File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1364, in main
return config.func(config, plugins)
File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1271, in renew
renewal.handle_renewal_request(config)
File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 477, in handle_renewal_request
len(renew_failures), len(parse_failures)))
Error: 1 renew failure(s), 0 parse failure(s)

[nadilas]

I managed to get around this by using acme.sh on the host:
GD_Key="" GD_Secret="" ~/.acme.sh/acme.sh --renew --renew-hook ~/scaledown.sh --post-hook ~/scaleup.sh --dns dns_gd --cert-home /etc/letsencrypt/live -d domain -d ".domain" -d ".apps.domain" --cert-file /etc/letsencrypt/live/domain/cert.pem --key-file /etc/letsencrypt/live/domain/privkey.pem --fullchain-file /etc/letsencrypt/live/domain/fullchain.pem --ca-file /etc/letsencrypt/live/domain/chain.pem

Even though it was done, OKD doesn't seem to recognize the renewed certificates (still showing old expiration date in the web browser). However console is happy:

Cert not yet due for renewal

I presume I need to redeploy them, but I forgot how. :)

[nadilas]

So it took time some time, but here's what you do, after changing to acme.sh:

ansible-playbook -i inventory.ini openshift-ansible/playbooks/redeploy-certificates.yml

use updated openshift-ansible (3.11.104-1) PR: openshift/openshift-ansible#11472