1503700: Updates to the job polling frequency

[cnsnyder]

After this update destination threads will poll for job status in the following
way.

The first job status check is made 15 seconds after submitting the job.
All subsequent job status checks that do not indicate the job is finished, cancelled or failed
result in a wait time of 30 seconds.

If during any job check an HTTP code 429 is returned including a 'Retry-After' header with a
value parsable as an integer, that value is used as the number of seconds to wait before the
next job check.

As soon as any job check returns success no more polling is done for that job.
This does not change the behavior of oneshot (still exits on 429 or other exception).

A fun way to test this is to use mitmproxy to fake 429s with different headers coming back.

1. Here is a libvirt config I used for testing this a bit (put this in a .conf file in /etc/virt-who.d):

[LOCAL_LIBVIRT]
type=libvirt
server=qemu:///system
owner=admin
env=admin
rhsm_hostname=localhost
rhsm_port=8443
rhsm_username=admin
rhsm_password=admin
rhsm_prefix=/candlepin
rhsm_proxy_hostname=localhost
rhsm_proxy_port=12345
#filter_hosts=d5b15581-5440-11cb-930e-b5dfde6103bc
hypervisor_id=hostname

2. deploy CP locally with test data

3. start mitmproxy (if you don't have it you should be able to install using pip pip install mitmproxy
mitmproxy --insecure --port 12345

4. Configure mitmproxy's intercept filter:
   a) press 'i'
   b) enter the following (without quotes) as the filter expression: ~bs [jJ]ob ~m GET

5. run virt-who from the checkout: ./virt-who -d

6. watch mitmproxy as requests come in. It should highlight and intercept any job status calls.

7. Move down to the intercepted call using the arrow keys and press enter

8. Press tab to go to the response

9. Press 'e' and then 'o' to edit the response code, delete the value already there and replace with 429 and press enter

10. Press 'e' and then 'h' to edit the headers

11. Move down to the last header by using the arrow keys and press 'a' to add a new one

12. Add the header 'Retry-After', press tab, type your desired wait time and press enter

13. press 'q' a few times to get back to the main screen of mitmproxy, then press 'a' to allow the modified response to be delivered to virt-who

14. Profit!! (and by this I mean, watch for virt-who to wait to retry the polling for the number of seconds you specified as the Retry-After header in step 12)

(with any luck this will serve as a useful mini-tutorial for someone)

[cnsnyder]

[13] Bug 1503700 - Virt-who polls job status too quickly

[wottop]

ack