[WIP] feat(pattern): add job harvester pattern

[estroz]

Description of the change: added the pattern/jobharvester package with Job log forwarding and garbage collection functionality.

Motivation for the change: lots of operators spawn jobs for various tasks. This simple controller exposes both the primitive Harvester type to get a Job's Pod logs then delete the Pod, or a higher level controller implementation for low-LOC, background Job harvesting.

/kind feature

Signed-off-by: Eric Stroczynski ericstroczynski@gmail.com

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please ask for approval from estroz after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coveralls]

Pull Request Test Coverage Report for Build 1083336552

• 93 of 332 (28.01%) changed or added relevant lines in 5 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-20.3%) to 60.532%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pattern/jobharvest/clientgo.go	0	11	0.0%
pattern/jobharvest/webhook.go	4	27	14.81%
pattern/jobharvest/controller.go	8	67	11.94%
pattern/jobharvest/ctrl.go	0	69	0.0%
pattern/jobharvest/harvester.go	81	158	51.27%

Totals
Change from base Build 1075789896:	-20.3%
Covered Lines:	523
Relevant Lines:	864

---

💛 - Coveralls

[estroz]

@tlwu2013 @jmccormick2001 whipped up this super simple (functionality-wise) POC after our discussion yesterday. All this does is write a Job's Pod logs streams to some writer, then delete the Job. I'd like to try out some ideas for log writer interfaces as the current LogReader API is rudimentary. Let me know if you have any!

[estroz]

There is an alternative to this implementation, which does not involve adding a new controller/helper for cleaning up jobs, that imo is a more desirable approach but involves more input from library users:

1. Provide a helper function that appends a log collection sidecar to a Job's Pod template that can access all container logs. This container must have authorization to connect to and be able to authenticate with a log server that receives logs.
2. Use the ttlSecondsAfterFinished field to garbage collect the Job and its child resources once logs are collected.

Implementing (1) is the difficult bit, since a custom log-forwarding image is required (for production). This setup is also very log-architecture dependent.

[estroz]

After some thought, it seems the controller + finalizer approach is the right way to go for a general solution that enforces lossless log collection. The TTL controller approach works great if you have logging infrastructure that you can drop into a pod spec, which many operators do; the logging implementation is difficult to create a pattern around since usage varies widely depending on what log aggregation system is used.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.