k8s job manager changes

[johnbradley]

Changes removed from #139 to simplify the review process.

Adds a k8s module under lando consisting of:

• ServerConfig config file for k8s version of lando
• ClusterApi class for simplifying interacting with k8s
• JobManager class that uses ClusterApi to create volumes and run jobs for various steps of running the job.

JobManager

The JobManager has methods for creating jobs for the following job steps:

lando/lando/k8s/jobmanager.py

Lines 15 to 19 in f8b661f

	class JobStepTypes(object):
	STAGE_DATA = "stage_data"
	RUN_WORKFLOW = "run_workflow"
	ORGANIZE_OUTPUT = "organize_output"
	SAVE_OUTPUT = "save_output"

For each job step handled by JobManager there are at most 3 methods:

• a method to setup volumes before running the job. eg: create_stage_data_persistent_volumes
• create job to perform the step. eg: create_stage_data_job
• cleanup job. eg: cleanup_stage_data_job

In PR #139 the k8s lando server will use JobManager to run k8s jobs for various steps.

Created #141 to improve the volume sizes for the persistent volume claims used in running a job.

ClusterApi

This class performs two main simplifications instead of using the python kubernetes module directly.

• removes the requirement to pass namespace to every call
• allows users to specify volumes to be mounted by jobs once (instead of twice)

[dleehr]

LGTM. I looked over everything, but as this is mostly underlying infrastructure did not attempt to run it myself.

[dleehr]

A super secret volume? 🔐

[dleehr]

👍