Cloud Run Jobs customization

[AdeelK93]

Hi I'm trying to customize a few things about the cloud run jobs deployment, but can't find any documentation or examples on how to change:

• cpu/memory resources
• timeout
• number of retries
• VPC connector
• total number of tasks

Would love some help! Thanks

[anovis]

Thanks for the issue and will add some more documentation to the example config. Most fields can be set in either the job spec or job container fields. https://cloud.google.com/run/docs/reference/rest/v1/TaskSpec for a reference.

The total number of tasks is calculated by goblet based on the number of task_id set for the specific job.
ie

app.job("test", task_id=0)
app.job("test", task_id=1)
app.job("test", task_id=2)

will result in number of tasks to be 3.

[rasnes]

By looking at the code it was possible to see how one could change this values by directly controlling the JSON that goblet deploys the Job with:

goblet/goblet/resources/jobs.py

Lines 46 to 88 in 0c0146a

	def _deploy(self, source=None, entrypoint=None, config={}):
	if not self.resources:
	return
	config = GConfig(config=config)
	artifact = getCloudbuildArtifact(
	self.versioned_clients.cloudbuild, self.name, config=config
	)
	log.info("deploying cloudrun jobs......")
	for job_name, job in self.resources.items():
	container = {**(config.job_container or {})}
	container["image"] = artifact
	container["command"] = [
	"goblet",
	"job",
	"run",
	job_name,
	]
	job_spec = {
	"apiVersion": "run.googleapis.com/v1",
	"kind": "Job",
	"metadata": {
	"name": job_name,
	"annotations": {"run.googleapis.com/launch-stage": "BETA"},
	},
	"spec": {
	"template": {
	"spec": {
	"taskCount": len(job.keys()) - 1,
	"template": {
	"spec": {
	"containers": [container],
	**(config.job_spec or {}),
	}
	},
	},
	**job["execution_spec"],
	}
	},
	}

For example, I now deploy with this in my .goblet/config.json:

{
	...,
    "job_spec": {
        "serviceAccountName": "<sa-name>@<project-id>.iam.gserviceaccount.com",
        "timeoutSeconds": "3600",
        "maxRetries": 3
    },
    "job_container": {
        "resources": {
            "limits": {
                "memory": "8Gi",
                "cpu": "2000m"
            }
        },
        "env": [
            {
                "name": "GOOGLE_PROJECT",
                "value": "<project-id>"
            },
            {
                "name": "GOOGLE_ENV",
                "value": "test"
            }
        ]
    }
}

I believe you can set the VPC connector in this JSON as well, but haven't checked.

[anovis]

@rasnes thanks for the example. i have updated the example config.json and will try to add more examples there going forward as it is a pain to read the code or go through the gcp docs to figure out where to place what.

also it looks like the vpc connector info is actually set as annotations in the metadata field, so added a pr that allows those fields to be passed in as well (#262 )

[AdeelK93]

Thank you both, this was very helpful!
job_annotations - who woulda guessed.