Feat: release-v2.11.0 - Adds save_logs and removes spot options from job command

[dapineyro]

Overview

This PR does two things:

• Now, cloudos job run command will save process logs. There is a new flag parameter: --do-not-save-logs to allow not saving logs (previous default).
• Removes the deprecated --spot option, which is no longer available in CloudOS. However, to maintain backwards compatibility, if the user uses --spot, a message will be displayed informing that the parameter is no longer valid.

Jira

https://lifebit.atlassian.net/browse/LIF-290
https://lifebit.atlassian.net/browse/LIF-293

Changes

• Adds new --do-not-save-logs parameters for all the job run commands (cloudos job run and cloudos job run-curated-examples).
• Removes --spot option from job run commands.
• Adds --do-not-save-logs to the message when running hpc executor, as it will be ignored.
• Bumps version to 2.11.0
• Updates CHANGELOG.md, README.md and _version.py
• Updates Pytest

Tests

Using docker image:

docker run --rm -it quay.io/lifebitaiorg/cloudos-cli:v2.11.0

Global variables:

MY_API_KEY="xxxx"
CLOUDOS="https://cloudos.lifebit.ai"
WORKSPACE_ID="xxxx"
PROJECT_NAME="davidp_testing"
WORKFLOW_NAME="Cufflinks pipeline"
JOB_PARAMS="/cloudos/examples/rnatoy.config"

Running cloudos job run (default)

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name "$WORKFLOW_NAME" \
    --job-config $JOB_PARAMS \
    --job-name "test_default_job_call" \
    --resumable
CloudOS python package: a package for interacting with CloudOS.

Version: 2.11.0

CloudOS job functionality: run and check jobs in CloudOS.

Executing run...
	No job_queue was specified, using the CloudOS default queue: On_demand_standard_stable_3TB.
	Job successfully launched to CloudOS, please check the following link: https://cloudos.lifebit.ai/app/jobs/661e55a7718a60a7415a4095
	Your assigned job id is: 661e55a7718a60a7415a4095

	Your current job status is: initializing
	To further check your job status you can either go to https://cloudos.lifebit.ai/app/jobs/661e55a7718a60a7415a4095 or use the following command:
	cloudos job status \
		--apikey $MY_API_KEY \
		--cloudos-url https://cloudos.lifebit.ai \
		--job-id 661e55a7718a60a7415a4095

Running cloudos job run-curated-examples (default)

NOTE: manually aborted at the third pipeline to prevent running so many jobs

cloudos job run-curated-examples \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME"
CloudOS python package: a package for interacting with CloudOS.

Version: 2.11.0

CloudOS job functionality: run and check jobs in CloudOS.

	Job successfully launched to CloudOS, please check the following link: https://cloudos.lifebit.ai/app/jobs/661e563f718a60a7415a573e
	Your assigned job id is: 661e563f718a60a7415a573e

	Job successfully launched to CloudOS, please check the following link: https://cloudos.lifebit.ai/app/jobs/661e565e718a60a7415a69f8
	Your assigned job id is: 661e565e718a60a7415a69f8

	Job successfully launched to CloudOS, please check the following link: https://cloudos.lifebit.ai/app/jobs/661e567e718a60a7415a7e61
	Your assigned job id is: 661e567e718a60a7415a7e61

^C
Aborted!

Running cloudos job run --execution-platform hpc

NOTE: test on DEV2

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name "$WORKFLOW_NAME" \
    --job-config $JOB_PARAMS \
    --execution-platform hpc \
    --hpc-id $HPC_ID \
    --job-name "test_1_hpc_cloudos_cli"
CloudOS python package: a package for interacting with CloudOS.

Version: 2.11.0

CloudOS job functionality: run and check jobs in CloudOS.

Executing run...

HPC execution platform selected
[Message] Please, take into account that HPC execution do not support the following parameters and all of them will be ignored:
	--job-queue
	--resumable | --do-not-save-logs
	--instance-type | --instance-disk | --cost-limit
	--storage-mode | --lustre-size
	--wdl-mainfile | --wdl-importsfile | --cromwell-token

	Job successfully launched to CloudOS, please check the following link: https://dev2.sdlc.lifebit.ai/app/jobs/661e6f86b38e7701782a613b
	Your assigned job id is: 661e6f86b38e7701782a613b

	Your current job status is: running
	To further check your job status you can either go to https://dev2.sdlc.lifebit.ai/app/jobs/661e6f86b38e7701782a613b or use the following command:
	cloudos job status \
		--apikey $MY_API_KEY \
		--cloudos-url https://dev2.sdlc.lifebit.ai \
		--job-id 661e6f86b38e7701782a613b

Running cloudos job run not saving logs (using --do-not-save-logs flag)

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name "$WORKFLOW_NAME" \
    --job-config $JOB_PARAMS \
    --job-name "test_default_job_call" \
    --do-not-save-logs \
    --resumable
CloudOS python package: a package for interacting with CloudOS.

Version: 2.11.0

CloudOS job functionality: run and check jobs in CloudOS.

Executing run...
	No job_queue was specified, using the CloudOS default queue: On_demand_standard_stable_3TB.
	Job successfully launched to CloudOS, please check the following link: https://cloudos.lifebit.ai/app/jobs/661e7139718a60a7415e2eb9
	Your assigned job id is: 661e7139718a60a7415e2eb9

	Your current job status is: initializing
	To further check your job status you can either go to https://cloudos.lifebit.ai/app/jobs/661e7139718a60a7415e2eb9 or use the following command:
	cloudos job status \
		--apikey $MY_API_KEY \
		--cloudos-url https://cloudos.lifebit.ai \
		--job-id 661e7139718a60a7415e2eb9

Running cloudos job run using --spot option

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name "$WORKFLOW_NAME" \
    --job-config $JOB_PARAMS \
    --job-name "test_default_job_call_spot" \
    --resumable --spot
CloudOS python package: a package for interacting with CloudOS.

Version: 2.11.0

CloudOS job functionality: run and check jobs in CloudOS.

Executing run...

[Message] You have specified spot instances but they are no longer available in CloudOS. Option ignored.

	No job_queue was specified, using the CloudOS default queue: On_demand_standard_stable_3TB.
	Job successfully launched to CloudOS, please check the following link: https://cloudos.lifebit.ai/app/jobs/661e7203718a60a7415eb4ff
	Your assigned job id is: 661e7203718a60a7415eb4ff

	Your current job status is: initializing
	To further check your job status you can either go to https://cloudos.lifebit.ai/app/jobs/661e7203718a60a7415eb4ff or use the following command:
	cloudos job status \
		--apikey $MY_API_KEY \
		--cloudos-url https://cloudos.lifebit.ai \
		--job-id 661e7203718a60a7415eb4ff

[danielboloc]

LGTM 🙌