Adds Feat: aws-cli, jq and curl tools by enadezhdin · Pull Request #155 · lifebit-ai/cloudos-cli

enadezhdin · 2023-05-04T11:25:58Z

Overview

This PR adds aws-cli, jq and curl tools to the Dockerfile.

Purpose

To achieve ability to use cloudos-cli functionality in nextflow pipelines integrated into DLC automation flows on CloudOS. The part of the automation triggers require ability to fetch and process parameters/secrets from AWS Parameter store, and added tools perform exactly these functions.

Acceptance Criteria/Jira Ticket

DEL-13256

Changes

Implements N/A
Refactors N/A
Adds/Removes: Adds the following command to Dockerfile:
RUN apt-get update && apt-get install jq awscli curl -y
Fixes/Mentions Issue/s # N/A

Internal CloudOS Tests

New functionality was tested on client's cloudOS workspace only due availability of properly configured AWS Parameter store parameters.

Client CloudOS Tests

-profile cpu_admin_prod
Downstream pipeline job failed due to lack of DLC-specific config, which will be addressed in the corresponding PR for that pipeline.

PR Checklist

Author to check:

The PR title is informative and begins with appropriate [Fix:|Feat:|Docs:] based on PR type
This PR is raised to dev branch for DSL1 or DSL2-dev for DSL2 translations
The description contains the acceptance criteria or link to the associated Jira ticket
The description contains links to CloudOS tests covering all test cases in client environment

Reviewer to check:

A new test profile covering new functions for acceptance criteria is created if it does not already exist
Nextflow configs have associated profiles named to match the config basename
The Jenkinsfile has been updated with current Nextflow profiles
The internal_lifebit_cloudos_ci.yml has been updated to run all current Nextflow profiles
Redundant tests have been removed
The README.md has been updated with parameter changes and current results/ and conforms to the README.md guidelines
The code is logical, readable, reasonably optimal and meets the acceptance criteria
The containers folder has been updated with the necessary containers to run this pipeline if needed

* feat(parameters): add parameter for customization. See #90 * docs: add documentation in changelog and readme * bump: increse minor version 1.0.0 to 1.1.0 * refactor: replace command line option from cromwell with constant variable * docs: update about request interval option * refactor: add control expression to not surpass wait time * refactor: change variable to the constant for Cromwell jobs

Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com>

* Add new cost limit as default 30$ * CHANGELOG updated * VERSION updated * Update cloudos/__main__.py Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> * version update * Update README.md * version bump to 1.2.1 Co-authored-by: Erdal Genc <erdalgenc@Erdals-MacBook-Pro.local> Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> Co-authored-by: Christina Chatzipantsiou <38183826+cgpu@users.noreply.github.com> Co-authored-by: dapineyro <dapineyro.dev@gmail.com>

* fix BadRequestException #84 * make wdl-importsfile optional * update docs * simplify code * simplify code 2 * simplify code 3 * simplify code 4 * small doc change * fix repo for projects * Update to prepare a release

* fix BadRequestException #84 * make wdl-importsfile optional * update docs * simplify code * simplify code 2 * simplify code 3 * simplify code 4 * small doc change * fix repo for projects * Use API key when no cromwell key is available * Adds click support * small help change * Updates docs * Minor doc changes

* Pytests added for _create_cromwell_header function * unused code removed * Apply suggestions from code review Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> * new folder for cloudos tests * init Co-authored-by: Erdal Genc <erdalgenc@Erdals-MacBook-Pro.local> Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com>

* test: add tests for 'get_job_list' * refactor: add new assert statement * refactor: apply suggestions from code review Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> * style: move to proper folder * style: update tests/test_clos/test_get_job_list.py Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com>

* fix error messages * Bump version to v1.3.1 * correct pytests using BadRequestException

* pytest for BadRequestException * renamed function * Apply suggestions from code review Co-authored-by: Daniel Boloc <danielboloc@users.noreply.github.com> * requested changes * test moved to folder Co-authored-by: Erdal Genc <erdalgenc@Erdals-MacBook-Pro.local> Co-authored-by: Daniel Boloc <danielboloc@users.noreply.github.com>

* pytest for function project_id * script updated * Apply suggestions from code review Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> * module docstring added * Update test_project_id.py * requested changes * Update tests/test_clos/test_project_id.py Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> Co-authored-by: Erdal Genc <erdalgenc@Erdals-MacBook-Pro.local> Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com>

* proper json added * pytest function added * script updated for proper workflow name * Apply suggestions from code review Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> * import added * module docstring * Update tests/test_clos/test_detect_workflow.py Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> Co-authored-by: Erdal Genc <erdalgenc@Erdals-MacBook-Pro.local> Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com>

* pytest added for output workflow_id * Object Class is modified to Job * values are masked * json update, assert update * module docstring added * Apply suggestions from code review Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> * requested changes Co-authored-by: Erdal Genc <erdalgenc@Erdals-MacBook-Pro.local> Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com>

* Pytest for process_workflow_list added * file moved to test_clos folder * Apply suggestions from code review Co-authored-by: Daniel Boloc <danielboloc@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> * updated functions * Apply suggestions from code review Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> * code improvement * Update tests/test_clos/test_process_workflow_list.py Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> * Update tests/test_clos/test_process_workflow_list.py Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> Co-authored-by: Erdal Genc <erdalgenc@Erdals-MacBook-Pro.local> Co-authored-by: Daniel Boloc <danielboloc@users.noreply.github.com> Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com>

* test: add tests for 'get_workflow_list' method * refactor: apply suggestions from code review Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> * test: update bad request code and workflows .json * move test to proper folder * style: remove comment line * test: add workflows .json file Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com>

* pytest timeoutexception * descr update * Update test_timeout_exception.py * test moved to folder Co-authored-by: Erdal Genc <erdalgenc@Erdals-MacBook-Pro.local>

* Pytest for send_job * code improvment * update * Apply suggestions from code review Co-authored-by: Daniel Boloc <danielboloc@users.noreply.github.com> Co-authored-by: Erdal Genc <erdalgenc@Erdals-MacBook-Pro.local> Co-authored-by: Daniel Boloc <danielboloc@users.noreply.github.com>

* Pytest added for function get_cromwell_status * code layout fix * Update test_get_cromwell_status.py * Apply suggestions from code review Co-authored-by: Daniel Boloc <danielboloc@users.noreply.github.com> Co-authored-by: Erdal Genc <erdalgenc@Erdals-MacBook-Pro.local> Co-authored-by: Daniel Boloc <danielboloc@users.noreply.github.com>

* Pytest added for function ssl_selector * code improvements * test added for check ssl cert is a file * Apply suggestions from code review Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> * requested changes are made * blank Co-authored-by: Erdal Genc <erdalgenc@Erdals-MacBook-Pro.local> Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com>

* Pytest for cromwell_switch * Apply suggestions from code review Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> Co-authored-by: Erdal Genc <erdalgenc@Erdals-MacBook-Pro.local> Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com>

* pytest get_job_status * naming * out message fix * Apply suggestions from code review Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> * proper request json added * print removed * job status assertion update * Apply suggestions from code review Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com> Co-authored-by: Erdal Genc <erdalgenc@Erdals-MacBook-Pro.local> Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com>

* fix the new api while maintaining backwards compatibility * update docs * update README.md date

* patch for the new api spec * update docs * update docs 2 * modify test data * fix test data * fix test data 2

* adding pytest * update setup * update setup 2 * update setup 3 * update setup 3 * adding job list * fix name * adds job run * adds job status * adds job status 2 * adds job status 2 * some debugging code * fix job status * adds workflow list * adding a job name * adding a job name 2 * point to staging * point to staging 2 * point to staging 3 * point to staging 4 * point to staging 5 * point to staging 6 * point to staging 7 * point to staging 8 * point to staging 9 * point to staging 10

* implementing new feat * fix multiple spaces * typos * including review suggestions * fix pytests

* remove version from README.md * docs: add commitizen support * changelog

* feat: add project list * bump: version 2.1.0 → 2.2.0 * update docs * add tests * adds ci test * update tests 2 * minor fixes * adapt to different responses * improve docstring * fix README.md * more docstring fixes

* first implementation + fix job list * adds parameters * fix pytests * remove curated-with-defaults * update docs * add ci test * small doc fixes * improve docstring

* first working prototype * implement first wait-completion * prettify the output * adding fixes * fix pytests * add new pytest * adding ci test * feat: add run-curated-examples command * bump: version 2.2.0 → 2.3.0 * updated changelog and version * README.md improvement * fixing pytest name * fix docstring typo

* main.py CLI command implementation * first working implementation * fix api call * update docs * add pytests * feat: add queue list command * bump: version 2.3.0 → 2.4.0 * update changelog * update ci.yml

* main.py CLI command implementation * first working implementation * fix api call * update docs * add pytests * feat: add queue list command * bump: version 2.3.0 → 2.4.0 * update changelog * update ci.yml * first implementation * improve messages * update pytest * fix pytest * update CI * update README.md * update README.md 2 * update README.md 3 * update README.md 4 * update README.md 5 * feat: adds job queue support to batch runs * bump: version 2.4.0 → 2.5.0 * update changelog

danielboloc and others added 30 commits October 3, 2022 19:08

Adds ssl verification control (#103)

70d3dd0

Docs patch for SSL functionality (#106)

2a1d4f0

Update README.md

8eb820c

Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com>

Update README.md

6477245

Co-authored-by: dapineyro <45285897+dapineyro@users.noreply.github.com>

Resolve conflicts

c816047

Feat: make --wdl-importsFile optional (#98)

ddedb58

* fix BadRequestException #84 * make wdl-importsfile optional * update docs * simplify code * simplify code 2 * simplify code 3 * simplify code 4 * small doc change * fix repo for projects * Update to prepare a release

Merge branch 'main' into dev

4b9bcf6

Remove cromwell_token

209d8ee

Fix: resolve JSON error in error message handling (#124)

dd6e1b4

* fix error messages * Bump version to v1.3.1 * correct pytests using BadRequestException

Pytest added for TimeOutException Class (#127)

f9ae545

* pytest timeoutexception * descr update * Update test_timeout_exception.py * test moved to folder Co-authored-by: Erdal Genc <erdalgenc@Erdals-MacBook-Pro.local>

Fix: adapt to new projects api (#135)

2db3234

* fix the new api while maintaining backwards compatibility * update docs * update README.md date

remove cb (#136)

e1729fa

Merge branch 'main' into dev

9d37dd0

dapineyro and others added 13 commits March 8, 2023 11:26

Fix: preparation for new jobs api endpoint spec (#138)

861799a

* patch for the new api spec * update docs * update docs 2 * modify test data * fix test data * fix test data 2

Merge branch 'main' into dev

2f48d31

Feat: adds response pagination support for job list endpoint (#144)

51cf842

* implementing new feat * fix multiple spaces * typos * including review suggestions * fix pytests

Feat: adds commitizen (#145)

bc8203e

* remove version from README.md * docs: add commitizen support * changelog

Feat: adds projects list (#146)

9f8f7da

* feat: add project list * bump: version 2.1.0 → 2.2.0 * update docs * add tests * adds ci test * update tests 2 * minor fixes * adapt to different responses * improve docstring * fix README.md * more docstring fixes

Feat: adds workflows --curated (#148)

f1c80d4

* first implementation + fix job list * adds parameters * fix pytests * remove curated-with-defaults * update docs * add ci test * small doc fixes * improve docstring

Dockerfile updated with awscli and jq tools added

6424b2b

Feat: adds queue command (#151)

3940cff

* main.py CLI command implementation * first working implementation * fix api call * update docs * add pytests * feat: add queue list command * bump: version 2.3.0 → 2.4.0 * update changelog * update ci.yml

Dockerfile updated with add curl tool

8372959

Merge branch 'main' into dev

e9d3e20

enadezhdin requested review from dapineyro and sk-sahu May 4, 2023 11:26

enadezhdin changed the base branch from main to dev May 4, 2023 12:04

dapineyro added 2 commits May 5, 2023 14:56

changing python requeriment to 3.7

7596cac

testing 3.7 with pytest

e3b12dc

Base automatically changed from dev to main May 5, 2023 13:43

enadezhdin changed the base branch from main to dev May 22, 2023 11:06

Merge branch 'dev' into adds-aws-cli-jq

5a1d268

Base automatically changed from dev to main June 23, 2023 13:52

Merge changes from main branch

66bd27d

dapineyro closed this May 20, 2025

dapineyro deleted the adds-aws-cli-jq branch May 20, 2025 09:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds Feat: aws-cli, jq and curl tools#155

Adds Feat: aws-cli, jq and curl tools#155
enadezhdin wants to merge 47 commits intomainfrom
adds-aws-cli-jq

enadezhdin commented May 4, 2023 •

edited by atlassian bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

enadezhdin commented May 4, 2023 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Purpose

Acceptance Criteria/Jira Ticket

Changes

Internal CloudOS Tests

Client CloudOS Tests

PR Checklist

Author to check:

Reviewer to check:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

enadezhdin commented May 4, 2023 •

edited by atlassian bot

Loading